Saravanan666/pypi-org huggingface dataset:
Package: llama-index-readers-airbyte-gong
Airbyte Gong LoaderThe Airbyte Gong Loader allows you to access different Gong objects.InstallationInstall llama_hub:pip install llama_hubInstall the gong source:pip install airbyte-source-gongUsageHere's an example usage of the AirbyteGongReader.fromllama_hub.airbyte_gongimportAirbyteGongReadergong_config={# ...}reader=AirbyteGongReader(config=gong_config)documents=reader.load_data(stream_name="calls")ConfigurationCheck out theAirbyte documentation pagefor details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github:https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-gong/source_gong/spec.yaml.The general shape looks like this:{"access_key":"<access key name>","access_key_secret":"<access key secret>","start_date":"<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",}By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing arecord_handlerto the reader:defhandle_record(record,id):returnDocument(doc_id=id,text=record.data["title"],extra_info=record.data)reader=AirbyteGongReader(config=gong_config,record_handler=handle_record)Lazy loadsThereader.load_dataendpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By usingreader.lazy_load_datainstead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.Incremental loadsThis loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):reader=AirbyteGongReader(config={...})documents=reader.load_data(stream_name="calls")current_state=reader.last_state# can be pickled away or stored otherwiseupdated_documents=reader.load_data(stream_name="calls",state=current_state)# only loads documents that were updated since last timeThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-airbyte-hubspot
Airbyte Hubspot LoaderThe Airbyte Hubspot Loader allows you to access different Hubspot objects.InstallationInstall llama_hub:pip install llama_hubInstall the hubspot source:pip install airbyte-source-hubspotUsageHere's an example usage of the AirbyteHubspotReader.fromllama_hub.airbyte_hubspotimportAirbyteHubspotReaderhubspot_config={# ...}reader=AirbyteHubspotReader(config=hubspot_config)documents=reader.load_data(stream_name="products")ConfigurationCheck out theAirbyte documentation pagefor details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github:https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-hubspot/source_hubspot/spec.yaml.The general shape looks like this:{"start_date":"<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>","credentials":{"credentials_title":"Private App Credentials","access_token":"<access token of your private app>",},}By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing arecord_handlerto the reader:defhandle_record(record,id):returnDocument(doc_id=id,text=record.data["title"],extra_info=record.data)reader=AirbyteHubspotReader(config=hubspot_config,record_handler=handle_record)Lazy loadsThereader.load_dataendpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By usingreader.lazy_load_datainstead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.Incremental loadsThis loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):reader=AirbyteHubspotReader(config={...})documents=reader.load_data(stream_name="products")current_state=reader.last_state# can be pickled away or stored otherwiseupdated_documents=reader.load_data(stream_name="products",state=current_state)# only loads documents that were updated since last timeThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-airbyte-salesforce
Airbyte Salesforce LoaderThe Airbyte Salesforce Loader allows you to access different Salesforce objects.InstallationInstall llama_hub:pip install llama_hubInstall the salesforce source:pip install airbyte-source-salesforceUsageHere's an example usage of the AirbyteSalesforceReader.fromllama_hub.airbyte_salesforceimportAirbyteSalesforceReadersalesforce_config={# ...}reader=AirbyteSalesforceReader(config=salesforce_config)documents=reader.load_data(stream_name="asset")ConfigurationCheck out theAirbyte documentation pagefor details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github:https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce/source_salesforce/spec.yaml.The general shape looks like this:{"client_id":"<oauth client id>","client_secret":"<oauth client secret>","refresh_token":"<oauth refresh token>","start_date":"<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>","is_sandbox":False,# set to True if you're using a sandbox environment"streams_criteria":[# Array of filters for salesforce objects that should be loadable{"criteria":"exacts","value":"Account",},# Exact name of salesforce object{"criteria":"starts with","value":"Asset"},# Prefix of the name# Other allowed criteria: ends with, contains, starts not with, ends not with, not contains, not exacts],}By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing arecord_handlerto the reader:defhandle_record(record,id):returnDocument(doc_id=id,text=record.data["title"],extra_info=record.data)reader=AirbyteSalesforceReader(config=salesforce_config,record_handler=handle_record)Lazy loadsThereader.load_dataendpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By usingreader.lazy_load_datainstead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.Incremental loadsThis loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):reader=AirbyteSalesforceReader(config={...})documents=reader.load_data(stream_name="asset")current_state=reader.last_state# can be pickled away or stored otherwiseupdated_documents=reader.load_data(stream_name="asset",state=current_state)# only loads documents that were updated since last timeThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-airbyte-shopify
Airbyte Shopify LoaderThe Airbyte Shopify Loader allows you to access different Shopify objects.InstallationInstall llama_hub:pip install llama_hubInstall the shopify source:pip install airbyte-source-shopifyUsageHere's an example usage of the AirbyteShopifyReader.fromllama_hub.airbyte_shopifyimportAirbyteShopifyReadershopify_config={# ...}reader=AirbyteShopifyReader(config=shopify_config)documents=reader.load_data(stream_name="orders")ConfigurationCheck out theAirbyte documentation pagefor details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github:https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-shopify/source_shopify/spec.json.The general shape looks like this:{"start_date":"<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>","shop":"<name of the shop you want to retrieve documents from>","credentials":{"auth_method":"api_password","api_password":"<your api password>",},}By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing arecord_handlerto the reader:defhandle_record(record,id):returnDocument(doc_id=id,text=record.data["title"],extra_info=record.data)reader=AirbyteShopifyReader(config=shopify_config,record_handler=handle_record)Lazy loadsThereader.load_dataendpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By usingreader.lazy_load_datainstead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.Incremental loadsThis loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):reader=AirbyteShopifyReader(config={...})documents=reader.load_data(stream_name="orders")current_state=reader.last_state# can be pickled away or stored otherwiseupdated_documents=reader.load_data(stream_name="orders",state=current_state)# only loads documents that were updated since last timeThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-airbyte-stripe
Airbyte Stripe LoaderThe Airbyte Stripe Loader allows you to access different Stripe objects.InstallationInstall llama_hub:pip install llama_hubInstall the stripe source:pip install airbyte-source-stripeUsageHere's an example usage of the AirbyteStripeReader.fromllama_hub.airbyte_stripeimportAirbyteStripeReaderstripe_config={# ...}reader=AirbyteStripeReader(config=stripe_config)documents=reader.load_data(stream_name="invoices")ConfigurationCheck out theAirbyte documentation pagefor details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github:https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/spec.yaml.The general shape looks like this:{"client_secret":"<secret key>","account_id":"<account id>","start_date":"<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",}By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing arecord_handlerto the reader:defhandle_record(record,id):returnDocument(doc_id=id,text=record.data["title"],extra_info=record.data)reader=AirbyteStripeReader(config=stripe_config,record_handler=handle_record)Lazy loadsThereader.load_dataendpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By usingreader.lazy_load_datainstead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.Incremental loadsThis loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):reader=AirbyteStripeReader(config={...})documents=reader.load_data(stream_name="invoices")current_state=reader.last_state# can be pickled away or stored otherwiseupdated_documents=reader.load_data(stream_name="invoices",state=current_state)# only loads documents that were updated since last timeThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-airbyte-typeform
Airbyte Typeform LoaderThe Airbyte Typeform Loader allows you to access different Typeform objects.InstallationInstall llama_hub:pip install llama_hubInstall the typeform source:pip install airbyte-source-typeformUsageHere's an example usage of the AirbyteTypeformReader.fromllama_hub.airbyte_typeformimportAirbyteTypeformReadertypeform_config={# ...}reader=AirbyteTypeformReader(config=typeform_config)documents=reader.load_data(stream_name="forms")ConfigurationCheck out theAirbyte documentation pagefor details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github:https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-typeform/source_typeform/spec.json.The general shape looks like this:{"credentials":{"auth_type":"Private Token","access_token":"<your auth token>",},"start_date":"<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>","form_ids":["<id of form to load records for>"],# if omitted, records from all forms will be loaded}By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing arecord_handlerto the reader:defhandle_record(record,id):returnDocument(doc_id=id,text=record.data["title"],extra_info=record.data)reader=AirbyteTypeformReader(config=typeform_config,record_handler=handle_record)Lazy loadsThereader.load_dataendpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By usingreader.lazy_load_datainstead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.Incremental loadsThis loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):reader=AirbyteTypeformReader(config={...})documents=reader.load_data(stream_name="forms")current_state=reader.last_state# can be pickled away or stored otherwiseupdated_documents=reader.load_data(stream_name="forms",state=current_state)# only loads documents that were updated since last timeThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-airbyte-zendesk-support
Airbyte ZendeskSupport LoaderThe Airbyte ZendeskSupport Loader allows you to access different ZendeskSupport objects.InstallationInstall llama_hub:pip install llama_hubInstall the zendesk_support source:pip install airbyte-source-zendesk-supportUsageHere's an example usage of the AirbyteZendeskSupportReader.fromllama_hub.airbyte_zendesk_supportimportAirbyteZendeskSupportReaderzendesk_support_config={# ...}reader=AirbyteZendeskSupportReader(config=zendesk_support_config)documents=reader.load_data(stream_name="tickets")ConfigurationCheck out theAirbyte documentation pagefor details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github:https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/spec.json.The general shape looks like this:{"subdomain":"<your zendesk subdomain>","start_date":"<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>","credentials":{"credentials":"api_token","email":"<your email>","api_token":"<your api token>",},}By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing arecord_handlerto the reader:defhandle_record(record,id):returnDocument(doc_id=id,text=record.data["title"],extra_info=record.data)reader=AirbyteZendeskSupportReader(config=zendesk_support_config,record_handler=handle_record)Lazy loadsThereader.load_dataendpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By usingreader.lazy_load_datainstead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.Incremental loadsThis loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):reader=AirbyteZendeskSupportReader(config={...})documents=reader.load_data(stream_name="tickets")current_state=reader.last_state# can be pickled away or stored otherwiseupdated_documents=reader.load_data(stream_name="tickets",state=current_state)# only loads documents that were updated since last timeThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-airtable
Airtable LoaderThis loader loads documents from Airtable. The user specifies an API token to initialize the AirtableReader. They then specify atable_idand abase_idto load in the corresponding Document objects.UsageHere's an example usage of the AirtableReader.fromllama_indeximportdownload_loaderimportosAirtableReader=download_loader("AirtableReader")reader=AirtableReader("<Airtable_TOKEN>")documents=reader.load_data(table_id="<TABLE_ID>",base_id="<BASE_ID>")This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-apify
Apify LoadersApify Actor LoaderApifyis a cloud platform for web scraping and data extraction,
which provides anecosystemof more than a thousand
ready-made apps calledActorsfor various scraping, crawling, and extraction use cases.This loader runs a specific Actor and loads its results.UsageIn this example, we’ll use theWebsite Content CrawlerActor,
which can deeply crawl websites such as documentation, knowledge bases, help centers,
or blogs, and extract text content from the web pages.
The extracted text then can be fed to a vector index or language model like GPT
in order to answer questions from it.To use this loader, you need to have a (free) Apify account
and set yourApify API tokenin the code.fromllama_indeximportdownload_loaderfromllama_index.readers.schemaimportDocument# Converts a single record from the Actor's resulting dataset to the LlamaIndex formatdeftranform_dataset_item(item):returnDocument(text=item.get("text"),extra_info={"url":item.get("url"),},)ApifyActor=download_loader("ApifyActor")reader=ApifyActor("<My Apify API token>")documents=reader.load_data(actor_id="apify/website-content-crawler",run_input={"startUrls":[{"url":"https://gpt-index.readthedocs.io/en/latest"}]},dataset_mapping_function=tranform_dataset_item,)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently
used as a Tool in aLangChainAgent.
Seeherefor examples.Apify Dataset LoaderApifyis a cloud platform for web scraping and data extraction,
which provides anecosystemof more than a thousand
ready-made apps calledActorsfor various scraping, crawling, and extraction use cases.This loader loads documents from an existingApify dataset.UsageIn this example, we’ll load a dataset generated by
theWebsite Content CrawlerActor,
which can deeply crawl websites such as documentation, knowledge bases, help centers,
or blogs, and extract text content from the web pages.
The extracted text then can be fed to a vector index or language model like GPT
in order to answer questions from it.To use this loader, you need to have a (free) Apify account
and set yourApify API tokenin the code.fromllama_indeximportdownload_loaderfromllama_index.readers.schemaimportDocument# Converts a single record from the Apify dataset to the LlamaIndex formatdeftranform_dataset_item(item):returnDocument(text=item.get("text"),extra_info={"url":item.get("url"),},)ApifyDataset=download_loader("ApifyDataset")reader=ApifyDataset("<Your Apify API token>")documents=reader.load_data(dataset_id="<Apify Dataset ID>",dataset_mapping_function=tranform_dataset_item,)
Package: llama-index-readers-arango-db
LlamaIndex Readers Integration: Arango DbThis loader loads documents from ArangoDB. The user specifies a ArangoDB instance to
initialize the reader. They then specify the collection name and query params to
fetch the relevant docs.UsageHere's an example usage of the SimpleArangoDBReader.fromllama_index.core.readersimportdownload_loaderimportosSimpleArangoDBReader=download_loader("SimpleArangoDBReader")host="<host>"db_name="<db_name>"collection_name="<collection_name>"# query_dict is passed into db.collection.find()query_dict={}# Attribute of interests to load, by default ["text"]field_names=["title","description"]reader=SimpleArangoDBReader(host)# or pass ArangoClientdocuments=reader.load_data(username,password,db_name,collection_name,query_dict=query_dict,field_names=field_names,)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-asana
Asana LoaderThis loader loads documents from Asana. The user specifies an API token to initialize the AsanaReader. They then specify aworkspace_idOR aproject_idto load in the corresponding Document objects.UsageHere's an example usage of the AsanaReader.fromllama_indeximportdownload_loaderimportosAsanaReader=download_loader("AsanaReader")reader=AsanaReader("<ASANA_TOKEN>")# Option 1documents=reader.load_data(workspace_id="<WORKSPACE_ID>")# Option 2documents=reader.load_data(project_id="<PROJECT_ID>")This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-astra-db
Astra DB LoaderThe Astra DB Loader returns a set of documents retrieved from Astra DB.
The user initializes the loader with an Astra DB index. They then pass in a vector.UsageHere's an example usage of the AstraDBReader.fromopenaiimportOpenAIfromllama_indeximportdownload_loader# Get the credentials for Astra DBapi_endpoint="https://324<...>f1c.astra.datastax.com"token="AstraCS:<...>"# EXAMPLE: OpenAI embeddingsclient=OpenAI(api_key="sk-<...>")# Call OpenAI (or generate embeddings another way)response=client.embeddings.create(input="Your text string goes here",model="text-embedding-ada-002")# Get the embeddingquery_vector=response.data[0].embedding# Initialize the Reader objectAstraDBReader=download_loader("AstraDBReader")# Your Astra DB Account will provide you with the endpoint URL and Tokenreader=AstraDBReader(collection_name="astra_v_table",token=token,api_endpoint=api_endpoint,embedding_dimension=len(query_vector),)# Fetch data from the readerdocuments=reader.load_data(vector=query_vector,limit=5)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a tool in aLangChainAgent. Seeherefor examples.Note: Please see the AstraDB documentationhere.
Package: llama-index-readers-athena
Athena reader.Athena reader allow execute SQL with AWS Athena. We using SQLAlchemy and PyAthena under the hood.PermissionsWE HIGHLY RECOMMEND USING THIS LOADER WITH AWS EC2 IAM ROLE.UsageHere's an example usage of the AthenaReader.import os
import dotenv
from llama_index import SQLDatabase,ServiceContext
from llama_index.indices.struct_store import NLSQLTableQueryEngine
from llama_index.llms import OpenAI
from llama_hub.athena import AthenaReader

dotenv.load_dotenv()

AWS_REGION = os.environ['AWS_REGION']
S3_STAGING_DIR = os.environ['S3_STAGING_DIR']
DATABASE = os.environ['DATABASE']
WORKGROUP = os.environ['WORKGROUP']
TABLE = os.environ['TABLE']

llm = OpenAI(model="gpt-4",temperature=0, max_tokens=1024)

engine = AthenaReader.create_athena_engine(
    aws_region=AWS_REGION,
    s3_staging_dir=S3_STAGING_DIR,
    database=DATABASE,
    workgroup=WORKGROUP
)

service_context = ServiceContext.from_defaults(
  llm=llm
)

sql_database = SQLDatabase(engine, include_tables=[TABLE])

query_engine = NLSQLTableQueryEngine(
    sql_database=sql_database,
    tables=[TABLE],
    service_context=service_context
)
query_str = (
    "Which blocknumber has the most transactions?"
)
response = query_engine.query(query_str)Screeshot