Saravanan666/pypi-org huggingface dataset:
Package: llama-index-readers-azcognitive-search
Azure Cognitive Search LoaderThe AzCognitiveSearchReader Loader returns a set of texts corresponding to documents retrieved from specific index of Azure Cognitive Search.
The user initializes the loader with credentials (service name and key) and the index name.UsageHere's an example usage of the AzCognitiveSearchReader.fromllama_indeximportdownload_loaderAzCognitiveSearchReader=download_loader("AzCognitiveSearchReader")reader=AzCognitiveSearchReader("<Azure_Cognitive_Search_NAME>","<Azure_Cognitive_Search_KEY>","<Index_name>",)query_sample=""documents=reader.load_data(query="<search_term>",content_field="<content_field_name>",filter="<azure_search_filter>",)Usage in combination with langchainfromllama_indeximportVectorStoreIndex,download_loaderfromlangchain.chains.conversation.memoryimportConversationBufferMemoryfromlangchain.agentsimportTool,AgentExecutor,load_tools,initialize_agentAzCognitiveSearchReader=download_loader("AzCognitiveSearchReader")az_loader=AzCognitiveSearchReader(COGNITIVE_SEARCH_SERVICE_NAME,COGNITIVE_SEARCH_KEY,INDEX_NAME)documents=az_loader.load_data(query,field_name)index=VectorStoreIndex.from_documents(documents,service_context=service_context)tools=[Tool(name="Azure cognitive search index",func=lambdaq:index.query(q),description=f"Useful when you want answer questions about the text on azure cognitive search.",),]memory=ConversationBufferMemory(memory_key="chat_history")agent_chain=initialize_agent(tools,llm,agent="zero-shot-react-description",memory=memory)result=agent_chain.run(input="How can I contact with my health insurance?")This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-azstorage-blob
Azure Storage Blob LoaderThis loader parses any file stored as an Azure Storage blob or the entire container (with an optional prefix / attribute filter) if no particular file is specified. When initializingAzStorageBlobReader, you may pass in your account url with a SAS token or crdentials to authenticate.All files are temporarily downloaded locally and subsequently parsed withSimpleDirectoryReader. Hence, you may also specify a customfile_extractor, relying on any of the loaders in this library (or your own)! If you need a clue on finding the file extractor object because you'd like to use your own file extractor, follow this sample.importllama_indexfile_extractor=llama_index.readers.file.base.DEFAULT_FILE_READER_CLS# Make sure to use an instantiation of a classfile_extractor.update({".pdf":SimplePDFReader()})UsageTo use this loader, you need to pass in the name of your Azure Storage Container. After that, if you want to just parse a single file, pass in its blob name. Note that if the file is nested in a subdirectory, the blob name should contain the path such assubdirectory/input.txt. This loader is a thin wrapper over theAzure Blob Storage Client for Python, seeContainerClientfor detailed parameter usage options.Using a Storage Account SAS URLfromllama_indeximportdownload_loaderAzStorageBlobReader=download_loader("AzStorageBlobReader")loader=AzStorageBlobReader(container="scrabble-dictionary",blob="dictionary.txt",account_url="<SAS_URL>",)documents=loader.load_data()Using a Storage Account with connection stringThe sample below will download all files in a container, by only specifying the storage account's connection string and the container name.fromllama_indeximportdownload_loaderAzStorageBlobReader=download_loader("AzStorageBlobReader")loader=AzStorageBlobReader(container_name="<CONTAINER_NAME>",connection_string="<STORAGE_ACCOUNT_CONNECTION_STRING>",)documents=loader.load_data()Using Azure ADEnsure the Azure Identity library is availablepip install azure-identityThe sample below downloads all files in the container using the default credential, alternative credential options are available such as a service principalClientSecretCredentialfromllama_indeximportdownload_loaderfromazure.identityimportDefaultAzureCredentialdefault_credential=DefaultAzureCredential()AzStorageBlobReader=download_loader("AzStorageBlobReader")loader=AzStorageBlobReader(container_name="scrabble-dictionary",account_url="https://<storage account name>.blob.core.windows.net",credential=default_credential,)documents=loader.load_data()This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.Updates[2023-12-14] byJAlexMcGraw(#765)Added functionality to allow user to connect to blob storage with connection stringChanged temporary file names from random to back to original names
Package: llama-index-readers-bilibili
Bilibili Transcript LoaderThis loader utilizes thebilibili_apito fetch the text transcript from Bilibili, one of the most beloved long-form video sites in China.With this BilibiliTranscriptReader, users can easily obtain the transcript of their desired video content on the platform.UsageTo use this loader, you need to pass in an array of Bilibili video links.fromllama_indeximportdownload_loaderBilibiliTranscriptReader=download_loader("BilibiliTranscriptReader")loader=BilibiliTranscriptReader()documents=loader.load_data(video_urls=["https://www.bilibili.com/video/BV1yx411L73B/"])Note that there is no official API available for Bilibili Transcript, so changes to the official website can sometimes cause issues.This loader is designed to be used as a way to load data intoLlama Indexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-bitbucket
Bitbucket LoaderThis loader utilizes the Bitbucket API to load the files inside a Bitbucket repository as Documents in an index.UsageTo use this loader, you need to provide as environment variables theBITBUCKET_API_KEYand theBITBUCKET_USERNAME.importosfromllama_indeximportVectorStoreIndex,download_loaderos.environ["BITBUCKET_USERNAME"]="myusername"os.environ["BITBUCKET_API_KEY"]="myapikey"base_url="https://myserver/bitbucket"project_key="mykey"BitbucketReader=download_loader("BitbucketReader")loader=BitbucketReader(base_url=base_url,project_key=project_key,branch="refs/heads/develop",repository="ms-messaging",)documents=loader.load_data()index=VectorStoreIndex.from_documents(documents)This loader is designed to be used as a way to load data intoLlama Indexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.For a step-by-step guide, checkout thistutorial
Package: llama-index-readers-boarddocs
BoardDocs LoaderThis loader retrieves an agenda and associated material from a BoardDocs site.This loader is not endorsed by, developed by, supported by, or in any way formally affiliated with Diligent Corporation.UsageTo use this loader, you'll need to specify which BoardDocs site you want to load,
as well as the committee on the site you want to scrape.fromllama_indeximportdownload_loaderBoardDocsReader=download_loader("BoardDocsReader")# For a site URL https://go.boarddocs.com/ca/redwood/Board.nsf/Public# your site should be set to 'ca/redwood'# You'll also need to specify which committee on the site you want to index,# in this case A4EP6J588C05 is the Board of Trustees meeting.loader=BoardDocsReader(site="ca/redwood",committee_id="A4EP6J588C05")# You can optionally specify to load a specific set of meetings; if you don't# pass in meeting_ids, the loader will attempt to load *all* meeting content.# Since we're actually scraping a site, this can take a little while.documents=loader.load_data(meeting_ids=["CPSNV9612DF1"])This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-couchbase
LlamaIndex Readers Integration: CouchbaseThis loader loads documents from Couchbase cluster.
The user specifies a Couchbase client or credentials to initialize the reader. They can specify the SQL++ query to
fetch the relevant docs.UsageHere's an example usage of the CouchbaseReader.fromllama_index.core.readersimportdownload_loaderimportosCouchbaseLoader=download_loader("CouchbaseReader")connection_string=("couchbase://localhost"# valid Couchbase connection string)db_username="<valid_database_user_with_read_access_to_bucket_with_data>"db_password="<password_for_database_user>"# query is a valid SQL++ query that is passed to client.query()query="""SELECT h.* FROM `travel-sample`.inventory.hotel hWHERE h.country = 'United States'LIMIT 5"""reader=CouchbaseLoader(connection_string=connection_string,db_username=db_username,db_password=db_password,)# It is also possible to pass an initialized Couchbase client to the document loader# from couchbase.auth import PasswordAuthenticator  # noqa: E402# from couchbase.cluster import Cluster # noqa: E402# from couchbase.options import ClusterOptions # noqa: E402# auth = PasswordAuthenticator(#    db_username,#    db_password,# )# couchbase_client = Cluster(connection_string, ClusterOptions(auth))# reader = CouchbaseLoader(client=couchbase_client)# fields to be written to the documenttext_fields=["name","title","address","reviews"]# metadata fields to be written to the document's metadatametadata_fields=(["country","city"],)documents=reader.load_data(query=query,text_fields=text_fields,metadata_fields=metadata_fields)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-couchdb
CouchDB LoaderThis loader loads documents from CouchDB. The loader currently supports CouchDB 3.x
using the CouchDB3 python wrapper fromhttps://github.com/n-vlahovic/couchdb3The user specifies a CouchDB instance to initialize the reader. They then specify
the database name and query params to fetch the relevant docs.UsageHere's an example usage of the SimpleCouchDBReader.fromllama_indeximportdownload_loaderimportosSimpleCouchDBReader=download_loader("SimpleCouchDBReader")host="<host>"port="<port>"db_name="<db_name>"# query is passed into db.find()query_str="{ couchdb_find_sytax_json }"reader=SimpleCouchDBReader(host,port)documents=reader.load_data(db_name,query=query_str)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-dad-jokes
DadJoke LoaderThis loader fetches a joke from icanhazdadjoke.UsageTo use this loader, load it.fromllama_indeximportdownload_loaderDadJokesReader=download_loader("DadJokesReader")loader=DadJokesReader()documents=loader.load_data()This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-discord
Discord LoaderThis loader loads conversations from Discord. The user specifieschannel_idsand we fetch conversations from
thosechannel_ids.UsageHere's an example usage of the DiscordReader.fromllama_indeximportdownload_loaderimportosDiscordReader=download_loader("DiscordReader")discord_token=os.getenv("DISCORD_TOKEN")channel_ids=[1057178784895348746]# Replace with your channel_idreader=DiscordReader(discord_token=discord_token)documents=reader.load_data(channel_ids=channel_ids)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-earnings-call-transcript
EARNING CALL TRANSCRIPTS LOADERThis loader fetches the earning call transcripts of US based companies from the websitediscountingcashflows.com. It is not available for commercial purposesInstall the required dependenciespip install -r requirements.txtThe Earning call transcripts takes in three argumentsYearTicker symbolQuarter name from the list ["Q1","Q2","Q3","Q4"]Usagefromllama_indeximportdownload_loaderIMDBReviewsloader=download_loader("EarningsCallTranscript")loader=EarningsCallTranscript(2023,"AAPL","Q3")docs=loader.load_data()The metadata of the transcripts are the followingtickerquarterdate_timespeakers_listExamplesLlama Indexfromllama_indeximportdownload_loaderfromllama_indeximportVectorStoreIndex,download_loaderEarningsCallTranscript=download_loader("EarningsCallTranscript")loader=EarningsCallTranscript(2023,"AAPL","Q3")docs=loader.load_data()index=VectorStoreIndex.from_documents(documents)query_engine=index.as_query_engine()response=query_engine.query("What was discussed about Generative AI?",)print(response)Langchainfromllama_indeximportdownload_loaderfromlangchain.agentsimportToolfromlangchain.agentsimportinitialize_agentfromlangchain.chat_modelsimportChatOpenAIfromlangchain.llmsimportOpenAIEarningsCallTranscript=download_loader("EarningsCallTranscript")loader=EarningsCallTranscript(2023,"AAPL","Q3")docs=loader.load_data()tools=[Tool(name="LlamaIndex",func=lambdaq:str(index.as_query_engine().query(q)),description="useful for questions about investor transcripts calls for a company. The input to this tool should be a complete english sentence.",return_direct=True,),]llm=ChatOpenAI(temperature=0)agent=initialize_agent(tools,llm,agent="conversational-react-description")agent.run("What was discussed about Generative AI?")
Package: llama-index-readers-feedly-rss
Feedly LoaderThis loader fetches the entries from a list of RSS feeds subscribed inFeedly. You must initialize the loader with yourFeedly API token, and then pass the category name which you want to extract.Usagefromllama_indeximportdownload_loaderfeedlyRssReader=download_loader("FeedlyRssReader")loader=feedlyRssReader(bearer_token="[YOUR_TOKEN]")documents=loader.load_data(category_name="news",max_count=100)Dependenciesfeedly-client
Package: llama-index-readers-feishu-docs
Feishu Doc LoaderThis loader takes in IDs of Feishu Docs and parses their text intodocuments. You can extract a Feishu Doc's ID directly from its URL. For example, the ID ofhttps://test-csl481dfkgqf.feishu.cn/docx/HIH2dHv21ox9kVxjRuwc1W0jnkfisHIH2dHv21ox9kVxjRuwc1W0jnkf. As a prerequisite, you will need to register with Feishu and build an custom app. Seeherefor instructions.UsageTo use this loader, you simply need to pass in an array of Feishu Doc IDs. The default API endpoints are for Feishu, in order to switch to Lark, we should useset_lark_domain.fromllama_indeximportdownload_loaderapp_id="cli_slkdjalasdkjasd"app_secret="dskLLdkasdjlasdKK"doc_ids=["HIH2dHv21ox9kVxjRuwc1W0jnkf"]FeishuDocsReader=download_loader("FeishuDocsReader")loader=FeishuDocsReader(app_id,app_secret)documents=loader.load_data(document_ids=doc_ids)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-firebase-realtimedb
Firebase Realtime Database LoaderThis loader retrieves documents from Firebase Realtime Database. The user specifies the Firebase Realtime Database URL and, optionally, the path to a service account key file for authentication.UsageHere's an example usage of the FirebaseRealtimeDatabaseReader.fromllama_indeximportdownload_loaderFirebaseRealtimeDatabaseReader=download_loader("FirebaseRealtimeDatabaseReader")database_url="<database_url>"service_account_key_path="<service_account_key_path>"path="<path>"reader=FirebaseRealtimeDatabaseReader(database_url,service_account_key_path)documents=reader.load_data(path)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-firestore
Firestore LoaderThis loader loads from a Firestore collection or a specific document from Firestore. The loader assumes your project already has the google cloud credentials loaded. To find out how to set up credentials,see here.UsageTo initialize the loader, provide the project-id of the google cloud project.Initializing the readerfromllama_indeximportdownload_loaderFirestoreReader=download_loader("FirestoreReader")reader=FirestoreReader(project_id="<Your Project ID>")Loading Data from a Firestore CollectionLoad data from a Firestore collection with the load_data method:
The collection path should include all previous documents and collections if it is a nested collection.documents=reader.load_data(collection="foo/bar/abc/")Loading a Single Document from FirestoreLoad a single document from Firestore with the load_document method:document=reader.load_document(document_url="foo/bar/abc/MY_DOCUMENT")Note: load_data returns a list of Document objects, whereas load_document returns a single Document object.This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-genius
LlamaIndex Readers Integration: GeniusThis loader connects to the Genius API and loads lyrics, metadata, and album art intoDocuments.As a prerequisite, you will need to register withGenius APIand create an app in order to get aclient_idand aclient_secret. You should then set aredirect_urifor the app. Theredirect_uridoes not need to be functional. You should then generate an access token as an instantiator for the GeniusReader.UsageHere's an example usage of the GeniusReader. It will retrieve songs that match specific lyrics. Acceptable arguments are lyrics (str): The lyric snippet you're looking for and will return List[Document]: A list of documents containing songs with those lyrics.GeniusReader Class Methodsload_artist_songsDescription: Fetches all or a specified number of songs by an artist.Arguments:artist_name(str): The name of the artist.max_songs(Optional[int]): Maximum number of songs to retrieve.Returns: List ofDocumentobjects with song lyrics.load_all_artist_songsDescription: Fetches all songs of an artist and saves their lyrics.Arguments:artist_name(str): The name of the artist.Returns: List ofDocumentobjects with the artist's song lyrics.load_artist_songs_with_filtersDescription: Loads the most or least popular song of an artist based on filters.Arguments:artist_name(str): The artist's name.most_popular(bool):Truefor most popular song,Falsefor least popular.max_songs(Optional[int]): Max number of songs to consider for popularity.max_pages(int): Max number of pages to fetch.Returns:Documentwith lyrics of the selected song.load_song_by_url_or_idDescription: Loads a song by its Genius URL or ID.Arguments:song_url(Optional[str]): URL of the song on Genius.song_id(Optional[int]): ID of the song on Genius.Returns: List ofDocumentobjects with the song's lyrics.search_songs_by_lyricsDescription: Searches for songs by a snippet of lyrics.Arguments:lyrics(str): Lyric snippet to search for.Returns: List ofDocumentobjects with songs matching the lyrics.load_songs_by_tagDescription: Loads songs by a specific tag or genre.Arguments:tag(str): Tag or genre to search for.max_songs(Optional[int]): Max number of songs to fetch.max_pages(int): Max number of pages to fetch.Returns: List ofDocumentobjects with song lyrics.fromllama_index.core.readersimportdownload_loaderGeniusReader=download_loader("GeniusReader")access_token="your_generated_access_token"loader=GeniusReader(access_token)documents=loader.search_songs_by_lyrics("Imagine")ExampleThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent.LlamaIndexfromllama_index.coreimportVectorStoreIndex,download_loaderGeniusReader=download_loader("GeniusReader")access_token="your_generated_access_token"loader=GeniusReader(access_token)documents=loader.search_songs_by_lyrics("Imagine")index=VectorStoreIndex.from_documents(documents)index.query("What artists have written songs that have the lyrics imagine in them?")
Package: llama-index-readers-gpt-repo
GPT Repository LoaderThis loader is an adaptation ofhttps://github.com/mpoon/gpt-repository-loaderto LlamaHub. Full credit goes to mpoon for coming up with this!UsageTo use this loader, you need to pass in a path to a local Git repositoryfromllama_indeximportdownload_loaderGPTRepoReader=download_loader("GPTRepoReader")loader=GPTRepoReader()documents=loader.load_data(repo_path="/path/to/git/repo",preamble_str="<text to put at beginning of Document>",)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-graphdb-cypher
Graph Database Cypher LoaderThis loader populates documents from results of Cypher queries from a Graph database endpoint.
The user specifies a GraphDB endpoint URL with optional credentials to initialize the reader.
By declaring the Cypher query and optional parameters the loader can fetch the nested result docs.
The results will be turned into a yaml representation to be turned into a string for the document.The approach should work for Neo4j, AWS Neptune and Memgraph.UsageHere's an example usage of theGraphDBCypherReader.You can test out queries directly with the Neo4j labs demo server: demo.neo4jlabs.com or with a free instancehttps://neo4j.com/aurafromllama_indeximportdownload_loaderimportosGraphDBCypherReader=download_loader("GraphDBCypherReader")uri="neo4j+s://demo.neo4jlabs.com"username="stackoverflow"password="stackoverflow"database="stackoverflow"query="""MATCH (q:Question)-[:TAGGED]->(:Tag {name:$tag})RETURN q.title as titleORDER BY q.createdAt DESC LIMIT 10"""reader=GraphDBCypherReader(uri,username,password,database)documents=reader.load_data(query,parameters={"tag":"lua"})This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent.
Seeherefor examples.It uses theNeo4j Graph Databasefor the Cypher queries.
Package: llama-index-readers-graphql
GraphQL LoaderThis loader loads documents via GraphQL queries from a GraphQL endpoint.
The user specifies a GraphQL endpoint URL with optional credentials to initialize the reader.
By declaring the GraphQL query and optional variables (parameters) the loader can fetch the nested result docs.UsageHere's an example usage of the GraphQLReader.
You can test out queries directlyon the sitefromllama_indeximportdownload_loaderimportosGraphQLReader=download_loader("GraphQLReader")uri="https://countries.trevorblades.com/"headers={}query="""query getContinents {continents {codename}}"""reader=GraphQLReader(uri,headers)documents=reader.query(query,variables={})This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent.
Seeherefor examples.It uses thegql GraphQL libraryfor the GraphQL queries.
Package: llama-index-readers-guru
Guru LoaderThis loader loads documents fromGuru. The user specifies a username and api key to initialize the GuruReader.Note this is not your password. You need to create a new api key in the admin tab of the portal.UsageHere's an example usage of the GuruReader.fromllama_indeximportdownload_loaderGuruReader=download_loader("GuruReader")reader=GuruReader(username="<GURU_USERNAME>",api_key="<GURU_API_KEY>")# Load all documents in a collectiondocuments=reader.load_data(collection_ids=["<COLLECTION_ID_1>","<COLLECTION_ID_2>"])# Load specific cards by card iddocuments=reader.load_data(card_ids=["<CARD_ID_1>","<CARD_ID_2>"])This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-hatena-blog
Hatena Blog LoaderThis loader fetches article from your ownHatena Blogblog posts using the AtomPub API.You can get AtomPub info from the admin page after logging into Hatena Blog.UsageHere's an example usage of the HatenaBlogReader.fromllama_indeximportdownload_loaderimportosHatenaBlogReader=download_loader("HatenaBlogReader")root_endpoint=os.getenv("ATOM_PUB_ROOT_ENDPOINT")api_key=os.getenv("ATOM_PUB_API_KEY")username=os.getenv("HATENA_BLOG_USERNAME")reader=HatenaBlogReader(root_endpoint=root_endpoint,api_key=api_key,username=username)documents=reader.load_data()This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.