Saravanan666/pypi-org huggingface dataset:
Package: llama-index-readers-spotify
Spotify LoaderThis loader reads your Spotify account and loads saved albums, tracks, or playlists intoDocuments.As a prerequisite, you will need to register withSpotify for Developersand create an app in order to get aclient_idand aclient_secret. You should then set aredirect_urifor the app (in the web dashboard under app settings). Theredirect_uridoes not need to be functional. You should then set theclient_id,client_secret, andredirect_urias environmental variables.export SPOTIPY_CLIENT_ID='xxxxxxxxxxxxxxxxx'export SPOTIPY_CLIENT_SECRET='xxxxxxxxxxxxxxxxxx'export SPOTIPY_REDIRECT_URI='http://localhost:8080/redirect'UsageHere's an example usage of the SpotifyReader. It will retrieve your saved albums, unless an optionalcollectionargument is passed. Acceptable arguments are "albums", "tracks", and "playlists".fromllama_indeximportdownload_loaderSpotifyReader=download_loader("SpotifyReader")loader=SpotifyReader()documents=loader.load_data()ExampleThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent.LlamaIndexfromllama_indeximportVectorStoreIndex,download_loaderSpotifyReader=download_loader("SpotifyReader")loader=SpotifyReader()documents=loader.load_data()index=VectorStoreIndex.from_documents(documents)index.query("When are some other artists i might like based on what i listen to ?")
Package: llama-index-readers-stackoverflow
StackoverflowReader (In Beta)Using the Stackoverflow API, this class will read the Stackoverflow Teams API and return a list of questions and answers based on posts.It also supports caching the results to a local directory, so that you can run the load_data() method multiple times without hitting the API.getting a tokenVisit:https://stackoverflowteams.com/users/pats/Click Create a new PATName the token, and pick the team scopeSelect an expiration dateClick CreateAdd this to your env, or to the instantiation of theStackoverflowReader(pa_token, team_name, cache_dir='./stackoverflow_cache')exportSTACKOVERFLOW_PAT=your_tokenexportSTACKOVERFLOW_TEAM_NAME=your_teamOther features which could be added:Add articlesAdd commentsAdd tagsAdd usersAdd votesAdd badges
Package: llama-index-readers-stripe-docs
StripeDocs LoaderThis loader asynchronously loads data from theStripe documentation. It iterates through the Stripe sitemap to get all/docsreferences.It is based on theAsync Website Loader.Usagefromllama_indeximportVectorStoreIndexfromllama_hub.stripe_docsimportStripeDocsReaderloader=StripeDocsReader()documents=loader.load_data()index=VectorStoreIndex.from_documents(documents)query_engine=index.as_query_engine()query_engine.query("How do I accept payments on my website?")TheStripeDocsReaderallows you to return plain text docs by settinghtml_to_text=True. You can also adjust the maximum concurrent requests by settinglimit=10.FilteringYou can filter pages from the Stripe sitemap by adding thefiltersargument to the load_data method. This allows you to control what pages from the Stripe website, including documentation, will be loaded.The default filters are set to["/docs"]to scope everything to docs only.documents=loader.load_data(filters=["/terminal"])
Package: llama-index-readers-telegram
Telegram LoaderThis loader fetches posts/chat messages/comments from Telegram channels or chats intoDocuments.Before working with Telegram’s API, you need to get your own API ID and hash:Login to your Telegram accountwith the phone number of the developer account to use.Click under API Development tools.A Create new application window will appear. Fill in your application details. There is no need to enter any URL, and only the first two fields (App title and Short name) can currently be changed later.Click on Create application at the end. Remember that your API hash is secret and Telegram won’t let you revoke it. Don’t post it anywhere!This API ID and hash is the one used by your application, not your phone number. You can use this API ID and hash with any phone number.UsageThe first parameter you pass to the constructor of the TelegramReader is the session_name, and defaults to be the session name (or full path). That is, if you create a TelegramReader(session_name='anon', ...) instance and run load_data(), ananon.sessionfile will be created in the working directory where you run this loader.The Auth procedure asks for:Security CodePasswordPleaseenterthecodeyoureceived:12345Pleaseenteryourpassword:*******(Youarenowloggedin)If the.sessionfile already existed, it will not login again, so be aware of this if you move or rename the file! Seeherefor more instructions.To use this loader, you simply need to pass in a entity name.fromllama_index.coreimportdownload_loaderTelegramReader=download_loader("TelegramReader")loader=TelegramReader(session_name="[YOUR_SESSION_NAME]",api_id="[YOUR_API_ID]",api_hash="[YOUR_API_HASH]",phone_number="[YOUR_PHONE_NUMBER]",)documents=loader.load_data(entity_name="https://t.me/[ENTITY_NAME]",limit=100)ExamplesThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent.
Package: llama-index-readers-trello
Trello LoaderThis loader loads documents from Trello. The user specifies an API key and API token to initialize the TrelloReader. They then specify a board_id to
load in the corresponding Document objects representing Trello cards.UsageHere's an example usage of the TrelloReader.fromllama_indeximportdownload_loaderimportosTrelloReader=download_loader("TrelloReader")reader=TrelloReader("<Trello_API_KEY>","<Trello_API_TOKEN>")documents=reader.load_data(board_id="<BOARD_ID>")This loader is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for
examples.
Package: llama-index-readers-weather
Weather LoaderThis loader fetches the weather data from theOpenWeatherMap's OneCall API, using thepyowmPython package. You must initialize the loader with your OpenWeatherMap API token, and then pass in the names of the cities you want the weather data for.OWM's One Call API provides the following weather data for any geographical coordinate: - Current weather - Hourly forecast for 48 hours - Daily forecast for 7 daysUsageTo use this loader, you need to pass in an array of city names (eg. [chennai, chicago]). Pass in the country codes as well for better accuracy.fromllama_indeximportdownload_loaderWeatherReader=download_loader("WeatherReader")loader=WeatherReader(token="[YOUR_TOKEN]")documents=loader.load_data(places=["Chennai, IN","Dublin, IE"])This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-whatsapp
Whatsapp chat loaderExport a Whatsapp chatOpen a chatTap on the menu > More > Export chatSelectWithout mediaSave the.txtfile in your working directoryFor more info seeWhatsapp's Help CenterUsageMessages will get saved in the format:{timestamp} {author}: {message}. Useful for when you want to ask about specific people in a group chat.Metadata automatically included:source(file name),authorandtimestamp.frompathlibimportPathfromllama_indeximportdownload_loaderWhatsappChatLoader=download_loader("WhatsappChatLoader")path="whatsapp.txt"loader=WhatsappChatLoader(path=path)documents=loader.load_data()# see what's createddocuments[0]# >>> Document(text='2023-02-20 00:00:00 ur mom: Hi 😊', doc_id='e0a7c508-4ba0-48e1-a2ba-9af133225636', embedding=None, extra_info={'source': 'WhatsApp Chat with ur mom', 'author': 'ur mom', 'date': '2023-02-20 00:00:00'})This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-wordlift
WordLift ReaderThe WordLift GraphQL Reader is a connector to fetch and transform data from a WordLift Knowledge Graph using your the WordLift Key. The connector provides a convenient way to load data from WordLift using a GraphQL query and transform it into a list of documents for further processing.UsageTo use the WordLift GraphQL Reader, follow the steps below:Set up the necessary configuration options, such as the API endpoint, headers, query, fields, and configuration options (make sure you have with you theWordlift Key).Create an instance of theWordLiftLoaderclass, passing in the configuration options.Use theload_datamethod to fetch and transform the data.Process the loaded documents as needed.Here's an example of how to use the WordLift GraphQL Reader:importjsonfromllama_indeximportVectorStoreIndexfromllama_index.readers.schemaimportDocumentfromlangchain.llmsimportOpenAIfromllama_hub.wordliftimportWordLiftLoader# Set up the necessary configuration optionsendpoint="https://api.wordlift.io/graphql"headers={"Authorization":"<YOUR_WORDLIFT_KEY>","Content-Type":"application/json",}query="""# Your GraphQL query here"""fields="<YOUR_FIELDS>"config_options={"text_fields":["<YOUR_TEXT_FIELDS>"],"metadata_fields":["<YOUR_METADATA_FIELDS>"],}# Create an instance of the WordLiftLoaderreader=WordLiftLoader(endpoint,headers,query,fields,config_options)# Load the datadocuments=reader.load_data()# Convert the documentsconverted_doc=[]fordocindocuments:converted_doc_id=json.dumps(doc.doc_id)converted_doc.append(Document(text=doc.text,doc_id=converted_doc_id,embedding=doc.embedding,doc_hash=doc.doc_hash,extra_info=doc.extra_info,))# Create the index and query engineindex=VectorStoreIndex.from_documents(converted_doc)query_engine=index.as_query_engine()# Perform a queryresult=query_engine.query("<YOUR_QUERY>")# Process the result as neededlogging.info("Result:%s",result)This loader is designed to be used as a way to load data from WordLift KGs intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent.
Package: llama-index-readers-wordpress
Wordpress LoaderThis loader fetches the text from Wordpress blog posts using the Wordpress API. It also uses the BeautifulSoup library to parse the HTML and extract the text from the articles.UsageTo use this loader, you need to pass base url of the Wordpress installation (e.g.https://www.mysite.com), a username, and an application password for the user (more about application passwordshere)fromllama_indeximportdownload_loaderWordpressReader=download_loader("WordpressReader")loader=WordpressReader(url="https://www.mysite.com",username="my_username",password="my_password",)documents=loader.load_data()This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-youtube-transcript
Youtube Transcript LoaderThis loader fetches the text transcript of Youtube videos using theyoutube_transcript_apiPython package.UsageTo use this loader, you will need to firstpip install youtube_transcript_api.Then, simply pass an array of YouTube links intoload_data:fromllama_hub.youtube_transcriptimportYoutubeTranscriptReaderloader=YoutubeTranscriptReader()documents=loader.load_data(ytlinks=["https://www.youtube.com/watch?v=i3OYlaoj-BM"])Supported URL formats: + youtube.com/watch?v={video_id} (with or without 'www.') + youtube.com/embed?v={video_id} (with or without 'www.') + youtu.be/{video_id} (never includes www subdomain)To programmatically check if a URL is supported:fromllama_hub.youtube_transcriptimportis_youtube_videois_youtube_video("https://youtube.com/watch?v=j83jrh2")# => Trueis_youtube_video("https://vimeo.com/272134160")# => FalseThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-zendesk
Zendesk LoaderThis loader fetches the text from Zendesk help articles using the Zendesk API. It also uses the BeautifulSoup library to parse the HTML and extract the text from the articles.UsageTo use this loader, you need to pass in the subdomain of a Zendesk account. No authentication is required. You can also set the locale of articles as needed.fromllama_indeximportdownload_loaderZendeskReader=download_loader("ZendeskReader")loader=ZendeskReader(zendesk_subdomain="my_subdomain",locale="en-us")documents=loader.load_data()This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-zep
Zep ReaderThe Zep Reader returns a set of texts corresponding to a text query or embeddings retrieved from a Zep Collection.
The Reader is initialized with a Zep API URL and optionally an API key. The Reader can then be used to load data
from a Zep Document Collection.About ZepZep is a long-term memory store for LLM applications. Zep makes it simple to add relevant documents, chat history memory
and rich user data to your LLM app's prompts.For more information about Zep and the Zep Quick Start Guide, see theZep documentation.UsageHere's an end-to-end example usage of the ZepReader. First, we create a Zep Collection, chunk a document,
and add it to the collection.We then wait for Zep's async embedder to embed the document chunks. Finally, we query the collection and print the
results.importtimefromuuidimportuuid4fromllama_index.node_parserimportSimpleNodeParserfromllama_index.readers.schemaimportDocumentfromzep_pythonimportZepClientfromzep_python.documentimportDocumentasZepDocumentfromllama_indeximportdownload_loaderZepReader=download_loader("ZepReader")# Create a Zep collectionzep_api_url="http://localhost:8000"# replace with your Zep API URLcollection_name=f"babbage{uuid4().hex}"file="babbages_calculating_engine.txt"print(f"Creating collection{collection_name}")client=ZepClient(base_url=zep_api_url,api_key="optional_api_key")collection=client.document.add_collection(name=collection_name,# requireddescription="Babbage's Calculating Engine",# optionalmetadata={"foo":"bar"},# optional metadataembedding_dimensions=1536,# this must match the model you've configured in Zepis_auto_embedded=True,# use Zep's built-in embedder. Defaults to True)node_parser=SimpleNodeParser.from_defaults(chunk_size=250,chunk_overlap=20)withopen(file)asf:raw_text=f.read()print("Splitting text into chunks and adding them to the Zep vector store.")docs=node_parser.get_nodes_from_documents([Document(text=raw_text)],show_progress=True)# Convert nodes to ZepDocumentzep_docs=[ZepDocument(content=d.get_content())fordindocs]uuids=collection.add_documents(zep_docs)print(f"Added{len(uuids)}documents to collection{collection_name}")print("Waiting for documents to be embedded")whileTrue:c=client.document.get_collection(collection_name)print("Embedding status: "f"{c.document_embedded_count}/{c.document_count}documents embedded")time.sleep(1)ifc.status=="ready":breakquery="Was Babbage awarded a medal?"# Using the ZepReader to load data from Zepreader=ZepReader(api_url=zep_api_url,api_key="optional_api_key")results=reader.load_data(collection_name=collection_name,query=query,top_k=3)print("\n\n".join([r.textforrinresults]))
Package: llama-index-readers-zulip
Zulip LoaderThe Zulip Loader is a Python script that allows you to load data from Zulip streams using a Zulip bot's API token. It fetches messages from specified streams or all streams if none are specified, and returns a list of documents with the stream content.PrerequisitesCreate a Zulip bot and obtain its API token. Follow the instructions in the Zulip documentation to create a bot and get the API key (token).Set the ZULIP_TOKEN environment variable to your Zulip bot's API token:exportZULIP_TOKEN="your-zulip-bot-api-token"Use the ZulipReader class to load data from Zulip streams:fromzulip_loaderimportZulipReader# Initialize the ZulipReader with the bot's email and Zulip domainreader=ZulipReader(zulip_email="your-bot-email@your-zulip-domain.zulipchat.com",zulip_domain="your-zulip-domain.zulipchat.com",)# Load data from all streamsdata=reader.load_data(reader.get_all_streams())# Load data from specific streamsstream_names=["stream1","stream2"]data=reader.load_data(stream_names)# This will return a list of documents containing the content of the specified streams.For more customization, you can pass thereverse_chronologicalparameter to the load_data() method to indicate the order of messages in the output.
Package: llama-index-tools-arxiv
ArXiv Search ToolThis tool connects to ArXiv and allows an Agent to search for recent papers and their summaries to retrieve recent information on mathematical and scientific informationUsageThis tool has a more extensive example usage documented in a Jupyter notebookhereHere's an example usage of the ArxivToolSpec.fromllama_index.tools.arxivimportArxivToolSpecfromllama_index.agentimportOpenAIAgenttool_spec=ArxivToolSpec()agent=OpenAIAgent.from_tools(tool_spec.to_tool_list())agent.chat("What's going on with the superconductor lk-99")agent.chat("what are the latest developments in machine learning")arxiv_query: Search arXiv for results related to the queryThis loader is designed to be used as a way to load data as a Tool in a Agent. Seeherefor examples.
Package: llama-index-tools-azure-cv
Azure Computer Vision ToolThis tool connects to a Azure account and allows an Agent to perform a variety of computer vision tasks on image urls.You will need to set up an api key and computer vision instance using Azure, learn more here:https://azure.microsoft.com/en-ca/products/cognitive-services/computer-visionUsageThis tool has a more extensive example usage documented in a Jupyter notebookhereHere's an example usage of the AzureCVToolSpec.fromllama_index.tools.azure_cvimportAzureCVToolSpecfromllama_index.agentimportOpenAIAgenttool_spec=AzureCVToolSpec(api_key="your-key",resource="your-resource")agent=OpenAIAgent.from_tools(tool_spec.to_tool_list())agent.chat("caption this image and tell me what tags are in it https://portal.vision.cognitive.azure.com/dist/assets/ImageCaptioningSample1-bbe41ac5.png")agent.chat("caption this image and read any text https://portal.vision.cognitive.azure.com/dist/assets/OCR3-4782f088.jpg")process_image: Send an image for computer vision classification of objects, tags, captioning or OCR.This loader is designed to be used as a way to load data as a Tool in a Agent. Seeherefor examples.
Package: llama-index-tools-azure-speech
Azure Speech ToolThis tool allows Agents to use Microsoft Azure speech services to transcribe audio files to text, and create audio files from text. To see more and get started, visithttps://azure.microsoft.com/en-us/products/ai-services/ai-speechUsageThis tool has a more extensive example usage documented in a Jupyter notebookherefromllama_index.tools.azure_speechimportAzureSpeechToolSpecfromllama_index.agentimportOpenAIAgentspeech_tool=AzureSpeechToolSpec(speech_key="your-key",region="eastus")agent=OpenAIAgent.from_tools(speech_tool.to_tool_list(),verbose=True,)print(agent.chat('Say "hello world"'))print(agent.chat("summarize the data/speech.wav audio file into a few sentences"))text_to_speech: Takes an input string and synthesizes audio to play on the users computerspeech_to_text: Takes a .wav file and transcribes it into textThis loader is designed to be used as a way to load data as a Tool in a Agent. Seeherefor examples.
Package: llama-index-tools-azure-translate
Azure Translate ToolThis tool connects to a Azure account and allows an Agent to perform text translation into a variet of different languagesYou will need to set up an api key and translate instance using Azure, learn more here:https://learn.microsoft.com/en-us/azure/ai-services/translator/translator-overviewFor a full list of supported languages see here:https://learn.microsoft.com/en-us/azure/ai-services/translator/language-supportThis tool has a more extensive example usage documented in a Jupyter notebookhereUsageHere's an example usage of the AzureTranslateToolSpec.fromllama_index.agentimportOpenAIAgentfromllama_index.tools.azure_translateimportAzureTranslateToolSpectranslate_tool=AzureTranslateToolSpec(api_key="your-key",region="eastus")agent=OpenAIAgent.from_tools(translate_tool.to_tool_list(),verbose=True,)print(agent.chat('Say "hello world" in 5 different languages'))translate: Translate text to a target languageThis loader is designed to be used as a way to load data as a Tool in a Agent. Seeherefor examples.
Package: llama-index-tools-bing-search
Bing Search ToolThis tool connects to a Bing account and allows an Agent to perform searches for news, images and videos.You will need to set up a search key using Azure,learn more here:https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/overviewUsageThis tool has a more extensive example usage documented in a Jupyter notebookhereHere's an example usage of the BingSearchToolSpec.fromllama_index.tools.bing_searchimportBingSearchToolSpecfromllama_index.agentimportOpenAIAgenttool_spec=BingSearchToolSpec(api_key="your-key")agent=OpenAIAgent.from_tools(tool_spec.to_tool_list())agent.chat("what's the latest news about superconductors")agent.chat("what does lk-99 look like")agent.chat("is there any videos of it levitating")bing_news_search: Search for news results related to a querybing_image_search: Search for images related to a querybing_video_search: Search for videos related to a queryThis loader is designed to be used as a way to load data as a Tool in a Agent. Seeherefor examples.
Package: llama-index-tools-chatgpt-plugin
ChatGPT Plugin ToolThis tool allows Agents to load a plugin using a ChatGPT manifest file, and have the Agent interact with it.UsageThis tool has more extensive example usage documented in a Jupyter notebookhere# Load the manifestimportrequestsimportyamlf=requests.get("https://raw.githubusercontent.com/sisbell/chatgpt-plugin-store/main/manifests/today-currency-converter.oiconma.repl.co.json").textmanifest=yaml.safe_load(f)fromllama_index.tools.chatgpt_pluginimportChatGPTPluginToolSpecfromllama_index.agentimportOpenAIAgentfromllama_index.tools.requestsimportRequestsToolSpecrequests_spec=RequestsToolSpec()plugin_spec=ChatGPTPluginToolSpec(manifest)# ORplugin_spec=ChatGPTPluginToolSpec(manifest_url="https://raw.githubusercontent.com/sisbell/chatgpt-plugin-store/main/manifests/today-currency-converter.oiconma.repl.co.json")agent=OpenAIAgent.from_tools([*plugin_spec.to_tool_list(),*requests_spec.to_tool_list()],verbose=True)print(agent.chat("Convert 100 euros to CAD"))describe_plugin: Describe the plugin that has been loaded.load_openapi_spec: Returns the parsed OpenAPI spec that the class was initialized withIn addition to the above method, this tool makes all of the tools available from the OpenAPI Tool Spec and Requests Tool Spec available to the agent. The plugin OpenAPI definition is loaded into the OpenAPI tool spec, and authentication headers are passed in to the Requests tool specThis loader is designed to be used as a way to load data as a Tool in a Agent. Seeherefor examples.
Package: llama-index-tools-code-interpreter
Code Interpreter ToolThis tool can be used to run python scripts and capture the results of stdout and stderrWARNING: This tool provides the Agent access to thesubprocess.runcommand.
Arbitrary code execution is possible on the machine running this tool.
This tool is not recommended to be used in a production setting, and would require heavy sandboxing or virtual machinesUsageHere's an example usage of the CodeInterpreterToolSpec.fromllama_index.tools.code_interpreterimportCodeInterpreterToolSpecfromllama_index.agentimportOpenAIAgentcode_spec=CodeInterpreterToolSpec()agent=OpenAIAgent.from_tools(code_spec.to_tool_list())# Prime the agent to use the toolagent.chat("Can you help me write some python code to pass to the code_interpreter tool")agent.chat("write a python function to calculate volume of a sphere with radius 4.3cm")The tools available are:code_interpreter: A tool to evaluate a python scriptThis loader is designed to be used as a way to load data as a Tool in a Agent. Seeherefor examples.