Saravanan666/pypi-org huggingface dataset:
Package: llama-index-readers-opensearch
Opensearch LoaderThe Opensearch Loader returns a set of texts corresponding to documents retrieved from an Opensearch index.
The user initializes the loader with an Opensearch index. They then pass in a field, and optionally a JSON query DSL object to fetch the fields they want.UsageHere's an example usage of the OpensearchReader to load 100 documents.fromllama_indeximportdownload_loaderOpensearchReader=download_loader("OpensearchReader")reader=OpensearchReader(host="localhost",port=9200,index="<index_name>",basic_auth=("<user_name>","<password>"),)query={"size":100,"query":{"match_all":{}}}documents=reader.load_data("<field_name>",query=query,embedding_field="field_name")This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-pandas-ai
Pandas AI LoaderThis loader is a light wrapper around thePandasAIPython package.See here:https://github.com/gventuri/pandas-ai.You can directly get the result ofpandasai.runcommand, or
you can choose to load inDocumentobjects viaload_data.Usagefromllama_indeximportdownload_loaderfrompandasai.llm.openaiimportOpenAIimportpandasaspd# Sample DataFramedf=pd.DataFrame({"country":["United States","United Kingdom","France","Germany","Italy","Spain","Canada","Australia","Japan","China",],"gdp":[21400000,2940000,2830000,3870000,2160000,1350000,1780000,1320000,516000,14000000,],"happiness_index":[7.3,7.2,6.5,7.0,6.0,6.3,7.3,7.3,5.9,5.0],})llm=OpenAI()PandasAIReader=download_loader("PandasAIReader")# use run_pandas_ai directly# set is_conversational_answer=False to get parsed outputloader=PandasAIReader(llm=llm)response=reader.run_pandas_ai(df,"Which are the 5 happiest countries?",is_conversational_answer=False)print(response)# load data with is_conversational_answer=False# will use our PandasCSVReader under the hooddocs=reader.load_data(df,"Which are the 5 happiest countries?",is_conversational_answer=False)# load data with is_conversational_answer=True# will use our PandasCSVReader under the hooddocs=reader.load_data(df,"Which are the 5 happiest countries?",is_conversational_answer=True)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-papers
Papers LoadersArxiv Papers LoaderThis loader fetches the text from the most relevant scientific papers on Arxiv specified by a search query (e.g. "Artificial Intelligence"). For each paper, the abstract is extracted and put in a separate document. The search query may be any string, Arxiv paper id, or a general Arxiv query string (see the full list of capabilitieshere).UsageTo use this loader, you need to pass in the search query. You may also optionally specify a local directory to temporarily store the paper PDFs (they are deleted automatically) and the maximum number of papers you want to parse for your search query (default is 10).fromllama_indeximportdownload_loaderArxivReader=download_loader("ArxivReader")loader=ArxivReader()documents=loader.load_data(search_query="au:Karpathy")Alternatively, if you would like to load papers and abstracts separately:fromllama_indeximportdownload_loaderArxivReader=download_loader("ArxivReader")loader=ArxivReader()documents,abstracts=loader.load_papers_and_abstracts(search_query="au:Karpathy")This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.Pubmed Papers LoaderThis loader fetches the text from the most relevant scientific papers on Pubmed specified by a search query (e.g. "Alzheimers"). For each paper, the abstract is included in theDocument. The search query may be any string.UsageTo use this loader, you need to pass in the search query. You may also optionally specify the maximum number of papers you want to parse for your search query (default is 10).fromllama_indeximportdownload_loaderPubmedReader=download_loader("PubmedReader")loader=PubmedReader()documents=loader.load_data(search_query="amyloidosis")This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-patentsview
Patentsview LoaderThis loader loads patent abstract froma list of patent numberswith API provided byPatentsview.UsageHere'a an example usage of PatentsviewReader.fromllama_indeximportdownload_loaderPatentsviewReader=download_loader("PatentsviewReader")loader=PatentsviewReader()patents=["8848839","10452978"]abstracts=loader.load_data(patents)This loader is designed for loading data intoLlamaIndex.
Package: llama-index-readers-pdb
Protein Data Bank (PDB) publication LoaderThis loader fetches the abstract of PDB entries using the RCSB (Research Collaboratory for Structural Bioinformatics) or EBI (European Bioinformatics Institute) REST api.UsageTo use this loader, simply pass an array of PDB ids intoload_data:fromllama_hub.pdbimportPdbAbstractReaderloader=PdbAbstractReader()documents=loader.load_data(pdb_id=["1cbs"])This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-pdf-table
PDF Table LoaderThis loader reads the tables included in the PDF.Users can input the PDFfileand thepagesfrom which they want to extract tables, and they can read the tables included on those pages.UsageHere's an example usage of the PDFTableReader.pagesparameter is the same as camelot'spages. Therefore, you can use patterns such asall,1,2,3,10-20, and so on.fromllama_hub.pdf_tableimportPDFTableReaderfrompathlibimportPathreader=PDFTableReader()pdf_path=Path("/path/to/pdf")documents=reader.load_data(file=pdf_path,pages="80-90")ExampleThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent.
Package: llama-index-readers-rayyan
Rayyan LoaderThis loader fetches review articles fromRayyanusing theRayyan SDK. All articles
for a given review are fetched by default unless a filter is specified.UsageTo use this loader, you need to specify the path to the Rayyan credentials file
and optionally the API server URL if different from the default. More details
about these parameters can be found in the official Rayyan SDK repository.fromllama_indeximportdownload_loaderRayyanReader=download_loader("RayyanReader")loader=RayyanReader(credentials_path="path/to/rayyan-creds.json")Once the loader is initialized, you can load data from Rayyan, either all or filtered:# Load all documents for a review with ID 123456documents=loader.load_data(review_id=123456)# Load only those that contain the word "outcome"documents=loader.load_data(review_id=123456,filters={"search[value]":"outcome"})The Rayyan SDK has more information about the available filters.This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-readwise
Readwise ReaderUse Readwise's export API to fetch your highlights from web articles, epubs, pdfs, Kindle, YouTube, and load the resulting text into LLMs.SetupGet your Readwise API key fromreadwise.io/access_token.UsageHere is an example usage of the Readwise Reader:importosfromllama_indeximportVectorStoreIndex,download_loaderReadwiseReader=download_loader("ReadwiseReader")token=os.getenv("READWISE_API_KEY")loader=ReadwiseReader(api_key=token)documents=loader.load_data()index=VectorStoreIndex.from_documents(documents)index.query("What was the paper 'Attention is all you need' about?")You can also query for highlights that have been created after a certain time:importosimportdatetimefromllama_indeximportVectorStoreIndex,download_loaderReadwiseReader=download_loader("ReadwiseReader")token=os.getenv("READWISE_API_KEY")loader=ReadwiseReader(api_key=token)seven_days_ago=datetime.datetime.now()-datetime.timedelta(days=7)documents=loader.load_data(updated_after=seven_days_ago)index=VectorStoreIndex.from_documents(documents)index.query("What has Elon Musk done this time?")This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-reddit
Reddit ReaderFor any subreddit(s) you're interested in, search for relevant posts using keyword(s) and load the resulting text in the post and and top-level comments into LLMs/ LangChains.Get your Reddit credentials readyVisit Reddit App Preferences (https://www.reddit.com/prefs/apps) orhttps://old.reddit.com/prefs/apps/Scroll to the bottom and click "create another app..."Fill out the name, description, and redirect url for your app, then click "create app"Now you should be able to see the personal use script, secret, and name of your app. Store those as environment variables REDDIT_CLIENT_ID, REDDIT_CLIENT_SECRET, and REDDIT_USER_AGENT respectively.Additionally store the environment variables REDDIT_USERNAME and REDDIT_PASSWORD, which correspond to the credentials for your Reddit account.UsageLlamaIndexfromllama_indeximportVectorStoreIndex,download_loaderRedditReader=download_loader("RedditReader")subreddits=["MachineLearning"]search_keys=["PyTorch","deploy"]post_limit=10loader=RedditReader()documents=loader.load_data(subreddits=subreddits,search_keys=search_keys,post_limit=post_limit)index=VectorStoreIndex.from_documents(documents)index.query("What are the pain points of PyTorch users?")LangChainfromllama_indeximportVectorStoreIndex,download_loaderfromlangchain.agentsimportinitialize_agent,Toolfromlangchain.llmsimportOpenAIfromlangchain.chains.conversation.memoryimportConversationBufferMemoryRedditReader=download_loader("RedditReader")subreddits=["MachineLearning"]search_keys=["PyTorch","deploy"]post_limit=10loader=RedditReader()documents=loader.load_data(subreddits=subreddits,search_keys=search_keys,post_limit=post_limit)index=VectorStoreIndex.from_documents(documents)tools=[Tool(name="Reddit Index",func=lambdaq:index.query(q),description=f"Useful when you want to read relevant posts and top-level comments in subreddits.",),]llm=OpenAI(temperature=0)memory=ConversationBufferMemory(memory_key="chat_history")agent_chain=initialize_agent(tools,llm,agent="zero-shot-react-description",memory=memory)output=agent_chain.run(input="What are the pain points of PyTorch users?")print(output)This loader is designed to be used as a way to load data intoGPT Indexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-remote
Remote Page/File LoaderThis loader makes it easy to extract the text from any remote page or file using just its url. If there's a file at the url, this loader will download it temporarily and parse it usingSimpleDirectoryReader. It is an all-in-one tool for (almost) any url.As a result, any page or type of file is supported. For instance, if a.txturl such as aProject Gutenberg bookis passed in, the text will be parsed as is. On the other hand, if a hosted .mp3 url is passed in, it will be downloaded and parsed usingAudioTranscriber.UsageTo use this loader, you need to pass in aPathto a local file. Optionally, you may specify afile_extractorfor theSimpleDirectoryReaderto use, other than the default one.fromllama_indeximportdownload_loaderRemoteReader=download_loader("RemoteReader")loader=RemoteReader()documents=loader.load_data(url="https://en.wikipedia.org/wiki/File:Example.jpg")This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-remote-depth
Remote Page/File LoaderThis loader makes it easy to extract the text from the links available in a webpage URL, and extract the links presents in the page. It's based onRemoteReader(reading single page), that is based onSimpleDirectoryReader(parsing the document if file is a pdf, etc). It is an all-in-one tool for (almost) any group of urls.You can try with this MIT lecture link, it will be able to extract the syllabus, the PDFs, etc:https://ocw.mit.edu/courses/5-05-principles-of-inorganic-chemistry-iii-spring-2005/pages/syllabus/UsageYou need to specify the parameterdepthto specify how many levels of links you want to extract. For example, if you want to extract the links in the page, and the links in the links in the page, you need to specifydepth=2.fromllama_indeximportdownload_loaderRemoteDepthReader=download_loader("RemoteDepthReader")loader=RemoteDepthReader()documents=loader.load_data(url="https://ocw.mit.edu/courses/5-05-principles-of-inorganic-chemistry-iii-spring-2005/pages/syllabus/")This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-s3
S3 File or Directory LoaderThis loader parses any file stored on S3, or the entire Bucket (with an optional prefix filter) if no particular file is specified. When initializingS3Reader, you may pass in yourAWS Access Key. If none are found, the loader assumes they are stored in~/.aws/credentials.All files are temporarily downloaded locally and subsequently parsed withSimpleDirectoryReader. Hence, you may also specify a customfile_extractor, relying on any of the loaders in this library (or your own)!UsageTo use this loader, you need to pass in the name of your S3 Bucket. After that, if you want to just parse a single file, pass in its key. Note that if the file is nested in a subdirectory, the key should contain that, so likesubdirectory/input.txt.Otherwise, you may specify a prefix if you only want to parse certain files in the Bucket, or a subdirectory. AWS Access Key credentials may either be passed in during initialization or stored locally (see above).fromllama_indeximportdownload_loaderS3Reader=download_loader("S3Reader")loader=S3Reader(bucket="scrabble-dictionary",key="dictionary.txt",aws_access_id="[ACCESS_KEY_ID]",aws_access_secret="[ACCESS_KEY_SECRET]",)documents=loader.load_data()This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-sec-filings
SEC DATA DOWNLOADERPlease checkout this repo that I am building on SEC Question Answering AgentSEC-QAThis repository downloads all the texts from SEC documents (10-K and 10-Q). Currently, it is not supporting documents that are amended, but that will be added in the near futures.Install the required dependenciespython install -r requirements.txtThe SEC Downloader expects 5 attributestickers: It is a list of valid tickersamount: Number of documents that you want to downloadfiling_type: 10-K or 10-Q filing typenum_workers: It is for multithreading and multiprocessing. We have multi-threading at the ticker level and multi-processing at the year level for a given tickerinclude_amends: To include amendments or not.Usagefromllama_indeximportdownload_loaderSECFilingsLoader=download_loader("SECFilingsLoader")loader=SECFilingsLoader(tickers=["TSLA"],amount=3,filing_type="10-K")loader.load_data()It will download the data in the following directories and sub-directories-AAPL- 2018- 10-K.json- 2019- 10-K.json- 2020- 10-K.json- 2021- 10-K.json- 10-Q_12.json- 2022- 10-K.json- 10-Q_03.json- 10-Q_06.json- 10-Q_12.json- 2023- 10-Q_04.json-GOOGL- 2018- 10-K.json- 2019- 10-K.json- 2020- 10-K.json- 2021- 10-K.json- 10-Q_09.json- 2022- 10-K.json- 10-Q_03.json- 10-Q_06.json- 10-Q_09.json- 2023- 10-Q_03.json-TSLA- 2018- 10-K.json- 2019- 10-K.json- 2020- 10-K.json- 2021- 10-K.json- 10-KA.json- 10-Q_09.json- 2022- 10-K.json- 10-Q_03.json- 10-Q_06.json- 10-Q_09.json- 2023- 10-Q_03.jsonHere for each ticker we have separate folders with 10-K data inside respective years and 10-Q data is saved in the respective year along with the month.10-Q_03.jsonmeans March data of 10-Q document. Also, the amended documents are stored in their respective yearEXAMPLESThis loader is can be used with both Langchain and LlamaIndex.LlamaIndexfromllama_indeximportVectorStoreIndex,download_loaderfromllama_indeximportSimpleDirectoryReaderSECFilingsLoader=download_loader("SECFilingsLoader")loader=SECFilingsLoader(tickers=["TSLA"],amount=3,filing_type="10-K")loader.load_data()documents=SimpleDirectoryReader("data\TSLA\2022").load_data()index=VectorStoreIndex.from_documents(documents)index.query("What are the risk factors of Tesla for the year 2022?")Langchainfromllama_indeximportdownload_loaderfromlangchain.llmsimportOpenAIfromlangchain.chainsimportRetrievalQAfromlangchain.document_loadersimportDirectoryLoaderfromlangchain.indexesimportVectorstoreIndexCreatorSECFilingsLoader=download_loader("SECFilingsLoader")loader=SECFilingsLoader(tickers=["TSLA"],amount=3,filing_type="10-K")loader.load_data()dir_loader=DirectoryLoader("data\TSLA\2022")index=VectorstoreIndexCreator().from_loaders([dir_loader])retriever=index.vectorstore.as_retriever()qa=RetrievalQA.from_chain_type(llm=OpenAI(),chain_type="stuff",retriever=retriever)query="What are the risk factors of Tesla for the year 2022?"qa.run(query)REFERENCESUnstructured SEC Filings API:repo linkSEC Edgar Downloader:repo link
Package: llama-index-readers-semanticscholar
Semantic Scholar LoaderWelcome to Semantic Scholar Loader. This module serves as a crucial utility for researchers and professionals looking to get scholarly articles and publications from the Semantic Scholar database.For any research topic you are interested in, this loader reads relevant papers from a search result in Semantic Scholar intoDocuments.Please go throughdemo_s2.ipynbSome preliminaries -query_space: broad area of researchquery_string: a specific question to the documents in the query spaceUPDATE:To download the open access pdfs and extract text from them, simply mark thefull_textflag asTrue:s2reader=SemanticScholarReader()documents=s2reader.load_data(query_space,total_papers,full_text=True)UsageHere is an example of how to use this loader inllama_indexand get citations for a given query.LlamaIndexfromllama_index.llmsimportOpenAIfromllama_index.query_engineimportCitationQueryEnginefromllama_indeximport(VectorStoreIndex,ServiceContext,)fromllama_hub.semanticscholarimportSemanticScholarReaders2reader=SemanticScholarReader()# narrow down the search spacequery_space="large language models"# increase limit to get more documentsdocuments=s2reader.load_data(query=query_space,limit=10)service_context=ServiceContext.from_defaults(llm=OpenAI(model="gpt-3.5-turbo",temperature=0))index=VectorStoreIndex.from_documents(documents,service_context=service_context)query_engine=CitationQueryEngine.from_args(index,similarity_top_k=3,citation_chunk_size=512,)# query the indexresponse=query_engine.query("limitations of using large language models")print("Answer: ",response)print("Source nodes: ")fornodeinresponse.source_nodes:print(node.node.metadata)OutputAnswer:Thelimitationsofusinglargelanguagemodelsincludethestruggletolearnlong-tailknowledge[2],theneedforscalingbymanyordersofmagnitudetoreachcompetitiveperformanceonquestionswithlittlesupportinthepre-trainingdata[2],andthedifficultyinsynthesizingcomplexprogramsfromnaturallanguagedescriptions[3].
Sourcenodes:{'venue':'arXiv.org','year':2022,'paperId':'3eed4de25636ac90f39f6e1ef70e3507ed61a2a6','citationCount':35,'openAccessPdf':None,'authors':['M. Shanahan'],'title':'Talking About Large Language Models'}{'venue':'arXiv.org','year':2022,'paperId':'6491980820d9c255b9d798874c8fce696750e0d9','citationCount':31,'openAccessPdf':None,'authors':['Nikhil Kandpal','H. Deng','Adam Roberts','Eric Wallace','Colin Raffel'],'title':'Large Language Models Struggle to Learn Long-Tail Knowledge'}{'venue':'arXiv.org','year':2021,'paperId':'a38e0f993e4805ba8a9beae4c275c91ffcec01df','citationCount':305,'openAccessPdf':None,'authors':['Jacob Austin','Augustus Odena','Maxwell Nye','Maarten Bosma','H. Michalewski','David Dohan','Ellen Jiang','Carrie J. Cai','Michael Terry','Quoc V. Le','Charles Sutton'],'title':'Program Synthesis with Large Language Models'}
Package: llama-index-readers-singlestore
SingleStore LoaderThe SingleStore Loader retrieves a set of documents from a specified table in a SingleStore database. The user initializes the loader with database information and then provides a search embedding for retrieving similar documents.UsageHere's an example usage of the SingleStoreReader:fromllama_hub.singlestoreimportSingleStoreReader# Initialize the reader with your SingleStore database credentials and other relevant detailsreader=SingleStoreReader(scheme="mysql",host="localhost",port="3306",user="username",password="password",dbname="database_name",table_name="table_name",content_field="text",vector_field="embedding",)# The search_embedding is an embedding representation of your query_vector.# Example search_embedding:#   search_embedding=[0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]search_embedding=[n1,n2,n3,...]# load_data fetches documents from your SingleStore database that are similar to the search_embedding.# The top_k argument specifies the number of similar documents to fetch.documents=reader.load_data(search_embedding=search_embedding,top_k=5)
Package: llama-index-readers-smart-pdf-loader
Smart PDF LoaderSmartPDFLoader is a super fast PDF reader that understands the layout structure of PDFs such as nested sections, nested lists, paragraphs and tables.
It uses layout information to smartly chunk PDFs into optimal short contexts for LLMs.RequirementsInstall the llmsherpa library if it is not already present:pip install llmsherpaUsageHere's an example usage of the SmartPDFLoader:fromllama_hub.smart_pdf_loaderimportSmartPDFLoaderllmsherpa_api_url="https://readers.llmsherpa.com/api/document/developer/parseDocument?renderFormat=all"pdf_url="https://arxiv.org/pdf/1910.13461.pdf"# also allowed is a file path e.g. /home/downloads/xyz.pdfpdf_loader=SmartPDFLoader(llmsherpa_api_url=llmsherpa_api_url)documents=pdf_loader.load_data(pdf_url)Now you can use the documents with other LlamaIndex components. For example, for retrieval augmented generation, try this:fromllama_indeximportVectorStoreIndexindex=VectorStoreIndex.from_documents(documents)query_engine=index.as_query_engine()response=query_engine.query("list all the tasks that work with bart")print(response)response=query_engine.query("what is the bart performance score on squad")print(response)More ExamplesSmartPDFLoader is based on LayoutPDFReader fromllmsherpalibrary. See thedocumentationthere to explore other ways to use the library for connecting data from your PDFs with LLMs.Summarize a section using promptsAnalyze a table using promptsVector search and RAG
Package: llama-index-readers-snowflake
Snowflake LoaderThis loader connects to Snowflake (using SQLAlchemy under the hood). The user specifies a query and extracts Document objects corresponding to the results. You can use this loader to easily connect to a database on Snowflake and pass the documents into aGPTSQLStructStoreIndexfrom LlamaIndex.UsageOption 1: Pass your own SQLAlchemy Engine object of the database connectionHere's an example usage of the SnowflakeReader.fromllama_indeximportdownload_loaderSnowflakeReader=download_loader("SnowflakeReader")reader=SnowflakeReader(engine=your_sqlalchemy_engine,)query="SELECT * FROM your_table"documents=reader.load_data(query=query)Option 2: Pass the required parameters to esstablish Snowflake connectionHere's an example usage of the SnowflakeReader.fromllama_indeximportdownload_loaderSnowflakeReader=download_loader("SnowflakeReader")reader=SnowflakeReader(account="your_account",user="your_user",password="your_password",database="your_database",schema="your_schema",warehouse="your_warehouse",role="your_role",# Optional role settingproxy="http://proxusername:proxypassword@myproxy:port",# Optional proxy setting)query="SELECT * FROM your_table"documents=reader.load_data(query=query)AuthorGodwin Paul VincentThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.