Saravanan666/pypi-org huggingface dataset:
Package: llama-index-readers-hive
Hive LoaderThe Hive Loader returns a set of texts corresponding to documents from Hive based on the customized query.
The user initializes the loader with Hive connection args and then using query to fetch data from Hive.UsageHere's an example usage of the hiveReader to load 100 documents.fromllama_indeximportdownload_loaderHiveReader=download_loader("HiveReader")reader=HiveReader(host="localhost",port=10000,database="PERSON_DB",username="hiveuser_test",auth="NOSASL",)query="SELECT * FROM p1 LIMIT 100"documents=reader.load_data(query=query)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-hubspot
Hubspot LoaderThis loader loads documents from Hubspot. The user specifies an access token to initialize the HubspotReader.At the moment, this loader only supports access token authentication. To obtain an access token, you will need to create a private app by following instructionshere.UsageHere's an example usage of the HubspotReader.fromllama_indeximportdownload_loaderimportosHubspotReader=download_loader("HubspotReader")reader=HubspotReader("<HUBSPOT_ACCESS_TOKEN>")documents=reader.load_data()This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-huggingface-fs
Hugging Face FS LoaderThis loader uses Hugging Face Hub's Filesystem API (> 0.14) to
load datasets.Besides the existingload_datafunction, you may also choose to useload_dictsandload_df.UsageTo use this loader, you need to pass in a path to a Hugging Face dataset.frompathlibimportPathfromllama_indeximportdownload_loaderHuggingFaceFSReader=download_loader("HuggingFaceFSReader")# load documentsloader=HuggingFaceFSReader()documents=loader.load_data("datasets/dair-ai/emotion/data/data.jsonl.gz")# load dictsdicts=loader.load_dicts("datasets/dair-ai/emotion/data/data.jsonl.gz")# load dfdf=loader.load_df("datasets/dair-ai/emotion/data/data.jsonl.gz")This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-hwp
HWP LoaderThis loader reads the HWP file, which is the format of many official documents in South Korea.UsageTo use this loader, you need to pass in a file name. It's fine whether the file is compressed or not.fromllama_hub.hangeulimportHWPReaderfrompathlibimportPathhwp_path=Path("/path/to/hwp")reader=HWPReader()documents=reader.load_data(file=hwp_path)
Package: llama-index-readers-imdb-review
IMDB MOVIE REVIEWS LOADERThis loader fetches all the reviews of a movie or a TV-series from IMDB official site. This loader is working on Windows machine and it requires further debug on Linux. Fixes are on the wayInstall the required dependenciespip install -r requirements.txtThe IMDB downloader takes in two attributesmovie_name_year: The name of the movie or series and yearwebdriver_engine: To use edge, google or gecko (mozilla) webdrivergenerate_csv: Whether to generate csv filemultithreading: whether to use multithreading or notUsagefromllama_indeximportdownload_loaderIMDBReviewsloader=download_loader("IMDBReviews")loader=IMDBReviews(movie_name_year="The Social Network 2010",webdriver_engine="edge")docs=loader.load_data()The metadata has the following informationdate of the review (date)title of the review (title)rating of the review (rating)link of the review (link)whether the review is spoiler or not (spoiler)number of people found the review helpful (found_helpful)total number of votes (total)It will download the files inside the foldermovie_reviewswith the filename as the movie nameEXAMPLESThis loader can be used with both Langchain and LlamaIndex.LlamaIndexfromllama_indeximportVectorStoreIndex,download_loaderfromllama_indeximportVectorStoreIndexIMDBReviewsloader=download_loader("IMDBReviews")loader=IMDBReviewsloader(movie_name_year="The Social Network 2010",webdriver_engine="edge",generate_csv=False,multithreading=False,)docs=loader.load_data()index=VectorStoreIndex.from_documents(documents)query_engine=index.as_query_engine()response=query_engine.query("What did the movie say about Mark Zuckerberg?",)print(response)Langchainfromllama_indeximportdownload_loaderfromlangchain.llmsimportOpenAIfromlangchain.agents.agent_toolkits.pandasimport(create_pandas_dataframe_agent,)fromlangchain.agentsimportToolfromlangchain.agentsimportinitialize_agentfromlangchain.chat_modelsimportChatOpenAIIMDBReviewsloader=download_loader("IMDBReviews")loader=IMDBReviewsloader(movie_name_year="The Social Network 2010",webdriver_engine="edge",generate_csv=False,multithreading=False,)docs=loader.load_data()tools=[Tool(name="LlamaIndex",func=lambdaq:str(index.as_query_engine().query(q)),description="useful for when you want to answer questions about the movies and their reviews. The input to this tool should be a complete english sentence.",return_direct=True,),]llm=ChatOpenAI(temperature=0)agent=initialize_agent(tools,llm,agent="conversational-react-description")agent.run("What did the movie say about Mark Zuckerberg?")
Package: llama-index-readers-intercom
Intercom LoaderThis loader fetches the text from Intercom help articles using the Intercom API. It also uses the BeautifulSoup library to parse the HTML and extract the text from the articles.UsageTo use this loader, you need to pass in an Intercom account access token.fromllama_indeximportdownload_loaderIntercomReader=download_loader("IntercomReader")loader=IntercomReader(intercom_access_token="my_access_token")documents=loader.load_data()This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-jira
JIRA ReaderThe Jira loader returns a set of issues based on the query provided to the dataloader.
We can follow two methods to initialize the loader-
1- basic_auth -> this takes a dict with the following keysbasic_auth:{ "email": "email", "api_token": "token", "server_url": "server_url" }2- Oauth2 -> this takes a dict with the following keysoauth:{ "cloud_id": "cloud_id", "api_token": "token" }You can follow this link for more information regarding Oauth2 ->https://developer.atlassian.com/cloud/confluence/oauth-2-3lo-apps/UsageHere's an example of how to use itfromllama_hub.jiraimportJiraReaderreader=JiraReader(email=email,api_token=api_token,server_url="your-jira-server.com")documents=reader.load_data(query="project = <your-project>")Alternately, you can also use download_loader from llama_indexfromllama_indeximportdownload_loaderJiraReader=download_loader("JiraReader")reader=JiraReader(email=email,api_token=api_token,server_url="your-jira-server.com")documents=reader.load_data(query="project = <your-project>")
Package: llama-index-readers-joplin
Joplin (Markdown) LoaderJoplinis an open source note-taking app. Capture your thoughts and securely access them from any device.This readme covers how to load documents from aJoplindatabase.Joplinhas aREST APIfor accessing its local database. This reader uses the API to retrieve all notes in the database and their metadata. This requires an access token that can be obtained from the app by following these steps:Open theJoplinapp. The app must stay open while the documents are being loaded.Go to settings / options and select "Web Clipper".Make sure that the Web Clipper service is enabled.Under "Advanced Options", copy the authorization token.You may either initialize the reader directly with the access token, or store it in the environment variable JOPLIN_ACCESS_TOKEN.An alternative to this approach is to export theJoplin's note database to Markdown files (optionally, with Front Matter metadata) and use a Markdown reader, such as ObsidianReader, to load them.UsageHere's an example usage of the JoplinReader.fromllama_indeximportdownload_loaderimportosJoplinReader=download_loader("JoplinReader")documents=JoplinReader(access_token="<access_token>").load_data()# Returns list of documents
Package: llama-index-readers-kibela
Kibela ReaderThis reader fetches article from yourKibelanotes using the GraphQL API.UsageHere's an example of how to use it. You can get your access token fromhere.importosfromllama_hub.kibelaimportKibelaReaderteam=os.environ["KIBELA_TEAM"]token=os.environ["KIBELA_TOKEN"]reader=KibelaReader(team=team,token=token)documents=reader.load_data()Alternately, you can also use download_loader from llama_indeximportosfromllama_indeximportdownload_loaderKibelaReader=download_loader("KibelaReader")team=os.environ["KIBELA_TEAM"]token=os.environ["KIBELA_TOKEN"]reader=KibelaReader(team=team,token=token)documents=reader.load_data()
Package: llama-index-readers-lilac
Lilac readerLilacis an open-source product that helps you analyze, enrich, and clean unstructured data with AI.It can be used to analyze, clean, structure, and label data that can be used in downstream LlamaIndex and LangChain applications.Lilac projectsThis assumes you've already run Lilac locally, and have a project directory with a dataset. For more details on Lilac projects, seeLilac ProjectsYou can use any LlamaIndex loader to load data into Lilac, clean data, and then bring it back into LlamaIndex Documents.UsageLlamaIndex => LilacSeethis notebookfor getting data into Lilac from LlamaHub.fromllama_indeximportdownload_loaderimportlilacasll# See: https://llamahub.ai/l/papers-arxivArxivReader=download_loader("ArxivReader")loader=ArxivReader()documents=loader.load_data(search_query="au:Karpathy")# Set the project directory for Lilac.ll.set_project_dir("./data")# This assumes you already have a lilac project set up.# If you don't, use ll.init(project_dir='./data')ll.create_dataset(config=ll.DatasetConfig(namespace="local",name="arxiv-karpathy",source=ll.LlamaIndexDocsSource(# documents comes from the loader.load_data call in the previous cell.documents=documents),))# You can start a lilac server with. Once you've cleaned the dataset, you can come back into GPTIndex.ll.start_server(project_dir="./data")Lilac => LlamaIndex Documentsfromllama_indeximportVectorStoreIndex,download_loaderLilacReader=download_loader("LilacReader")loader=LilacReader()documents=loader.load_data(project_dir="~/my_project",# The name of your dataset in the project dir.dataset="local/arxiv-karpathy",)index=VectorStoreIndex.from_documents(documents)index.query("How are ImageNet labels validated?")This loader is designed to be used as a way to load data intoGPT Indexand/or subsequently used in aLangChainAgent.
Package: llama-index-readers-linear
Linear ReaderThe Linear loader returns issue based on the query.UsageHere's an example of how to use itfromllama_hub.linearimportLinearReaderreader=LinearReader(api_key=api_key)query="""query Team {team(id: "9cfb482a-81e3-4154-b5b9-2c805e70a02d") {idnameissues {nodes {idtitledescriptionassignee {idname}createdAtarchivedAt}}}}"""documents=reader.load_data(query=query)Alternately, you can also use download_loader from llama_indexfromllama_indeximportdownload_loaderLinearReader=download_loader("LinearReader")reader=LinearReader(api_key=api_key)query="""query Team {team(id: "9cfb482a-81e3-4154-b5b9-2c805e70a02d") {idnameissues {nodes {idtitledescriptionassignee {idname}createdAtarchivedAt}}}}"""documents=reader.load_data(query=query)
Package: llama-index-readers-llama-parse
LlamaParseLlamaParse is an API created by LlamaIndex to efficiently parse and represent files for efficient retrieval and context augmentation using LlamaIndex frameworks.LlamaParse directly integrates withLlamaIndex.Currently available forfree. Try it out today!NOTE:Currently, only PDF files are supported.Getting StartedFirst, login and get an api-key fromhttps://cloud.llamaindex.ai.Then, make sure you have the latest LlamaIndex version installed.NOTE:If you are upgrading from v0.9.X, we recommend following ourmigration guide, as well as uninstalling your previous version first.pip uninstall llama-index  # run this if upgrading from v0.9.x or older
pip install -U llama-index --upgrade --no-cache-dir --force-reinstallLastly, install the package:pip install llama-parseNow you can run the following to parse your first PDF file:importnest_asyncionest_asyncio.apply()fromllama_parseimportLlamaParseparser=LlamaParse(api_key="llx-...",# can also be set in your env as LLAMA_CLOUD_API_KEYresult_type="markdown",# "markdown" and "text" are availableverbose=True,)# syncdocuments=parser.load_data("./my_file.pdf")# sync batchdocuments=parser.load_data(["./my_file1.pdf","./my_file2.pdf"])# asyncdocuments=awaitparser.aload_data("./my_file.pdf")# async batchdocuments=awaitparser.aload_data(["./my_file1.pdf","./my_file2.pdf"])Using withSimpleDirectoryReaderYou can also integrate the parser as the default PDF loader inSimpleDirectoryReader:importnest_asyncionest_asyncio.apply()fromllama_parseimportLlamaParsefromllama_index.coreimportSimpleDirectoryReaderparser=LlamaParse(api_key="llx-...",# can also be set in your env as LLAMA_CLOUD_API_KEYresult_type="markdown",# "markdown" and "text" are availableverbose=True,)file_extractor={".pdf":parser}documents=SimpleDirectoryReader("./data",file_extractor=file_extractor).load_data()Full documentation forSimpleDirectoryReadercan be found on theLlamaIndex Documentation.ExamplesSeveral end-to-end indexing examples can be found in the examples folderGetting StartedAdvanced RAG ExampleRaw API UsageTerms of ServiceSee theTerms of Service Here.
Package: llama-index-readers-macrometa-gdn
Macrometa GDN LoaderThis loader takes in a Macrometa federation URL, API key, and collection name and returns a list of vectors.UsageTo use this loader, you need to pass the URL and API key through the class constructor, and then load the data using an array of collection names.fromllama_indeximportdownload_loaderMacrometaGDNReader=download_loader("MacrometaGDNReader")collections=["test_collection"]loader=MacrometaGDNReader(url="https://api-macrometa.io",apikey="test")vectors=loader.load_data(collection_list=collections)
Package: llama-index-readers-mangadex
MangaDex LoaderThis loader fetches information from the MangaDex API, by manga title.Usagefromllama_indeximportdownload_loaderMangaDexReader=download_loader("MangaDexReader")loader=MangaDexReader()documents=loader.load_data(titles=["manga title 1","manga title 2"],lang="en")OutputTextDocument text is the manga title. There are alternate titles for many manga, so the canonical title will be returned, even if it is not the title that the user queried with.Extra InfoDataDescriptionid (str)MangaDex manga idauthor (str)Author's full nameartist (str)Artist's full namedescription (str)Manga descriptionoriginal_language (str)The language of the source material (before translation)tags (List[str])Describes the manga's genre, e.g. "slice of life"chapter_count (int)How many chapters exist in the requested languagelatest_chapter_published_at (str)Timestamp (YYYY-MM-DDTHH:MM:SS in timezone UTC+0) for the latest chapter in the requested languageExamplesThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent.
Package: llama-index-readers-mangoapps-guides
MangoppsGuides LoaderThis loader fetches the text from Mangopps Guides.UsageTo use this loader, you need to pass base url of the MangoppsGuides installation (e.g.https://guides.mangoapps.com/) and the limit , i.e. max number of links it should crawlfromllama_indeximportdownload_loaderMangoppsGuidesReader=download_loader("MangoppsGuidesReader")loader=MangoppsGuidesReader()documents=loader.load_data(domain_url="https://guides.mangoapps.com",limit=1)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-maps
Osmmap LoaderThe Osmmap Loader will fetch map data from theOverpassapi for a certain place or area. VersionOverpass API 0.7.60is used by this loader.The api will provide you with all thenodes, relations, and waysfor the particular region when you request data for a region or location.Functions of the loaderTo start, it first filters out those nodes that are already tagged, leaving just those nodes that are within 2 kilometres of the target location. The following keys are removed during filtering:["nodes," "geometry," "members"] from each node. The response we received is based on the tags and values we provided, so be sure to do that. The actions are covered below.Steps to find the suitable tag and valuesVisitTaginfo. In essence, this website has all conceivable tags and values.Perform a search for the feature you're looking for, for instance, "hospital" will return three results: "hospital" as an amenity, "hospital" as a structure, and "hospital" as a healthcare facility.We may infer from the outcome that tag=amenity and value=hospital.Leave the values parameter to their default value if you do not need to filter.UsageThe use case is here.Let's meetJayasree, who is extracting map features from her neighbourhood using the OSM map loader.
She requires all the nodes, routes, and relations within a five-kilometer radius of her locale (Guduvanchery).She must use the following arguments in order to accomplish the aforementioned. Localarea = "Guduvanchery" (the location she wants to seek), local_area_buffer = 5000 (5 km).And the code snippet looks likefromllama_indeximportdownload_loaderMapReader=download_loader("OpenMap")loader=MapReader()documents=loader.load_data(localarea="Guduvanchery",search_tag="",tag_only=True,local_area_buffer=5000,tag_values=[""],)Now she wants only the list hospitals around the locationso she search for hospital tag in theTaginfoand she gotfromllama_indeximportdownload_loaderMapReader=download_loader("OpenMap")loader=MapReader()documents=loader.load_data(localarea="Guduvanchery",search_tag="amenity",tag_only=True,local_area_buffer=5000,tag_values=["hospital","clinic"],)This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-memos
Memos LoaderThis loader fetches text from self-hostedmemos.UsageTo use this loader, you need to specify the host where memos is deployed. If you need to filter, pass thecorresponding parameterinload_data.fromllama_indeximportdownload_loaderMemosReader=download_loader("MemosReader")loader=MemosReader("https://demo.usememos.com/")documents=loader.load_data({"creatorId":101})This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-microsoft-outlook
Outlook Local Calendar LoaderThis loader reads your past and upcoming Calendar events from your local Outlook .ost or .pst and parses the relevant info intoDocuments.It runs on Windows only and has only been tested with Windows 11. It has been designed to have a supoerset of the functionality of the Google Calendar reader.UsageHere's an example usage of the OutlookCalendar Reader. It will retrieve up to 100 future events, unless an optionalnumber_of_resultsargument is passed. It will also retrieve only future events, unless an optionalstart_dateargument is passed. Optionally events can be restricted to those which occur on or before a specific date by specifying the optionalend-dateparameter. By default,end-dateis 2199-01-01.It always returns Start, End, Subject, Location, and Organizer attributes and optionally returns additional attributes specified in themore_attributesparameter, which, if specified, must be a list of strings eg. ['Body','someotherattribute',...]. Attributes which don't exist in a calendar entry are ignored without warning.fromllama_indeximportdownload_loaderOutlookCalendarReader=download_loader("OutlookLocalCalendarReader")loader=OutlookCalendarReader()documents=loader.load_data()ExampleThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent.LlamaIndexfromllama_indeximportVectorStoreIndex,download_loaderOutlookCalendarReader=download_loader("OutlookLocalCalendarReader")loader=OutlookCalendarReader(start_date="2022-01-01",number_of_documents=1000)documents=loader.load_data()index=VectorStoreIndex.from_documents(documents)index.query("When did I last see George Guava? When do I see him again?")Note: it is actually better to give a structured prompt with this data and be sure to it is clear what today's date is and whether you want any data besides the indexed data used in answering the prompt.
Package: llama-index-readers-microsoft-sharepoint
Microsoft SharePoint ReaderThe loader loads the files from a folder in sharepoint site.It also supports traversing recursively through the sub-folders.PrequsitesApp Authentication using Microsoft Entra ID(formerly Azure AD)You need to create an App Registration in Microsoft Entra ID. ReferhereAPI Permissions for the created app.Microsoft Graph --> Application Permissions --> Sites.ReadAll (Grant Admin Consent)Microsoft Graph --> Application Permissions --> Files.ReadAll (Grant Admin Consent)Microsoft Graph --> Application Permissions --> BrowserSiteLists.Read.All (Grant Admin Consent)More info on Microsoft Graph APIs -Refer hereUsageTo use this loaderclient_id,client_secretandtenant_idof the registered app in Microsoft Azure Portal is required.This loader loads the files present in a specific folder in sharepoint.If the files are present in theTestfolder in SharePoint Site underrootdirectory, then the input for the loader forfile_pathisTestfromllama_indeximportdownload_loaderSharePointLoader=download_loader("SharePointReader")loader=SharePointLoader(client_id="<Client ID of the app>",client_secret="<Client Secret of the app>",tenant_id="<Tenant ID of the Microsoft Azure Directory>",)documents=loader.load_data(sharepoint_site_name="<Sharepoint Site Name>",sharepoint_folder_path="<Folder Path>",recursive=True,)The loader doesn't access other components of theSharePoint Site.
Package: llama-index-readers-mondaydotcom
Monday LoaderThis loader loads data from monday.com. The user specifies an API token to initialize the MondayReader. They then specify a monday.com board id to load in the corresponding Document objects.UsageHere's an example usage of the MondayReader.fromllama_indeximportdownload_loaderMondayReader=download_loader("MondayReader")reader=MondayReader("<monday_api_token>")documents=reader.load_data("<board_id: int>")Check out monday.com API docs -hereThis loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.
Package: llama-index-readers-nougat-ocr
Nougat OCR loaderThis loader reads the equations, symbols, and tables included in the PDF.Users can input the path of the academic PDF documentfilewhich they want to parse. This OCR understands LaTeX math and tables.UsageHere's an example usage of the PDFNougatOCR.fromllama_hub.nougat_ocrimportPDFNougatOCRreader=PDFNougatOCR()pdf_path=Path("/path/to/pdf")documents=reader.load_data(pdf_path)MiscellaneousAnoutputfolder will be created with the same name as the pdf and.mmdextension.
Package: llama-index-readers-openalex
OpenAlex ReaderThis loader will search for papers in OpenAlex and load them in llama-index. The main advantage of using OpenAlex is that you can search the full-text for Open Access papers as well.Usagefromllama_hub.openalex_loaderimportOpenAlexReaderopenalex_reader=OpenAlexReader(email="shauryr@gmail.com")query="biases in large language models"# changing this to full_text=True will let you search full-textdocuments=openalex_reader.load_data(query,full_text=False)What can it do?As shown indemo.ipynbwe can get answers with citations.query="biases in large language models"response=query_engine.query("list the biases in large language models in a markdown table")OutputSourceBiasesSource 1Data selection bias, social bias (gender, age, sexual orientation, ethnicity, religion, culture)Source 2Biases of what is right and wrong to do, reflecting ethical and moral norms of societySource 3Anti-Muslim biasSource 6Gender biasSource 9Anti-LGBTQ+ biasSource 10Potential bias in the outputCreditsOpenAlex API details are listedhereSome code adopted frompyAlex