Package: chroma-datasets
Chroma DatasetsMaking it easy to load data into Chroma since 2023pip install chroma_datasetsCurrent DatasetsDatasetSizeContributorPython ClassState of the Union51kbChromafrom chroma_datasets import StateOfTheUnionPaul Graham Essay1.3mbChromafrom chroma_datasets import PaulGrahamEssaySciQ2.8mbHugging Facefrom chroma_datasets import SciQHuberman Podcasts4.3mbDexa AIfrom chroma_datasets import HubermanPodcastschroma_datasetsis generally backed by hugging face datasets, but it is not a requirement.How to useThe following will:Download the 2022 State of the UnionChunk it up for youEmbed it using Chroma's default open-source embedding functionImport it into Chromaimportchromadbfromchroma_datasetsimportStateOfTheUnionfromchroma_datasets.utilsimportimport_into_chromachroma_client=chromadb.Client()collection=import_into_chroma(chroma_client=chroma_client,dataset=StateOfTheUnion)result=collection.query(query_texts=["The United States of America"])print(result)Adding a New DatasetWe welcome new datasets!These datasets can be anything generally useful to developer education for processing and using embeddings.Datasets can be:raw text (likeStateOfTheUnion)pre-chunked data (likeSciPy)chunked and embedded (likePaulGrahamEssay)See the examples/upload.ipynb for an example of how to create a dataset on Hugging Face (the default path)Create a new dataset from a Chroma Collection(more examples of this inexamples/upload.ipynbandexamples/upload_embeddings.ipynb)Install dependenciespipinstalldatasetshuggingface_hubchromadbLogin into Hugging Facehuggingface-cliloginUpload an existing collection to Hugging Face
** Hugging Face requires the data to have a "split name" - I suggest using a default of "data" **importchromadbfromchroma_datasets.utilsimportexport_collection_to_hf_datasetclient=chromadb.PersistantClient(path="./chroma_data")dataset=export_collection_to_hf_dataset(client=client,collection_name="paul_graham_essay",license="MIT")dataset.push_to_hub(repo_id="chromadb/paul_graham_essay",split="data")Create a Dataset ClassSet the string name of the embedding function you used to embed the data, this will make it possible for users to use the embeddings. Please also customize the helpful error message so if users pass in no embedding function or the wrong one, they get help.raw_textis optionalchunkedandto_chromayou can copy letter for letter if you uploaded with the method above.classPaulGrahamEssay(Dataset):"""http://www.paulgraham.com/worked.htmlColumns:- id: unique identifier for each chunk- document: the text of the chunk- embedding: the embedding of the chunk (OpenAI-ada-002)- metadata: metadata about the chunk"""hf_data=Noneembedding_function="OpenAIEmbeddingFunction"# name of embedding function inside Chromaembedding_function_instructions="""from chromadb.utils import embedding_functionsopenai_ef = embedding_functions.OpenAIEmbeddingFunction(api_key="YOUR_API_KEY",model_name="text-embedding-ada-002")"""@classmethoddefload_data(cls):cls.hf_data=load_huggingface_dataset("chromadb/paul_graham_essay",split_name="data")@classmethoddefraw_text(cls)->str:ifcls.hf_dataisNone:cls.load_data()return"\n".join(cls.hf_data["document"])@classmethoddefchunked(cls)->List[Datapoint]:ifcls.hf_dataisNone:cls.load_data()returncls.hf_data@classmethoddefto_chroma(cls)->AddEmbedding:returnto_chroma_schema(cls.chunked())Add it to the manifest atchroma_datasets/__init__.pyto make it easy for people to retrieve (optional)Utility API DocumentationMany of these methods are purely conveneient. This makes it easy to save and load Chroma Collections to disk. See./examples/example_export.ipynbfor example use.fromchromadb.utilsimport(export_collection_to_hf_dataset,export_collection_to_hf_dataset_to_disk,import_chroma_exported_hf_dataset_from_disk,import_chroma_exported_hf_dataset)# Exports a Chroma collection to an in-memory HuggingFace Datasetdefexport_collection_to_hf_dataset(chroma_client,collection_name,license="MIT"):# Exports a Chroma collection to a HF dataset and saves to the pathdefexport_collection_to_hf_dataset_to_disk(chroma_client,collection_name,path,license="MIT"):# Imports a HuggingFace Dataset into a Chroma Collectiondefimport_chroma_exported_hf_dataset(chroma_client,dataset,collection_name,embedding_function=None):# Imports a HuggingFace Dataset from Disk and loads it into a Chroma Collectiondefimport_chroma_exported_hf_dataset_from_disk(chroma_client,path,collection_name,embedding_function=None):TodoAdd test suite to test some of the critical pathsAdd automated pypi releaseAdd loaders for other locations (remote like S3, local like CSV... etc)Super easy streamlit/gradio wrapper to push up a collection to interact with