Package: chromadb-data-pipes
ChromaDB Data Pipes 🖇️ - The easiest way to get data into and out of ChromaDBChromaDB Data Pipes is a collection of tools to build data pipelines for Chroma DB, inspired by the Unix philosophy of
"do one thing and do it well".Roadmap:✅ Integration with LangChain 🦜🔗🚫 Integration with LlamaIndex 🦙✅ Support more thanall-MiniLM-L6-v2as embedding functions (head over
toEmbedding Processorsfor more info)🚫 Multimodal support♾️ Much more!Installationpipinstallchromadb-data-pipesUsageGet help:cdp--helpExample Use CasesThis is a short list of use cases to evaluate whether this is the right tool for your needs:Importing large datasets from local documents (PDF, TXT, etc.), from HuggingFace, from local persisted Chroma DB or
even another remote Chroma DB.Exporting large dataset to HuggingFace or any other dataformat supported by the library (if your format is not
supported, either implement it in a small function or open an issue)Create a dataset from your data that you can share with others (including the embeddings)Clone Collection with different embedding function, distance function, and other HNSW fine-tuning parametersRe-embed documents in a collection with a different embedding functionBackup your data to ajsonlfileUse other existing unix or other tools to transform your data after exporting from or before importing into Chroma DBImportingImport data from HuggingFace Datasets to.jsonlfile:cdpds-get"hf://tazarov/chroma-qna?split=train">chroma-qna.jsonlImport data from HuggingFace Datasets to Chroma DB:The below command will import thetrainsplit of the given dataset to Chroma chroma-qnachroma-qnacollection. The
collection will be created if it does not exist and documents will be upserted.cdpds-get"hf://tazarov/chroma-qna?split=train"|cdpimport"http://localhost:8000/chroma-qna"--upsert--createImporting from a directory with PDF files into Local Persisted Chroma DB:cdpimppdfsample-data/papers/|grep"2401.02412.pdf"|head-1|cdpchunk-s500|cdpembed--efdefault|cdpimport"file://chroma-data/my-pdfs"--upsert--createNote: The above command will import the first PDF file from thesample-data/papers/directory, chunk it into 500
word chunks, embed each chunk and import the chunks to themy-pdfscollection in Chroma DB.ExportingExport data from Local Persisted Chroma DB to.jsonlfile:The below command will export the first 10 documents from thechroma-qnacollection tochroma-qna.jsonlfile.cdpexport"file://chroma-data/chroma-qna"--limit10>chroma-qna.jsonlExport data from Local Persisted Chroma DB to.jsonlfile with filter:The below command will export data from local persisted Chroma DB to a.jsonlfile using awherefilter to select
the documents to export.cdpexport"file://chroma-data/chroma-qna"--where'{"document_id": "123"}'>chroma-qna.jsonlExport data from Chroma DB to HuggingFace Datasets:The below command will export the first 10 documents with offset 10 from thechroma-qnacollection to HuggingFace
Datasetstazarov/chroma-qnadataset. The dataset will be uploaded to HF.HF Auth and Privacy: Make sure you haveHF_TOKEN=hf_....environment variable set. If you want your dataset to
be private, add--privateflag to thecdp ds-putcommand.cdpexport"http://localhost:8000/chroma-qna"--limit10--offset10|cdpds-put"hf://tazarov/chroma-qna-modified"To export a dataset to a file, use--uriwithfile://prefix:cdpexport"http://localhost:8000/chroma-qna"--limit10--offset10|cdpds-put"file://chroma-qna"File Location The file is relative to the current working directory.ProcessingCopy collection from one Chroma collection to another and re-embed the documents:cdpexport"http://localhost:8000/chroma-qna"|cdpembed--efdefault|cdpimport"http://localhost:8000/chroma-qna-def-emb"--upsert--createNote: SeeEmbedding Processorsfor more info about supported embedding functions.Import dataset from HF to Local Persisted Chroma and embed the documents:cdpds-get"hf://tazarov/ds2?split=train"|cdpembed--efdefault|cdpimport"file://chroma-data/chroma-qna-def-emb-hf"--upsert--createChunk Large Documents:cdpimppdfsample-data/papers/|grep"2401.02412.pdf"|head-1|cdpchunk-s500MiscCount the number of documents in a collection:cdpexport"http://localhost:8000/chroma-qna"|wc-l