Package: vector-lake
VectorLakeVectorLake is a robust, vector database designed for low maintenance, cost, efficient storage and ANN querying of any size vector data distributed across S3 files.🏷 FeaturesInspired by articleWhich Vector Database Should I Use? A Comparison CheatsheetVectorLake created with tradeoff to minimize db maintenance, cost and provide custom data partitioning strategiesNative Big Data Support: Specifically designed to handle large datasets, making it ideal for big data projects.Vector Data Handling: Capable of storing and querying high-dimensional vectors, commonly used for embedding storage in machine learning projects.projects.Efficient Search: Efficient nearest neighbors search, ideal for querying similar vectors in high-dimensional spaces. This makes it especially useful for querying for similar vectors in a high-dimensional space.Data Persistence: Supports data persistence on disk, network volume and S3, enabling long-term storage and retrieval of indexed data.Customizable Partitioning: Trade-off design to minimize database maintenance, cost, and provide custom data partitioning strategies.Native support of LLM Agents.Feature store for experimental data.📦 InstallationTo get started with VectorLake, simply install the package using pip:pipinstallvector_lake⛓️ Quick Startimportnumpyasnpfromvector_lakeimportVectorLakedb=VectorLake(location="s3://vector-lake",dimension=5,approx_shards=243)N=100# for exampleD=5# Dimensionality of each vectorembeddings=np.random.rand(N,D)foreminembeddings:db.add(em,metadata={},document="some document")db.persist()db=VectorLake(location="s3://vector-lake",dimension=5,approx_shards=243)# re-init testdb.query([0.56325391,0.1500543,0.88579166,0.73536349,0.7719873])Custom feature partitionCustom partition to group features by custom categoryimportnumpyasnpfromvector_lake.core.indeximportPartitionif__name__=="__main__":db=Partition(location="s3://vector-lake",partition_key="feature",dimension=5)N=100# for exampleD=5# Dimensionality of each vectorembeddings=np.random.rand(N,D)foreminembeddings:db.add(em,metadata={},document="some document")db.persist()db=Partition(location="s3://vector-lake",key="feature",dimension=5)# re-init testdb.bucketsdb.query([0.56325391,0.1500543,0.88579166,0.73536349,0.7719873])Local persistent volumeimportnumpyasnpfromvector_lakeimportVectorLakedb=VectorLake(location="/mnt/db",dimension=5,approx_shards=243)N=100# for exampleD=5# Dimensionality of each vectorembeddings=np.random.rand(N,D)foreminembeddings:db.add(em,metadata={},document="some document")db.persist()db=VectorLake(location="/mnt/db",dimension=5,approx_shards=243)# re-init testdb.query([0.56325391,0.1500543,0.88579166,0.73536349,0.7719873])Langchain Retrievalfromlangchain.document_loadersimportTextLoaderfromlangchain.embeddings.sentence_transformerimportSentenceTransformerEmbeddingsfromlangchain.text_splitterimportCharacterTextSplitterfromvector_lake.langchainimportVectorLakeStoreloader=TextLoader("Readme.md")documents=loader.load()# split it into chunkstext_splitter=CharacterTextSplitter(chunk_size=100,chunk_overlap=0)docs=text_splitter.split_documents(documents)# create the open-source embedding functionembedding=SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")db=VectorLakeStore.from_documents(documents=docs,embedding=embedding)query="What is Vector Lake?"docs=db.similarity_search(query)# print resultsprint(docs[0].page_content)Why VectorLake?VectorLake gives you the functionality of a simple, resilient vector database, but with very easy setup and low operational overhead. With it you've got a lightweight and reliable distributed vector store.VectorLake leverages Hierarchical Navigable Small World (HNSW) for data partitioning across all vector data shards. This ensures that each modification to the system aligns with vector distance. You can learn more about the design here.LimitationsTBD🛠️ Roadmap👋 ContributingContributions to VectorLake are welcome! If you'd like to contribute, please follow these steps:Fork the repository on GitHubCreate a new branch for your changesCommit your changes to the new branchPush your changes to the forked repositoryOpen a pull request to the main VectorLake repositoryBefore contributing, please read the contributing guidelines.LicenseVectorLake is released under the MIT License.