Making a generalized method to ingest documents in any vector database.
Labels: name: area: vector store, name: auto:improvement, name: integration: chroma

### Discussed in https://github.com/langchain-ai/langchain/discussions/15877

<div type='discussions-op-text'>

<sup>Originally posted by **arthurbrenno** January 11, 2024</sup>
I'm trying to make a **generalized method** that **works with any vector database**, so I can **change the implementation** later. The main focus is simply ingesting documents inside a vectorstore. 

My main issue always has been **_data persistency_** and the _flexibility_ to change the vector database implementation, because I like to explore different alternatives and there are many of them. That's why I'm (trying) to make a method like this.

Here's a snippet of my implementation:
```py
from langchain_community.document_loaders.base import BaseLoader
from langchain_core.documents.base import Document
from langchain_core.embeddings import Embeddings
from langchain.vectorstores import VectorStore
from conversai.core.services import BaseService
from langchain.text_splitter import TextSplitter
from typing import List, Type, Dict
from typing import Any


class IngestionService(BaseService):
    """Class that handles the ingestion of documents inside a vector store."""
    
    async def execute(url: str,
                     collection: str,
                     loader_class: Type[BaseLoader],
                     splitter: TextSplitter,
                     embeddings: Embeddings,
                     vector_store_cls: Type[VectorStore],
                     **vector_store_kwargs) -> Dict[str, Any]:
        """Ingests a document inside the desired vector store.

        Args:
            url (str): Url of the document to ingest.
            collection (str): Collection of the vector store to ingest the document.
            loader (BaseLoader): The loader of the document.
            splitter (TextSplitter): The splitter of the document.
            embeddings (Embeddings): The embedding model to apply on the splitted documents.
            vector_store (VectorStore): The vector store to store the document.

        Returns:
            bool: True on success, False on failure.
        """
        
        # Instantiating the loader
        loader: BaseLoader = loader_class(url)
        
        # Loading and splitting the document
        documents: List[Document] = loader.load_and_split(text_splitter=splitter)

        # Instantiating the vector store (assuming every implementation should support collection_name and embedding_func.)
        vector_store: VectorStore = vector_store_cls(collection_name=collection,
                                                     embedding_func=embeddings,
                                                     **vector_store_kwargs)
        
        # ids of the created texts
        texts_ids: List[str] = await vector_store.aadd_documents(documents)
        
        return {"success": True, "ingested_texts_ids": texts_ids}
```

The problem is that every vector database seems to have a different way of persisting itself (as I understood). For an example:

Chroma:
```py
db = Chroma(persist_directory=...)
```
(or using their separate client whitin Docker
```py
db = Chroma(client=client)
```

FAISS:
```py
# FAISS has 
db.save_local("faiss_index")
```

I need to ingest documents **at runtime.** This is important. I also need to group them inside `collections` for better searching later and group similar documents. Creating collections is one of my future implementations. 

My current vector store implementation is `Milvus`. And I think I'm generalising too much when I assume every vector db will have a "collection_name" and "embedding_func", like written in:
```py
vector_store: VectorStore = vector_store_cls(collection_name=collection,
                                                     embedding_func=embeddings,
                                                     **vector_store_kwargs)
```

Any support on making my method better is appreciated. I'm also reading the documentation and trying to solve this for myself, but I think community help would also be a great startpoint. One of my suggestions (I've seen some people talking about this) is making the documentation more clear about data persistency. The examples only show short-term usage of the vectorstores, which is not the case in many production enviroments</div>