Package: vectorflow-ai
VectorFlowOpen source, high-throughput, fault-tolerant vector embedding pipelineSimple API endpoint that ingests large volumes of raw data, processes, and stores or returns the vectors quickly and reliablyJoin our Discord|Website|Get in touch|Docs|Free Hosted VersionIntroductionVectorFlow is an open source, high throughput, fault tolerant vector embedding pipeline. With a simple API request, you can send raw data that will be chunked, embedded and stored in any vector database or returned back to you. VectorFlow is multi-modal and can ingest both textual and image data.This current version is an MVP. We recommend using it with Kubernetes in production. Right now the system only supports uploading single files at a time, up to 25 MB. For text-based files, it supports TXT, PDF and DOCX. For image files, it support JPG, JPEG, and PNG.Run it LocallyWith two simple commands you can set up VectorFlow locally:pip install vectorflow-ai
vectorflowTo start embedding documents, add your embedding and database keys to theenv_scrips/env_vars.shscript that was generated and set thefilepathvariable inclients/standard_upload_client.pyto point to the file you want to embed. Then run:source env_scrips/env_vars.sh
python clients/standard_upload_client.pyBelow is a more detailed description of how to manually set up and configure the system. Please note that thepipcommand will not create a development environment on your machine, it only sets up and runs the docker-compose.Docker-ComposeThe best way to run VectorFlow is viadocker compose. If you are running this on Mac, please grant Docker permissions to read from your Documents folderas instructed here. If this fails, remove thevolumesection from thedocker-compose.yml.1) Set Environment VariablesFirst create a folder in the root for all the environment variables:mkdir env_scripts
cd env_scripts
touch env_vars.envThis creates a file calledenv_vars.envin theenv_scriptsfolder to add all the environment variables mentioned below. You only need to set theLOCAL_VECTOR_DBvariable if you are running qdrant, Milvus or Weaviate locally.INTERNAL_API_KEY=your-choice
POSTGRES_USERNAME=postgres
POSTGRES_PASSWORD=your-choice
POSTGRES_DB=vectorflow
POSTGRES_HOST=postgres
RABBITMQ_USERNAME=guest
RABBITMQ_PASSWORD=guest
RABBITMQ_HOST=rabbitmq
EMBEDDING_QUEUE=embeddings
VDB_UPLOAD_QUEUE=vdb-upload
LOCAL_VECTOR_DB=qdrant | milvus | weaviateYou can choose a variable forINTERNAL_API_KEY,POSTGRES_PASSWORD, andPOSTGRES_DB, but they must be set.2) Run Docker-ComposeMake sure you pull Rabbit MQ and Postgres into your local docker repo. We also recommend running a vector DB in locally, so make sure to pull the image of the one you are using. Ourdocker-composefile will spin up qdrant by default and create two index/collections. If you plan to run Milvus or Weaviate, you will have to configure them on your own.docker pull rabbitmq
docker pull postgres
docker pull qdrant/qdrant | docker pull milvusdb/milvus | docker pull semitechnologies/weaviateThen run:docker-compose build --no-cache
docker-compose up -dNote that thedb-initcontainer is running a script that sets up the database schema and will stop after the script completes.3) (optional) Configure Sentence Transformer open face models.VectorFlow can run any Sentence Transformer model but thedocker-composefile will not spin it up automatically. Either runapp.py --model_name your-sentence-transformer-model, or build and run the docker image insrc/hugging_facewith:docker build --file hugging_face/Dockerfile -t vectorflow_hf:latest .
docker run --network=vectorflow --name=vectorflow_hf -d --env-file=/path/to/.env vectorflow_hf:latest --model_name "your_model_name_here"Note that the Sentence Transformer models can be large and take several minutes to download from Hugging Face. VectorFlow does not provision hardware, so you must ensure your hardware has enough RAM/VRAM for the model. By default, VectorFlow will run models on GPU with CUDA if available.Using VectorFlowTo use VectorFlow in a live system, make an HTTP request to your API's URL at port 8000 - for example,localhost:8000from your development machine, orvectorflow_api:8000from within another docker container.Request & Response PayloadAll requests require an HTTP Header withAuthorizationkey which is the same as yourINTERNAL_API_KEYenv var that you defined before (see above). You must pass your vector database api key with the HTTP HeaderX-VectorDB-Keyif you are running a connecting to a cloud-based instance of a vector DB, and the embedding api key withX-EmbeddingAPI-Keyif you are using OpenAI. HuggingFace Sentence Transformer embeddings do not require an api key, but you must follow the above steps to run the container with the model you need.VectorFlow currently supports Pinecone, Qdrant, Weaviate, and Milvus vector databases.To check the status of ajob, make aGETrequest to this endpoint:/jobs/<int:job_id>/status. The response will be in the form:{
    'JobStatus': job_status.value
}To submit ajobfor embedding, make aPOSTrequest to the/embedendpoint with a file attached, the'Content-Type: multipart/form-data'header and the following payload:{
    'SourceData=path_to_txt_file'
    'LinesPerBatch=4096'
    'EmbeddingsMetadata={
        "embeddings_type": "OPEN_AI | HUGGING_FACE | IMAGE",
        "chunk_size": 512,
        "chunk_overlap": 128,
        "chunk_strategy": "EXACT | PARAGRAPH | SENTENCE",
        "hugging_face_model_name": "sentence-transformer-model-name-here"
    }'
    'VectorDBMetadata={
        "vector_db_type": "PINECONE | QDRANT | WEAVIATE | MILVUS | REDIS",
        "index_name": "index_name",
        "environment": "env_name"
    }'
    'DocumentID=your-optional-internal-tracking-id'
}You will get the following payload back:{
    message': f"Successfully added {batch_count} batches to the queue",
    'JobID': job_id
}VectorFlow API ClientThe easiest way to use VectorFlow is with the our clients, located inclients/directory. There are several scripts, with different configurations for qickly uploading data. We recommmend starting with theclients/standard_upload_client.py- Running this script will submit a document to VectorFlow for embedding with Open AI ADA and upload to the local qdrant instance. You can change the values to match your configuration.Note that theTESTING_ENVvariable is the equivalent of theenviromentfield in theVectorDBMetadata, which corresponds to an environment in Pincone, a class in Weaviate, a collection in qdrant, etc.Vector Database Schema StandardVectorFlow enforces a standardized schema for uploading data to a vector store:id: string
source_data: string
source_document: string
embeddings: float arrayThe id can be used for deduplication and idempotency. Please note for Weaviate, the id is calledvectorflow_id. We plan to support dynamically detected and/or configurable schemas down the road.Raw Embeddings WebhookIf you wish to use VectorFlow only for chunking and generating embeddings, pass aWebhookURLparameter in the body of the/embedrequest and aX-Webhook-Keyas a header. VectorFlow assumes a webhook key is required for writing back to any endpoint. The embeddings are sent back along with the source chunks in tuple or lis of (source_chunk, embedding). This is sent as json with the following form:{
    'Embeddings': list[tuple(str, list[float])],
    'DocumentID': str,
    'JobID': int
}Chunk Validation WebhookIf you wish to validate which chunks you wish to embed, pass aChunkValidationURLparameter in the body of the/embedrequest. This will send the request with the following json payload,{"chunks": chunked_data}, wherechunked_datais a list of strings. It will expected back json containing keyvalid_chunkswith a list of valid chunks for embedding. This endpoint will timeout after 30 seconds by default but can be configured in the application code.S3 EndpointVectorFlow is integrated with AWS s3. You can pass a pre-signed s3 URL in the body of the HTTP instead of a file. Use the form fieldPreSignedURLand hit the endpoint/s3. This endpoint has the same configuration and restrictions as the/embedendpoint.Image Embedding & Similarity SearchVectorFlow can embed images using theImage2Vec library. It will create a 512 dimensional vector by default from any image. You can also perform a Top-K image similarity search.How to Use Image UploadSend a POST request to the/imagesendpoint to utilize this capability. You pass the same set of fields inVectorDBMetadataas you would for the/embedor/s3endpoints but forEmbeddingsMetadatayou only need to pass"embeddings_type": "IMAGE"Thedocker-composefile will not spin this capability up automatically. Either runapp.py --model_name your-sentence-transformer-model, or build and run the docker image insrc/imageswith:docker build --file images/Dockerfile -t vectorflow_image_embedding:latest .
docker run --network=vectorflow --name=vectorflow_image_embedding -d --env-file=/path/to/.env vectorflow_image_embedding:latestNote that the Image2Vec uses the ResNet 18 by default. You can configure it to run larger models with higher dimensionality for more fine-grained embeddings. These take several minutes to download the weights from the source. VectorFlow does not provision hardware, so you must ensure your hardware has enough RAM/VRAM for the model. By default, VectorFlow will run models on GPU with CUDA if available.How to Use Image SearchPerforming an image similarity search requires running another service. You can build and run it using docker:docker build --file images/Dockerfile.search -t vectorflow_image_search:latest .
docker run --network=vectorflow --name=vectorflow_image_search -d --env-file=/path/to/.env vectorflow_image_search:latestRequestTo perform a search, send aPOSTrequest to/images/searchendpoint with an image file attached, the'Content-Type: multipart/form-data'header and the following body:{
    'ReturnVectors': boolean,
    'TopK': integer, less than 1000,
    'VectorDBMetadata={
        "vector_db_type": "PINECONE | QDRANT | WEAVIATE | MILVUS | REDIS",
        "index_name": "index_name",
        "environment": "env_name"
    }'
}All requests require an HTTP Header withAuthorizationkey which is the same as yourINTERNAL_API_KEYenv var that you defined before (see above). You must pass your vector database api key with the HTTP HeaderX-VectorDB-Keyif you are running a connecting to a cloud-based instance of a vector DB.ResponseAn image similarity search will return a response object containing the top K matches, plus the raw vectors if requested, with of the following form:{
    "similar_images": list of match objects
    "vectors": list of list of floats
}where `match`` objects are defined as:{
    "id": str,
    "score": float,
    "metadata": {"source_document" : str}
}ContributingWe love feedback from the community. If you have an idea of how to make this project better, we encourage you to open an issue or join our Discord. Please tagdgarnitzanddanmeier2.Our roadmap is outlined in the section below and we would love help in building it out. Our open issues are a great place to start and can be viewedhere. If you want to work on something not listed there, we recommend you open an issue with a proposed approach in mind before submitting a PR.Please tagdgarnitzon all PRs and update the README to reflect your changes.TestingWhen submitting a PR, please add units tests to cover the functionality you have added. Please re-run existing tests to ensure there are no regressive bugs. Run from thesrcdirectory. To run an individual test use:python -m unittest module.tests.test_file.TestClass.test_methodTo run all the tests in the file use:python -m unittest module.tests.test_fileFor end-to-end testing, it is recommend to build and run using the docker-compose, but take down the container you are altering and run it locally on your development machine. This will avoid the need to constantly rebuild the images and re-run the containers. Make sure to change the environment variables in your development machine terminal to the correct values (i.e.localhostinstead ofrabbitmqorpostgres) so that the docker containers can communicate with your development machine. Once it works locally you can perform a final test with everything in docker-compose.VerifyingPlease verify that all changes work with docker-compose before opening a PR.We also recommend you add verification evidence, such as screenshots, that show that your code works in an end to end flow.RoadmapSupport for multi-file, directory data ingestion from sources such as Salesforce, Google Drive, etcRetry mechanismLangchain & Llama Index integrationsSupport callbacks for writing object metadata to a separate storeDynamically configurable vector DB schemasDeduplication capabilitiesVector version control"Smart" chunker"Smart" metadata extractor