Package: infinity-emb
Infinity ♾️Infinity is a high-throughput, low-latency REST API for serving vector embeddings, supporting a wide range of sentence-transformer models and frameworks. Infinity is developed underMIT Licenceand supported byGradient.ai.Why Infinity:Infinity provides the following features:Deploy virtually any SentenceTransformer- deploy the model you know fromSentenceTransformersFast inference backends: The inference server is built on top oftorch,fastembed(onnx-cpu)andCTranslate2, using FlashAttention to get the most out of yourCUDA,CPUorMPShardware.Dynamic batching: New embedding requests are queued while GPU is busy with the previous ones. New requests are squeezed intro your GPU/CPU as soon as ready. Similar max throughput on GPU as text-embeddings-inference.Correct and tested implementation: Unit and end-to-end tested. Embeddings via infinity are identical toSentenceTransformers(up to numerical precision). Lets API users create embeddings till infinity and beyond.Easy to use: The API is built on top ofFastAPI,Swaggermakes it fully documented. API are aligned toOpenAI's Embedding specs. See below on how to get started.Infinity demo:In this gif below, we usesentence-transformers/all-MiniLM-L6-v2, deployed at batch-size=2. After initialization, from a second terminal 3 requests  (payload 1,1,and 5 sentences) are sent via cURL.Getting startedInstall via pippipinstallinfinity-emb[all]Install from source with PoetryAdvanced:
To install via Poetry use Poetry 1.7.1, Python 3.11 on Ubuntu 22.04gitclonehttps://github.com/michaelfeil/infinitycdinfinitycdlibs/infinity_emb
poetryinstall--extrasallLaunch the CLI using a pre-built docker container (recommended)model=BAAI/bge-small-en-v1.5port=7997dockerrun-it--gpusall-p$port:$portmichaelf34/infinity:latest--model-name-or-path$model--port$portThe download path at runtime, can be controlled via the environment variableSENTENCE_TRANSFORMERS_HOME.or launch the cli after the pip installAfter your pip install, with your venv activate, you can run the CLI directly.
Check the--helpcommand to get a description for all parameters.infinity_emb--helpor launch it via PythonYou can use in a async context with asyncio.
This gives you most flexibility, but is a bit more advanced.importasynciofrominfinity_embimportAsyncEmbeddingEngine,EngineArgssentences=["Embed this is sentence via Infinity.","Paris is in France."]engine=AsyncEmbeddingEngine.from_args(EngineArgs(model_name_or_path="BAAI/bge-small-en-v1.5",engine="torch"))asyncdefmain():asyncwithengine:# engine starts with engine.astart()embeddings,usage=awaitengine.embed(sentences=sentences)# engine stops with engine.astop()asyncio.run(main())or launch thecreate_server()command from PythonThis executes the same command as the cli.
If you really don't enjoy to use the CLI, you can get the same options from python.frominfinity_embimportcreate_serverfastapi_app=create_server(**cli_kwargs)or launch on the cloud via dstackdstack allows you to provision a VM instance on the cloud of your choice. Write a service configuration file as below for the deployment ofBAAI/bge-small-en-v1.5model wrapped in Infinity.type:serviceimage:michaelf34/infinity:latestenv:-MODEL_ID=BAAI/bge-small-en-v1.5commands:-infinity_emb --model-name-or-path $MODEL_ID --port 80port:80Then, simply run the following dstack command. After this, a prompt will appear to let you choose which VM instance to deploy the Infinity.dstackrun.-finfinity/serve.dstack.yml--gpu16GBFor more detailed tutorial and general information about dstack, visit theofficial doc.Non-embedding featuresYou can also use rerank (beta):Reranking gives you a score for similarity between a query and multiple documents.
Use it in conjunction with a VectorDB+Embeddings, or as standalone for small amount of documents.importasynciofrominfinity_embimportAsyncEmbeddingEngine,EngineArgsquery="What is the python package infinity_emb?"docs=["This is a document not related to the python package infinity_emb, hence...","Paris is in France!","infinity_emb is a package for sentence embeddings and rerankings using transformer models in Python!"]engine_args=EngineArgs(model_name_or_path="BAAI/bge-small-en-v1.5",engine="torch")engine=AsyncEmbeddingEngine.from_args(engine_args)asyncdefmain():asyncwithengine:ranking,usage=awaitengine.rerank(query=query,docs=docs)print(list(zip(ranking,docs)))asyncio.run(main())You can also use text-classification (beta):Note: PR's to speed this section up are welcome, a 40% speedup is propable, currently the backend uses huggingface pipelines + dynamic batching.importasynciofrominfinity_embimportAsyncEmbeddingEngine,EngineArgssentences=["This is awesome.","I am bored."]engine_args=EngineArgs(model_name_or_path="SamLowe/roberta-base-go_emotions",engine="torch",model_warmup=True)engine=AsyncEmbeddingEngine.from_args(engine_args)asyncdefmain():asyncwithengine:predictions,usage=awaitengine.classify(sentences=sentences)returnpredictions,usageasyncio.run(main())Launch FAQ:What are embedding models?Embedding models can map any text to a low-dimensional dense vector which can be used for tasks like retrieval, classification, clustering, or semantic search. 
  And it also can be used in vector databases for LLMs.The most know architecture are encoder-only transformers such as BERT, and most popular implementation includeSentenceTransformers.What models are supported?All models of the sentence transformers org are supportedhttps://huggingface.co/sentence-transformers/ sbert.net.
LLM's like LLAMA2-7B are not intended for deployment.With the command--engine torchthe model must be compatible withhttps://github.com/UKPLab/sentence-transformers/.
- only models from Huggingface are supported.With the command--engine ctranslate2- onlyBERTmodels are supported.
- only models from Huggingface are supported.For the latest trends, you might want to check out one of the following models.https://huggingface.co/spaces/mteb/leaderboardLaunching multiple models in one dockerfileMultiple models on one GPU is in experimental mode. You can use the following temporary solution:FROMmichaelf34/infinity:latest# Dockerfile-ENTRYPOINT for multiple models via multiple portsENTRYPOINT["/bin/sh","-c",\"(. /app/.venv/bin/activate && infinity_emb --port 8080 --model-name-or-path sentence-transformers/all-MiniLM-L6-v2 &);\(. /app/.venv/bin/activate && infinity_emb --port 8081 --model-name-or-path intfloat/e5-large-v2 )"]You can build and run it via:dockerbuild-tcustominfinity.&&dockerrun-it--gpusall-p8080:8080-p8081:8081custominfinityBoth models now run on two instances in one dockerfile servers. Otherwise, you could build your own FastAPI/flask instance, which wraps around the Async API.Using Langchain with InfinityInfinity has a official integration intopip install langchain>=0.342.
You can find more documentation on that here:https://python.langchain.com/docs/integrations/text_embedding/infinityfromlangchain.embeddings.infinityimportInfinityEmbeddingsfromlangchain.docstore.documentimportDocumentdocuments=[Document(page_content="Hello world!",metadata={"source":"unknown"})]emb_model=InfinityEmbeddings(model="BAAI/bge-small",infinity_api_url="http://localhost:7997/v1")print(emb_model.embed_documents([doc.page_contentfordocindocs]))DocumentationAfter startup, the Swagger Ui will be available under{url}:{port}/docs, in this casehttp://localhost:7997/docs.Contribute and DevelopInstall via Poetry 1.7.1 and Python3.11 on Ubuntu 22.04cdlibs/infinity_emb
poetryinstall--extrasall--withtestTo pass the CI:cdlibs/infinity_emb
makeformat
makelint
poetryrunpytest./testsAll contributions must be made in a way to be compatible with the Apache 2 OSS License.