Package: openllm
🦾 OpenLLMAn open platform for operating large language models (LLMs) in production.Fine-tune, serve, deploy, and monitor any LLMs with ease.📖 IntroductionOpenLLM is an open-source platform designed to facilitate the deployment and operation of large language models (LLMs) in real-world applications. With OpenLLM, you can run inference on any open-source LLM, deploy them on the cloud or on-premises, and build powerful AI applications.Key features include:🚂State-of-the-art LLMs: Integrated support for a wide range of open-source LLMs and model runtimes, including but not limited to Llama 2, StableLM, Falcon, Dolly, Flan-T5, ChatGLM, and StarCoder.🔥Flexible APIs: Serve LLMs over a RESTful API or gRPC with a single command. You can interact with the model using a Web UI, CLI, Python/JavaScript clients, or any HTTP client of your choice.⛓️Freedom to build: First-class support for LangChain, BentoML, OpenAI endpoints, and Hugging Face, allowing you to easily create your own AI applications by composing LLMs with other models and services.🎯Streamline deployment: Automatically generate your LLM server Docker images or deploy as serverless endpoints via☁️ BentoCloud, which effortlessly manages GPU resources, scales according to traffic, and ensures cost-effectiveness.🤖️Bring your own LLM: Fine-tune any LLM to suit your needs. You can load LoRA layers to fine-tune models for higher accuracy and performance for specific tasks. A unified fine-tuning API for models (LLM.tuning()) is coming soon.⚡Quantization: Run inference with less computational and memory costs with quantization techniques such asLLM.int8,SpQR (int4),AWQ,GPTQ, andSqueezeLLM.📡Streaming: Support token streaming through server-sent events (SSE). You can use the/v1/generate_streamendpoint for streaming responses from LLMs.🔄Continuous batching: Support continuous batching viavLLMfor increased total throughput.OpenLLM is designed for AI application developers working to build production-ready applications based on LLMs. It delivers a comprehensive suite of tools and features for fine-tuning, serving, deploying, and monitoring these models, simplifying the end-to-end deployment workflow for LLMs.💾 TL/DRFor starter, we provide two ways to quickly try out OpenLLM:Jupyter NotebooksTry thisOpenLLM tutorial in Google Colab: Serving Llama 2 with OpenLLM.DockerWe provide a docker container that helps you start running OpenLLM:dockerrun--rm-it-p3000:3000ghcr.io/bentoml/openllmstartfacebook/opt-1.3b--backendpt[!NOTE]
Given you have access to GPUs and have setupnvidia-docker,  you can additionally pass in--gpusto use GPU for faster inference and optimizationdockerrun--rm--gpusall-p3000:3000-itghcr.io/bentoml/openllmstartHuggingFaceH4/zephyr-7b-beta--backendvllm🏃 Get startedThe following provides instructions for how to get started with OpenLLM locally.PrerequisitesYou have installed Python 3.8 (or later) andpip. We highly recommend using aVirtual Environmentto prevent package conflicts.Install OpenLLMInstall OpenLLM by usingpipas follows:pipinstallopenllmTo verify the installation, run:$openllm-h

Usage:openllm[OPTIONS]COMMAND[ARGS]...██████╗██████╗███████╗███╗██╗██╗██╗███╗███╗██╔═══██╗██╔══██╗██╔════╝████╗██║██║██║████╗████║██║██║██████╔╝█████╗██╔██╗██║██║██║██╔████╔██║██║██║██╔═══╝██╔══╝██║╚██╗██║██║██║██║╚██╔╝██║╚██████╔╝██║███████╗██║╚████║███████╗███████╗██║╚═╝██║╚═════╝╚═╝╚══════╝╚═╝╚═══╝╚══════╝╚══════╝╚═╝╚═╝.Anopenplatformforoperatinglargelanguagemodelsinproduction.Fine-tune,serve,deploy,andmonitoranyLLMswithease.

Options:-v,--versionShowtheversionandexit.-h,--helpShowthismessageandexit.

Commands:buildPackageagivenmodelsintoaBentoLLM.importSetupLLMinteractively.modelsListallsupportedmodels.pruneRemoveallsavedmodels,(andoptionallybentos)builtwithOpenLLMlocally.queryQueryaLLMinteractively,fromaterminal.startStartaLLMServerforanysupportedLLM.start-grpcStartagRPCLLMServerforanysupportedLLM.

Extensions:build-base-containerBaseimagebuilderforBentoLLM.dive-bentosDiveintoaBentoLLM.get-containerfileReturnContainerfileofanygivenBento.get-promptGetthedefaultpromptusedbyOpenLLM.list-bentosListavailablebentosbuiltbyOpenLLM.list-modelsThisisequivalenttoopenllmmodels...playgroundOpenLLMPlayground.Start a LLM serverOpenLLM allows you to quickly spin up an LLM server usingopenllm start. For example, to start aphi-2server, run the following:TRUST_REMOTE_CODE=Trueopenllmstartmicrosoft/phi-2This starts the server athttp://0.0.0.0:3000/. OpenLLM downloads the model to the BentoML local Model Store if it has not been registered before. To view your local models, runbentoml models list.To interact with the server, you can visit the web UI athttp://0.0.0.0:3000/or send a request usingcurl. You can also use OpenLLM’s built-in Python client to interact with the server:importopenllmclient=openllm.client.HTTPClient('http://localhost:3000')client.query('Explain to me the difference between "further" and "farther"')Alternatively, use theopenllm querycommand to query the model:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'Explain to me the difference between "further" and "farther"'OpenLLM seamlessly supports many models and their variants. You can specify different variants of the model to be served. For example:openllmstart<model_id>--<options>[!NOTE]
OpenLLM supports specifying fine-tuning weights and quantized weights
for any of the supported models as long as they can be loaded with the model
architecture. Use theopenllm modelscommand to see the complete list of supported
models, their architectures, and their variants.[!IMPORTANT]
If you are testing OpenLLM on CPU, you might want to pass inDTYPE=float32. By default,
OpenLLM will set modeldtypetobfloat16for the best performance.DTYPE=float32openllmstartmicrosoft/phi-2This will also applies to older GPUs. If your GPUs doesn't supportbfloat16, then you also
want to setDTYPE=float16.🧩 Supported modelsOpenLLM currently supports the following models. By default, OpenLLM doesn't include dependencies to run all models. The extra model-specific dependencies can be installed with the instructions below.BaichuanQuickstartNote:Baichuan requires to install with:pipinstall"openllm[baichuan]"Run the following command to quickly spin up a Baichuan server:TRUST_REMOTE_CODE=Trueopenllmstartbaichuan-inc/baichuan-7bIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any Baichuan variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more Baichuan-compatible models.Supported modelsYou can specify any of the following Baichuan models viaopenllm start:baichuan-inc/baichuan2-7b-basebaichuan-inc/baichuan2-7b-chatbaichuan-inc/baichuan2-13b-basebaichuan-inc/baichuan2-13b-chatSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"TRUST_REMOTE_CODE=Trueopenllmstartbaichuan-inc/baichuan2-7b-base--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:TRUST_REMOTE_CODE=Trueopenllmstartbaichuan-inc/baichuan2-7b-base--backendptChatGLMQuickstartNote:ChatGLM requires to install with:pipinstall"openllm[chatglm]"Run the following command to quickly spin up a ChatGLM server:TRUST_REMOTE_CODE=Trueopenllmstartthudm/chatglm-6bIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any ChatGLM variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more ChatGLM-compatible models.Supported modelsYou can specify any of the following ChatGLM models viaopenllm start:thudm/chatglm-6bthudm/chatglm-6b-int8thudm/chatglm-6b-int4thudm/chatglm2-6bthudm/chatglm2-6b-int4thudm/chatglm3-6bSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"TRUST_REMOTE_CODE=Trueopenllmstartthudm/chatglm-6b--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:TRUST_REMOTE_CODE=Trueopenllmstartthudm/chatglm-6b--backendptDollyV2QuickstartRun the following command to quickly spin up a DollyV2 server:openllmstartdatabricks/dolly-v2-3bIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any DollyV2 variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more DollyV2-compatible models.Supported modelsYou can specify any of the following DollyV2 models viaopenllm start:databricks/dolly-v2-3bdatabricks/dolly-v2-7bdatabricks/dolly-v2-12bSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"openllmstartdatabricks/dolly-v2-3b--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:openllmstartdatabricks/dolly-v2-3b--backendptFalconQuickstartNote:Falcon requires to install with:pipinstall"openllm[falcon]"Run the following command to quickly spin up a Falcon server:openllmstarttiiuae/falcon-7bIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any Falcon variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more Falcon-compatible models.Supported modelsYou can specify any of the following Falcon models viaopenllm start:tiiuae/falcon-7btiiuae/falcon-40btiiuae/falcon-7b-instructtiiuae/falcon-40b-instructSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"openllmstarttiiuae/falcon-7b--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:openllmstarttiiuae/falcon-7b--backendptFlanT5QuickstartRun the following command to quickly spin up a FlanT5 server:openllmstartgoogle/flan-t5-largeIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any FlanT5 variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more FlanT5-compatible models.Supported modelsYou can specify any of the following FlanT5 models viaopenllm start:google/flan-t5-smallgoogle/flan-t5-basegoogle/flan-t5-largegoogle/flan-t5-xlgoogle/flan-t5-xxlSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.PyTorch:openllmstartgoogle/flan-t5-small--backendptGPTNeoXQuickstartRun the following command to quickly spin up a GPTNeoX server:openllmstarteleutherai/gpt-neox-20bIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any GPTNeoX variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more GPTNeoX-compatible models.Supported modelsYou can specify any of the following GPTNeoX models viaopenllm start:eleutherai/gpt-neox-20bSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"openllmstarteleutherai/gpt-neox-20b--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:openllmstarteleutherai/gpt-neox-20b--backendptLlamaQuickstartRun the following command to quickly spin up a Llama server:openllmstartNousResearch/llama-2-7b-hfIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any Llama variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more Llama-compatible models.Supported modelsYou can specify any of the following Llama models viaopenllm start:meta-llama/Llama-2-70b-chat-hfmeta-llama/Llama-2-13b-chat-hfmeta-llama/Llama-2-7b-chat-hfmeta-llama/Llama-2-70b-hfmeta-llama/Llama-2-13b-hfmeta-llama/Llama-2-7b-hfNousResearch/llama-2-70b-chat-hfNousResearch/llama-2-13b-chat-hfNousResearch/llama-2-7b-chat-hfNousResearch/llama-2-70b-hfNousResearch/llama-2-13b-hfNousResearch/llama-2-7b-hfSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"openllmstartmeta-llama/Llama-2-70b-chat-hf--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:openllmstartmeta-llama/Llama-2-70b-chat-hf--backendptMistralQuickstartRun the following command to quickly spin up a Mistral server:openllmstartmistralai/Mistral-7B-Instruct-v0.1In a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any Mistral variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more Mistral-compatible models.Supported modelsYou can specify any of the following Mistral models viaopenllm start:HuggingFaceH4/zephyr-7b-alphaHuggingFaceH4/zephyr-7b-betamistralai/Mistral-7B-Instruct-v0.2mistralai/Mistral-7B-Instruct-v0.1mistralai/Mistral-7B-v0.1Supported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"openllmstartHuggingFaceH4/zephyr-7b-alpha--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:openllmstartHuggingFaceH4/zephyr-7b-alpha--backendptMixtralQuickstartRun the following command to quickly spin up a Mixtral server:openllmstartmistralai/Mixtral-8x7B-Instruct-v0.1In a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any Mixtral variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more Mixtral-compatible models.Supported modelsYou can specify any of the following Mixtral models viaopenllm start:mistralai/Mixtral-8x7B-Instruct-v0.1mistralai/Mixtral-8x7B-v0.1Supported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"openllmstartmistralai/Mixtral-8x7B-Instruct-v0.1--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:openllmstartmistralai/Mixtral-8x7B-Instruct-v0.1--backendptMPTQuickstartNote:MPT requires to install with:pipinstall"openllm[mpt]"Run the following command to quickly spin up a MPT server:TRUST_REMOTE_CODE=Trueopenllmstartmosaicml/mpt-7b-instructIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any MPT variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more MPT-compatible models.Supported modelsYou can specify any of the following MPT models viaopenllm start:mosaicml/mpt-7bmosaicml/mpt-7b-instructmosaicml/mpt-7b-chatmosaicml/mpt-7b-storywritermosaicml/mpt-30bmosaicml/mpt-30b-instructmosaicml/mpt-30b-chatSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"TRUST_REMOTE_CODE=Trueopenllmstartmosaicml/mpt-7b--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:TRUST_REMOTE_CODE=Trueopenllmstartmosaicml/mpt-7b--backendptOPTQuickstartRun the following command to quickly spin up a OPT server:openllmstartfacebook/opt-1.3bIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any OPT variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more OPT-compatible models.Supported modelsYou can specify any of the following OPT models viaopenllm start:facebook/opt-125mfacebook/opt-350mfacebook/opt-1.3bfacebook/opt-2.7bfacebook/opt-6.7bfacebook/opt-66bSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"openllmstartfacebook/opt-125m--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:openllmstartfacebook/opt-125m--backendptPhiQuickstartRun the following command to quickly spin up a Phi server:TRUST_REMOTE_CODE=Trueopenllmstartmicrosoft/phi-1_5In a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any Phi variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more Phi-compatible models.Supported modelsYou can specify any of the following Phi models viaopenllm start:microsoft/phi-1_5Supported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"TRUST_REMOTE_CODE=Trueopenllmstartmicrosoft/phi-1_5--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:TRUST_REMOTE_CODE=Trueopenllmstartmicrosoft/phi-1_5--backendptQwenQuickstartNote:Qwen requires to install with:pipinstall"openllm[qwen]"Run the following command to quickly spin up a Qwen server:TRUST_REMOTE_CODE=Trueopenllmstartqwen/Qwen-7B-ChatIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any Qwen variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more Qwen-compatible models.Supported modelsYou can specify any of the following Qwen models viaopenllm start:qwen/Qwen-7B-Chatqwen/Qwen-7B-Chat-Int8qwen/Qwen-7B-Chat-Int4qwen/Qwen-14B-Chatqwen/Qwen-14B-Chat-Int8qwen/Qwen-14B-Chat-Int4Supported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"TRUST_REMOTE_CODE=Trueopenllmstartqwen/Qwen-7B-Chat--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:TRUST_REMOTE_CODE=Trueopenllmstartqwen/Qwen-7B-Chat--backendptStableLMQuickstartRun the following command to quickly spin up a StableLM server:openllmstartstabilityai/stablelm-tuned-alpha-3bIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any StableLM variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more StableLM-compatible models.Supported modelsYou can specify any of the following StableLM models viaopenllm start:stabilityai/stablelm-tuned-alpha-3bstabilityai/stablelm-tuned-alpha-7bstabilityai/stablelm-base-alpha-3bstabilityai/stablelm-base-alpha-7bSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"openllmstartstabilityai/stablelm-tuned-alpha-3b--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:openllmstartstabilityai/stablelm-tuned-alpha-3b--backendptStarCoderQuickstartNote:StarCoder requires to install with:pipinstall"openllm[starcoder]"Run the following command to quickly spin up a StarCoder server:openllmstartbigcode/starcoderIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any StarCoder variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more StarCoder-compatible models.Supported modelsYou can specify any of the following StarCoder models viaopenllm start:bigcode/starcoderbigcode/starcoderbaseSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"openllmstartbigcode/starcoder--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:openllmstartbigcode/starcoder--backendptYiQuickstartRun the following command to quickly spin up a Yi server:TRUST_REMOTE_CODE=Trueopenllmstart01-ai/Yi-6BIn a different terminal, run the following command to interact with the server:exportOPENLLM_ENDPOINT=http://localhost:3000
openllmquery'What are large language models?'Note:Any Yi variants can be deployed with OpenLLM. Visit theHuggingFace Model Hubto see more Yi-compatible models.Supported modelsYou can specify any of the following Yi models viaopenllm start:01-ai/Yi-6B01-ai/Yi-34B01-ai/Yi-6B-200K01-ai/Yi-34B-200KSupported backendsOpenLLM will support vLLM and PyTorch as default backend. By default, it will use vLLM if vLLM is available, otherwise fallback to PyTorch.Important:We recommend user to explicitly specify--backendto choose the desired backend to run the model. If you have access to a GPU, always use--backend vllm.vLLM (Recommended):To install vLLM, runpip install "openllm[vllm]"TRUST_REMOTE_CODE=Trueopenllmstart01-ai/Yi-6B--backendvllmImportant:Using vLLM requires a GPU that has architecture newer than 8.0 to get the best performance for serving. It is recommended that for all serving usecase in production, you should choose vLLM for serving.Note:Currently, adapters are yet to be supported with vLLM.PyTorch:TRUST_REMOTE_CODE=Trueopenllmstart01-ai/Yi-6B--backendptMore models will be integrated with OpenLLM and we welcome your contributions if you want to incorporate your custom LLMs into the ecosystem. Check outAdding a New Model Guideto learn more.💻 Run your model on multiple GPUsOpenLLM allows you to start your model server on multiple GPUs and specify the number of workers per resource assigned using the--workers-per-resourceoption. For example, if you have 4 available GPUs, you set the value as one divided by the number as only one instance of the Runner server will be spawned.TRUST_REMOTE_CODE=Trueopenllmstartmicrosoft/phi-2--workers-per-resource0.25[!NOTE]
The amount of GPUs required depends on the model size itself.
You can usethe Model Memory Calculator from Hugging Faceto
calculate how much vRAM is needed to train and perform big model
inference on a model and then plan your GPU strategy based on it.When using the--workers-per-resourceoption with theopenllm buildcommand, the environment variable is saved into the resulting Bento.For more information, seeResource scheduling strategy.🛞 Runtime implementationsDifferent LLMs may support multiple runtime implementations. Models that havevLLM(vllm) supports will use vLLM by default, otherwise it fallback to usePyTorch(pt).To specify a specific runtime for your chosen model, use the--backendoption. For example:openllmstartmeta-llama/Llama-2-7b-chat-hf--backendvllmNote:To use the vLLM backend, you need a GPU with at least the Ampere architecture or newer and CUDA version 11.8.To see the backend options of each model supported by OpenLLM, see the Supported models section or runopenllm models.📐 QuantizationQuantization is a technique to reduce the storage and computation requirements for machine learning models, particularly during inference. By approximating floating-point numbers as integers (quantized values), quantization allows for faster computations, reduced memory footprint, and can make it feasible to deploy large models on resource-constrained devices.OpenLLM supports the following quantization techniquesLLM.int8(): 8-bit Matrix MultiplicationthroughbitsandbytesSpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight CompressionthroughbitsandbytesAWQ: Activation-aware Weight Quantization,GPTQ: Accurate Post-Training QuantizationSqueezeLLM: Dense-and-Sparse Quantization.PyTorch backendWith PyTorch backend, OpenLLM supportsint8,int4, andgptq.For using int8 and int4 quantization throughbitsandbytes, you can use the following command:TRUST_REMOTE_CODE=Trueopenllmstartmicrosoft/phi-2--quantizeint8To run inference withgptq, simply pass--quantize gptq:openllmstartTheBloke/Llama-2-7B-Chat-GPTQ--quantizegptq[!NOTE]
In order to run GPTQ, make sure you runpip install "openllm[gptq]"first to install the dependency. From the GPTQ paper, it is recommended to quantized the weights before serving.
SeeAutoGPTQfor more information on GPTQ quantization.vLLM backendWith vLLM backend, OpenLLM supportsawq,squeezellmTo run inference withawq, simply pass--quantize awq:openllmstartTheBloke/zephyr-7B-alpha-AWQ--quantizeawqTo run inference withsqueezellm, simply pass--quantize squeezellm:openllmstartsqueeze-ai-lab/sq-llama-2-7b-w4-s0--quantizesqueezellm--serializationlegacy[!IMPORTANT]
Since bothsqueezellmandawqare weight-aware quantization methods, meaning the quantization is done during training, all pre-trained weights needs to get quantized before inference time. Make sure to find compatible weights on HuggingFace Hub for your model of choice.🛠️ Serving fine-tuning layersPEFT, or Parameter-Efficient Fine-Tuning, is a methodology designed to fine-tune pre-trained models more efficiently. Instead of adjusting all model parameters, PEFT focuses on tuning only a subset, reducing computational and storage costs.LoRA(Low-Rank Adaptation) is one of the techniques supported by PEFT. It streamlines fine-tuning by using low-rank decomposition to represent weight updates, thereby drastically reducing the number of trainable parameters.With OpenLLM, you can take advantage of the fine-tuning feature by serving models with any PEFT-compatible layers using the--adapter-idoption. For example:openllmstartfacebook/opt-6.7b--adapter-idaarnphm/opt-6-7b-quotes:defaultOpenLLM also provides flexibility by supporting adapters from custom file paths:openllmstartfacebook/opt-6.7b--adapter-id/path/to/adapters:local_adapterTo use multiple adapters, use the following format:openllmstartfacebook/opt-6.7b--adapter-idaarnphm/opt-6.7b-lora:default--adapter-idaarnphm/opt-6.7b-french:french_loraBy default, all adapters will be injected into the models during startup. Adapters can be specified per request viaadapter_name:curl-X'POST'\'http://localhost:3000/v1/generate'\-H'accept: application/json'\-H'Content-Type: application/json'\-d'{"prompt": "What is the meaning of life?","stop": ["philosopher"],"llm_config": {"max_new_tokens": 256,"temperature": 0.75,"top_k": 15,"top_p": 1},"adapter_name": "default"}'To include this into the Bento, you can specify the--adapter-idoption when using theopenllm buildcommand:openllmbuildfacebook/opt-6.7b--adapter-id...If you use a relative path for--adapter-id, you need to add--build-ctx.openllmbuildfacebook/opt-6.7b--adapter-id./path/to/adapter_id--build-ctx.[!IMPORTANT]
Fine-tuning support is still experimental and currently only works with PyTorch backend. vLLM support is coming soon.🐍 Python SDKEach LLM can be instantiated withopenllm.LLM:importopenllmllm=openllm.LLM('microsoft/phi-2')The main inference API is the streaminggenerate_iteratormethod:asyncforgenerationinllm.generate_iterator('What is the meaning of life?'):print(generation.outputs[0].text)[!NOTE]
The motivation behind makingllm.generate_iteratoran async generator is to provide support for Continuous batching with vLLM backend. By having the async endpoints, each prompt
will be added correctly to the request queue to process with vLLM backend.There is also aone-shotgeneratemethod:awaitllm.generate('What is the meaning of life?')This method is easy to use for one-shot generation use case, but merely served as an example how to usellm.generate_iteratoras it usesgenerate_iteratorunder the hood.[!IMPORTANT]
If you need to call your code in a synchronous context, you can useasyncio.runthat wraps an async function:importasyncioasyncdefgenerate(prompt,**attrs):returnawaitllm.generate(prompt,**attrs)asyncio.run(generate("The meaning of life is",temperature=0.23))⚙️ IntegrationsOpenLLM is not just a standalone product; it's a building block designed to
integrate with other powerful tools easily. We currently offer integration withBentoML,OpenAI's Compatible Endpoints,LlamaIndex,LangChain, andTransformers Agents.OpenAI Compatible EndpointsOpenLLM Server can be used as a drop-in replacement for OpenAI's API. Simply
specify the base_url tollm-endpoint/v1and you are good to go:importopenaiclient=openai.OpenAI(base_url='http://localhost:3000/v1',api_key='na')# Here the server is running on localhost:3000completions=client.completions.create(prompt='Write me a tag line for an ice cream shop.',model=model,max_tokens=64,stream=stream)The compatible endpoints supports/completions,/chat/completions, and/models[!NOTE]
You can find out OpenAI example clients under theexamplesfolder.BentoMLOpenLLM LLM can be integrated as aRunnerin your
BentoML service. Simply callawait llm.generateto generate text. Note thatllm.generateusesrunnerunder the hood:importbentomlimportopenllmllm=openllm.LLM('microsoft/phi-2')svc=bentoml.Service(name='llm-phi-service',runners=[llm.runner])@svc.api(input=bentoml.io.Text(),output=bentoml.io.Text())asyncdefprompt(input_text:str)->str:generation=awaitllm.generate(input_text)returngeneration.outputs[0].textLlamaIndexTo start a local LLM withllama_index, simply usellama_index.llms.openllm.OpenLLM:importasynciofromllama_index.llms.openllmimportOpenLLMllm=OpenLLM('HuggingFaceH4/zephyr-7b-alpha')llm.complete('The meaning of life is')asyncdefmain(prompt,**kwargs):asyncforitinllm.astream_chat(prompt,**kwargs):print(it)asyncio.run(main('The time at San Francisco is'))If there is a remote LLM Server running elsewhere, then you can usellama_index.llms.openllm.OpenLLMAPI:fromllama_index.llms.openllmimportOpenLLMAPI[!NOTE]
All synchronous and asynchronous API fromllama_index.llms.LLMare supported.LangChainTo quickly start a local LLM withlangchain, simply do the following:fromlangchain.llmsimportOpenLLMllm=OpenLLM(model_name='llama',model_id='meta-llama/Llama-2-7b-hf')llm('What is the difference between a duck and a goose? And why there are so many Goose in Canada?')[!IMPORTANT]
By default, OpenLLM usesafetensorsformat for saving models.
If the model doesn't support safetensors, make sure to passserialisation="legacy"to use the legacy PyTorch bin format.langchain.llms.OpenLLMhas the capability to interact with remote OpenLLM
Server. Given there is an OpenLLM server deployed elsewhere, you can connect to
it by specifying its URL:fromlangchain.llmsimportOpenLLMllm=OpenLLM(server_url='http://44.23.123.1:3000',server_type='http')llm('What is the difference between a duck and a goose? And why there are so many Goose in Canada?')To integrate a LangChain agent with BentoML, you can do the following:llm=OpenLLM(model_id='google/flan-t5-large',embedded=False,serialisation='legacy')tools=load_tools(['serpapi','llm-math'],llm=llm)agent=initialize_agent(tools,llm,agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION)svc=bentoml.Service('langchain-openllm',runners=[llm.runner])@svc.api(input=Text(),output=Text())defchat(input_text:str):returnagent.run(input_text)[!NOTE]
You can find out more examples under theexamplesfolder.Transformers AgentsOpenLLM seamlessly integrates withTransformers Agents.[!WARNING]
The Transformers Agent is still at an experimental stage. It is
recommended to install OpenLLM withpip install -r nightly-requirements.txtto get the latest API update for HuggingFace agent.importtransformersagent=transformers.HfAgent('http://localhost:3000/hf/agent')# URL that runs the OpenLLM serveragent.run('Is the following `text` positive or negative?',text="I don't like how this models is generate inputs")🚀 Deploying models to productionThere are several ways to deploy your LLMs:🐳 Docker containerBuilding a Bento: With OpenLLM, you can easily build a Bento for a
specific model, likemistralai/Mistral-7B-Instruct-v0.1, using thebuildcommand.:openllmbuildmistralai/Mistral-7B-Instruct-v0.1ABento,
in BentoML, is the unit of distribution. It packages your program's source
code, models, files, artefacts, and dependencies.Containerize your Bentobentomlcontainerize<name:version>This generates a OCI-compatible docker image that can be deployed anywhere
docker runs. For best scalability and reliability of your LLM service in
production, we recommend deploy with BentoCloud。☁️ BentoCloudDeploy OpenLLM withBentoCloud, the
serverless cloud for shipping and scaling AI applications.Create a BentoCloud account:sign up herefor early accessLog into your BentoCloud account:bentomlcloudlogin--api-token<your-api-token>--endpoint<bento-cloud-endpoint>[!NOTE]
Replace<your-api-token>and<bento-cloud-endpoint>with your
specific API token and the BentoCloud endpoint respectively.Bulding a Bento: With OpenLLM, you can easily build a Bento for a
specific model, such asmistralai/Mistral-7B-Instruct-v0.1:openllmbuildmistralai/Mistral-7B-Instruct-v0.1Pushing a Bento: Push your freshly-built Bento service to BentoCloud via
thepushcommand:bentomlpush<name:version>Deploying a Bento: Deploy your LLMs to BentoCloud with a singlebentoml deployment createcommand following thedeployment instructions.👥 CommunityEngage with like-minded individuals passionate about LLMs, AI, and more on ourDiscord!OpenLLM is actively maintained by the BentoML team. Feel free to reach out and
join us in our pursuit to make LLMs more accessible and easy to use 👉Join our Slack community!🎁 ContributingWe welcome contributions! If you're interested in enhancing OpenLLM's
capabilities or have any questions, don't hesitate to reach out in ourdiscord channel.Checkout ourDeveloper Guideif you wish to contribute to OpenLLM's codebase.🍇 TelemetryOpenLLM collects usage data to enhance user experience and improve the product.
We only report OpenLLM's internal API calls and ensure maximum privacy by
excluding sensitive information. We will never collect user code, model data, or
stack traces. For usage tracking, check out thecode.You can opt out of usage tracking by using the--do-not-trackCLI option:openllm[command]--do-not-trackOr by setting the environment variableOPENLLM_DO_NOT_TRACK=True:exportOPENLLM_DO_NOT_TRACK=True📔 CitationIf you use OpenLLM in your research, we provide acitationto
use:@software{Pham_OpenLLM_Operating_LLMs_2023,author={Pham, Aaron and Yang, Chaoyu and Sheng, Sean and  Zhao, Shenyang and Lee, Sauyon and Jiang, Bo and Dong, Fog and Guan, Xipeng and Ming, Frost},license={Apache-2.0},month=jun,title={{OpenLLM: Operating LLMs in production}},url={https://github.com/bentoml/OpenLLM},year={2023}}Release InformationChangesBump vllm to 0.2.7 for a newly built bento#837Click me for full changelog