Package: xinference
Xorbits Inference: Model Serving Made Easy 🤖English |中文介绍|日本語Xorbits Inference(Xinference) is a powerful and versatile library designed to serve language,
speech recognition, and multimodal models. With Xorbits Inference, you can effortlessly deploy
and serve your or state-of-the-art built-in models using just a single command. Whether you are a
researcher, developer, or data scientist, Xorbits Inference empowers you to unleash the full
potential of cutting-edge AI models.👉 Join our Slack community!🔥 Hot TopicsFramework EnhancementsSupport speech recognition model:#929Metrics support:#906Docker image:#855Support multimodal:#829Auto recover:#694Function calling API:#701, here's example:https://github.com/xorbitsai/inference/blob/main/examples/FunctionCall.ipynbSupport rerank model:#672New ModelsBuilt-in support forYi-VL:#946Built-in support forWhisper:#929Built-in support forOrion-chat:#933Built-in support forInternLM2-chat:#829Built-in support forqwen-vl:#829Built-in support forphi-2:#828IntegrationsDify: an LLMOps platform that enables developers (and even non-developers) to quickly build useful applications based on large language models, ensuring they are visual, operable, and improvable.Chatbox: a desktop client for multiple cutting-edge LLM models, available on Windows, Mac and Linux.Key Features🌟Model Serving Made Easy: Simplify the process of serving large language, speech
recognition, and multimodal models. You can set up and deploy your models
for experimentation and production with a single command.⚡️State-of-the-Art Models: Experiment with cutting-edge built-in models using a single
command. Inference provides access to state-of-the-art open-source models!🖥Heterogeneous Hardware Utilization: Make the most of your hardware resources withggml. Xorbits Inference intelligently utilizes heterogeneous
hardware, including GPUs and CPUs, to accelerate your model inference tasks.⚙️Flexible API and Interfaces: Offer multiple interfaces for interacting
with your models, supporting OpenAI compatible RESTful API (including Function Calling API), RPC, CLI
and WebUI for seamless model management and interaction.🌐Distributed Deployment: Excel in distributed deployment scenarios,
allowing the seamless distribution of model inference across multiple devices or machines.🔌Built-in Integration with Third-Party Libraries: Xorbits Inference seamlessly integrates
with popular third-party libraries includingLangChain,LlamaIndex,Dify, andChatbox.Why XinferenceFeatureXinferenceFastChatOpenLLMRayLLMOpenAI-Compatible RESTful API✅✅✅✅vLLM Integrations✅✅✅✅More Inference Engines (GGML, TensorRT)✅❌✅✅More Platforms (CPU, Metal)✅✅❌❌Multi-node Cluster Deployment✅❌❌✅Image Models (Text-to-Image)✅✅❌❌Text Embedding Models✅❌❌❌Multimodal Models✅❌❌❌Audio Models✅❌❌❌More OpenAI Functionalities (Function Calling)✅❌❌❌Getting StartedPlease give us a star before you begin, and you'll receive instant notifications for every new release on GitHub!DocsBuilt-in ModelsCustom ModelsDeployment DocsExamples and TutorialsJupyter NotebookThe lightest way to experience Xinference is to try ourJuypter Notebook on Google Colab.DockerNvidia GPU users can start Xinference server usingXinference Docker Image. Prior to executing the installation command, ensure that bothDockerandCUDAare set up on your system.dockerrun--namexinference-d-p9997:9997-eXINFERENCE_HOME=/data-v</on/your/host>:/data--gpusallxprobe/xinference:latestxinference-local-H0.0.0.0Quick StartInstall Xinference by using pip as follows. (For more options, seeInstallation page.)pipinstall"xinference[all]"To start a local instance of Xinference, run the following command:$xinference-localOnce Xinference is running, there are multiple ways you can try it: via the web UI, via cURL,
via the command line, or via the Xinference’s python client. Check out ourdocsfor the guide.Getting involvedPlatformPurposeGithub IssuesReporting bugs and filing feature requests.SlackCollaborating with other Xorbits users.TwitterStaying up-to-date on new features.Contributors