Package: bentoml
BentoML: The Unified AI Application FrameworkBentoML is a framework for buildingreliable, scalable, and cost-efficient AI
applications. It comes with everything you need for model serving, application
packaging, and production deployment.👉 Join our Slack community!Highlights🍱 Bento is the container for AI appsOpen standard and SDK for AI apps, pack your code, inference pipelines, model
files, dependencies, and runtime configurations in aBento.Auto-generate API servers, supporting REST API, gRPC, and long-running
inference jobs.Auto-generate Docker container images.🏄 Freedom to build with any AI modelsImport from any model hub or bring your own models built with frameworks like
PyTorch, TensorFlow, Keras, Scikit-Learn, XGBoost and many more.Native support forLLM inference,generative AI,embedding creation, andmulti-modal AI apps.Run and debug your BentoML apps locally on Mac, Windows, or Linux.🤖️ Inference optimization for AI applicationsIntegrate with high-performance runtimes such as ONNX-runtime and TorchScript to boost response time and throughput.Support parallel processing of model inferences for improved speed and efficiency.Implement adaptive batching to optimize processing.Built-in optimization for specific model architectures (like OpenLLM for LLMs).🍭 Simplify modern AI application architecturePython-first! Effortlessly scale complex AI workloads.Enable GPU inference without the headache.Compose multiple models to run concurrently or sequentially, over multiple GPUs oron a Kubernetes Cluster.Natively integrates with MLFlow,LangChain,
Kubeflow, Triton, Spark, Ray, and many more to complete your production AI stack.🚀 Deploy anywhereOne-click deployment to☁️ BentoCloud, the
Serverless platform made for hosting and operating AI apps.Scalable BentoML deployment with🦄️ Yataion Kubernetes.Deploy auto-generated container images anywhere Docker runs.DocumentationInstallation:pip install bentomlDocumentation:docs.bentoml.comTutorial:Quickstart🛠️ What you can build with BentoMLOpenLLM- An open platform for operating large language models (LLMs) in production.BentoXTTS- Convert text to speech based on your custom audio data.BentoSDXLTurbo- Create an image generation application and run inference with a single step.BentoSD2Upscaler- Build an image generation application with upscaling capability.BentoControlNet- Influence image composition, adjust specific elements, and ensure spatial consistency by integrating ControlNet with your image generation process.BentoWhisperX- Convert spoken words into text for AI scenarios like virtual assistants, voice-controlled devices, and automated transcription services.Sentence Transformer- Transform text into numerical vectors for a variety of natural language processing (NLP) tasks.BentoCLIP- Build a CLIP (Contrastive Language-Image Pre-training) application for tasks like zero-shot learning, image classification, and image-text matching.BentoBLIP- Leverage BLIP (Bootstrapping Language Image Pre-training) to improve the way AI models understand and process the relationship between images and textual descriptions.BentoLCM- Deploy a REST API server for Stable Diffusion with Latent Consistency LoRAs.BentoSVD- Create a video generation application powered by Stable Video Diffusion (SVD).BentoVLLM- Accelerate your model inference and improve serving throughput by using vLLM as your LLM backend.Fraud Detection- Online model serving with custom XGBoost model.OCR as a Service- Turn any OCR models into online inference API endpoints.Replace Anything- Combine the power of Segment Anything and Stable Diffusion.DeepFloyd IF Multi-GPU serving- Serve IF models easily across multiple GPUs.Check out more exampleshere.Getting startedThis example demonstrates how to serve and deploy a simple text summarization application.Serving a model locallyInstall dependencies:pip install torch transformers "bentoml>=1.2.0a0"Define the serving logic of your model in aservice.pyfile.from__future__importannotationsimportbentomlfromtransformersimportpipeline@bentoml.service(resources={"cpu":"2"},traffic={"timeout":10},)classSummarization:def__init__(self)->None:# Load model into pipelineself.pipeline=pipeline('summarization')@bentoml.apidefsummarize(self,text:str)->str:result=self.pipeline(text)returnresult[0]['summary_text']Run this BentoML Service locally, which is accessible athttp://localhost:3000.bentomlserveservice:SummarizationSend a request to summarize a short news paragraph:curl-X'POST'\'http://localhost:3000/summarize'\-H'accept: text/plain'\-H'Content-Type: application/json'\-d'{"text": "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson'\''s cat, Whiskers, performed what witnesses are calling a '\''miraculous and gravity-defying leap.'\'' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson'\''s backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as '\''The Leap of the Century."}'DeploymentAfter your Service is ready, you can deploy it toBentoCloudor as a Docker image.First, create abentofile.yamlfile for building a Bento.service:"service:Summarization"labels:owner:bentoml-teamproject:galleryinclude:-"*.py"python:packages:-torch-transformersThen, choose one of the following ways for deployment:BentoCloudMake sure you havelogged in to BentoCloudand then run the following command:bentomldeploy.DockerBuild a Bento to package necessary dependencies and components into a standard distribution format.bentoml buildContainerize the Bento.bentoml containerize summarization:latestRun this image with Docker.dockerrun--rm-p3000:3000summarization:latestFor detailed explanations, readQuickstart.CommunityBentoML supports billions of model runs per day and is used by thousands of
organizations around the globe.Join ourCommunity Slack 💬, where thousands
of AI application developers contribute to the project and help each other.To report a bug or suggest a feature request, useGitHub Issues.ContributingThere are many ways to contribute to the project:Report bugs and "Thumbs up" on issues that are relevant to you.Investigate issues and review other developers' pull requests.Contribute code or documentation to the project by submitting a GitHub pull
request.Check out theContributing GuideandDevelopment Guideto learn moreShare your feedback and discuss roadmap plans in the#bentoml-contributorschannelhere.Thanks to all of our amazing contributors!Usage ReportingBentoML collects usage data that helps our team to improve the product. Only
BentoML's internal API calls are being reported. We strip out as much
potentially sensitive information as possible, and we will never collect user
code, model data, model names, or stack traces. Here's thecodefor usage
tracking. You can opt-out of usage tracking by the--do-not-trackCLI option:bentoml[command]--do-not-trackOr by setting environment variableBENTOML_DO_NOT_TRACK=True:exportBENTOML_DO_NOT_TRACK=TrueLicenseApache License 2.0CitationIf you use BentoML in your research, please cite using the followingcitation:@software{Yang_BentoML_The_framework,author={Yang, Chaoyu and Sheng, Sean and Pham, Aaron and  Zhao, Shenyang and Lee, Sauyon and Jiang, Bo and Dong, Fog and Guan, Xipeng and Ming, Frost},license={Apache-2.0},title={{BentoML: The framework for building reliable, scalable and cost-efficient AI application}},url={https://github.com/bentoml/bentoml}}