Package: truss
TrussThe simplest way to serve AI/ML models in productionWhy Truss?Write once, run anywhere:Package and test model code, weights, and dependencies with a model server that behaves the same in development and production.Fast developer loop:Implement your model with fast feedback from a live reload server, and skip Docker and Kubernetes configuration with a batteries-included model serving environment.Support for all Python frameworks: FromtransformersanddiffusorstoPyTorchandTensorflowtoXGBoostandsklearn, Truss supports models created with any framework, even entirely custom models.See Trusses for popular models including:🦙Llama 2 7B(13B) (70B)🎨Stable Diffusion XL🗣Whisperanddozens more examples.InstallationInstall Truss with:pip install --upgrade trussQuickstartAs a quick example, we'll package atext classification pipelinefrom the open-sourcetransformerspackage.Create a TrussTo get started, create a Truss with the following terminal command:trussinittext-classificationWhen prompted, give your Truss a name likeText classification.Then, navigate to the newly created directory:cdtext-classificationImplement the modelOne of the two essential files in a Truss ismodel/model.py. In this file, you write aModelclass: an interface between the ML model that you're packaging and the model server that you're running it on.There are two member functions that you must implement in theModelclass:load()loads the model onto the model server. It runs exactly once when the model server is spun up or patched.predict()handles model inference. It runs every time the model server is called.Here's the completemodel/model.pyfor the text classification model:fromtransformersimportpipelineclassModel:def__init__(self,**kwargs):self._model=Nonedefload(self):self._model=pipeline("text-classification")defpredict(self,model_input):returnself._model(model_input)Add model dependenciesThe other essential file in a Truss isconfig.yaml, which configures the model serving environment. For a complete list of the config options, seethe config reference.The pipeline model relies onTransformersandPyTorch. These dependencies must be specified in the Truss config.Inconfig.yaml, find the linerequirements. Replace the empty list with:requirements:-torch==2.0.1-transformers==4.30.0No other configuration is needed.DeploymentTruss is maintained byBaseten, which provides infrastructure for running ML models in production. We'll use Baseten as the remote host for your model.Other remotes are coming soon, starting with AWS SageMaker.Get an API keyTo set up the Baseten remote, you'll need aBaseten API key. If you don't have a Baseten account, no worries, justsign up for an accountand you'll be issued plenty of free credits to get you started.Runtruss pushWith your Baseten API key ready to paste when prompted, you can deploy your model:trusspushYou can monitor your model deployment fromyour model dashboard on Baseten.Invoke the modelAfter the model has finished deploying, you can invoke it from the terminal.Invocationtrusspredict-d'"Truss is awesome!"'Response[{"label":"POSITIVE","score":0.999873161315918}]Truss contributorsTruss is backed by Baseten and built in collaboration with ML engineers worldwide. Special thanks toStephan Auerhahn@stability.aiandDaniel Sarfati@Salad Technologiesfor their contributions.We enthusiastically welcome contributions in accordance with ourcontributors' guideandcode of conduct.