Package: nvidia-pytriton
PyTriton - a Flask/FastAPI-like framework designed to streamline
the use of NVIDIA’sTriton Inference Server.For comprehensive guidance on how to deploy your models, optimize performance,
and explore the API, delve into the extensive resources found in ourdocumentation.Features at a GlanceThe distinct capabilities of PyTriton are summarized in the feature matrix:FeatureDescriptionNative Python supportYou can create anyPython functionand expose it as an HTTP/gRPC API.Framework-agnosticYou can run any Python code with any framework of your choice, such as: PyTorch,
TensorFlow, or JAX.Performance
optimizationYou can benefit fromdynamic batching, response cache, model
pipelining,clusters, and GPU/CPU inference.DecoratorsYou can use batchingdecoratorsto handle batching  and other
pre-processing tasks for your inference function.Easyinstallationand setupYou can use a simple and familiar interface based on Flask/FastAPI for easy
installation andsetup.Model clientsYou can access high-level model clients for HTTP/gRPC requests with configurable
options and both synchronous andasynchronousAPI.Streaming (alpha)You can stream partial responses from a model by serving it in adecoupled mode.Learn more about PyTriton’sarchitecture.PrerequisitesBefore proceeding with the installation of PyTriton, ensure your system meets the following criteria:Operating System: Compatible with glibc version2.35or higher.
- Primarily tested on Ubuntu 22.04.
- Other supported OS include Debian 11+, Rocky Linux 9+, and Red Hat UBI 9+.
- Useldd--versionto verify your glibc version.Python: Version3.8or newer.pip: Version20.3or newer.libpython: Ensurelibpython3.*.sois installed, corresponding to your Python version.InstallThe PyTriton can be installed from pypi.org by running the following command:pip install nvidia-pytritonImportant: The Triton Inference Server binary is installed as part of the PyTriton package.Discover more about PyTriton’sinstallation procedures, including Docker usage, prerequisites, and insights intobuilding binaries from sourceto match your specific Triton server versions.Quick StartThe quick start presents how to run Python model in Triton Inference Server without need to change the current working
environment. In the example we are using a simpleLinearmodel.Theinfer_fnis a function that takes andatatensor and returns a list with single output tensor. The@batchfrombatching decoratorsis used to handle batching for the model.importnumpyasnpfrompytriton.decoratorsimportbatch@batchdefinfer_fn(data):result=data*np.array([[-1]],dtype=np.float32)# Process inputs and produce resultreturn[result]In the next step, you can create the binding between the inference callable and Triton Inference Server using thebindmethod from pyTriton. This method takes the model name, the inference callable, the inputs and outputs tensors, and an optional model configuration object.frompytriton.model_configimportTensorfrompytriton.tritonimportTritontriton=Triton()triton.bind(model_name="Linear",infer_func=infer_fn,inputs=[Tensor(name="data",dtype=np.float32,shape=(-1,)),],outputs=[Tensor(name="result",dtype=np.float32,shape=(-1,)),],)triton.run()Finally, you can send an inference query to the model using theModelClientclass. Theinfer_samplemethod takes the input data as a numpy array and returns the output data as a numpy array. You can learn more about theModelClientclass in theclientssection.frompytriton.clientimportModelClientclient=ModelClient("localhost","Linear")data=np.array([1,2,],dtype=np.float32)print(client.infer_sample(data=data))After the inference is done, you can stop the Triton Inference Server and close the client:client.close()triton.stop()The output of the inference should be:{'result':array([-1.,-2.],dtype=float32)}For the full example, including defining the model and binding it to the Triton server, check out our detailedQuick Startinstructions. Get your model up and running, explore how to serve it, and learn how toinvoke it from client applications.The full example code can be found inexamples/linear_random_pytorch.ExamplesTheexamplespage showcases various use cases of serving models using PyTriton. This includes simple examples of running models in PyTorch, TensorFlow2, JAX, and plain Python. In addition, more advanced scenarios are covered, such as online learning, multi-node models, and deployment on Kubernetes using PyTriton. Each example is accompanied by instructions on how to build and run it. Discover more about utilizing PyTriton by exploring our examples.LinksSourceIssuesChangelogKnown IssuesContributing