Package: pyotritonclient
Triton HTTP Client for PyodideA Pyodide python http client library and utilities for communicating with Triton Inference Server (based on tritonclient from NVIDIA).This is a simplified implemetation of the triton client from NVIDIA, it works both in the browser with Pyodide Python or the native Python.
It only implement the http client, and most of the API remains the similar but changed into async and with additional utility functions.InstallationTo use it in native CPython, you can install the package by running:pip install pyotritonclientFor Pyodide-based Python environment, for example:JupyterLiteorPyodide console, you can install the client by running the following python code:importmicropipmicropip.install("pyotritonclient")UsageBasic exampleTo execute the model, we provide utility functions to make it much easier:importnumpyasnpfrompyotritonclientimportexecute# create fake input tensorsinput0=np.zeros([2,349,467],dtype='float32')# run inferenceresults=awaitexecute(inputs=[input0,{"diameter":30}],server_url='https://ai.imjoy.io/triton',model_name='cellpose-python')The above example assumes you are running the code in a jupyter notebook or an environment supports top-level await, if you are trying the example code in a normal python script, please wrap the code into an async function and run with asyncio as follows:importasyncioimportnumpyasnpfrompyotritonclientimportexecuteasyncdefrun():results=awaitexecute(inputs=[np.zeros([2,349,467],dtype='float32'),{"diameter":30}],server_url='https://ai.imjoy.io/triton',model_name='cellpose-python')print(results)loop=asyncio.get_event_loop()loop.run_until_complete(run())You can access the lower level api, see thetest example.You can also find the officialclient examplesdemonstrate how to use the
package to issue request totriton inference server. However, please notice that you will need to
change the http client code into async style. For example, instead of doingclient.infer(...), you need to doawait client.infer(...).The http client code is forked fromtriton client git reposince commitb3005f9db154247a4c792633e54f25f35ccadff0.Using the sequence executor with stateful modelsTo simplify the manipulation on stateful models with sequence, we also provide theSequenceExecutorto make it easier to run models in a sequence.frompyotritonclientimportSequenceExcutorseq=SequenceExcutor(server_url='https://ai.imjoy.io/triton',model_name='cellpose-train',sequence_id=100)inputs=[image.astype('float32'),labels.astype('float32'),{"steps":1,"resume":True}]for(image,labels,info)intrain_samples:result=awaitseq.step(inputs)result=awaitseq.end(inputs)Note that above example calledseq.end()by sending the last inputs again to end the sequence. If you want to specify the inputs for the execution, you can runresult = await se.end(inputs).For a small batch of data, you can also run it like this:frompyotritonclientimportSequenceExcutorseq=SequenceExcutor(server_url='https://ai.imjoy.io/triton',model_name='cellpose-train',sequence_id=100)# a list of inputsinputs_batch=[[image.astype('float32'),labels.astype('float32'),{"steps":1,"resume":True}]for(image,labels,_)intrain_samples]defon_step(i,result):"""Function called on every step"""print(i)results=awaitseq(inputs_batch,on_step=on_step)Server setupSince we access the server from the browser environment which typically has more security restrictions, it is important that the server is configured to enable browser access.Please make sure you configured following aspects:The server must provide HTTPS endpoints instead of HTTPThe server should send the following headers:Access-Control-Allow-Headers: Inference-Header-Content-Length,Accept-Encoding,Content-Encoding,Access-Control-Allow-HeadersAccess-Control-Expose-Headers: Inference-Header-Content-Length,Range,Origin,Content-TypeAccess-Control-Allow-Methods: GET,HEAD,OPTIONS,PUT,POSTAccess-Control-Allow-Origin: *(This is optional depending on whether you want to support CORS)