Package: pythiags
pythiaPythonic Deepstream.NVidia Deepstreamis an excellent gstreamer
framework which allows to build ai-powered, performant applications running on nvidia
hardware. Its python API and bindings, however, have a bunch of painpoints which we've
here collected and addressed with pythia:Metadata Extraction: Deepstream metadata extraction requires using buffer probes:
pythia provides an easy to use interface which splits metadata extraction and
processing.Metadata Iteration: pyds api iterators are not pythonic: pythia provides intuitive
deepstream metadata iterators to use, likefor frame in frames_per_batch(buffer)wrapping pybind c++ casting and iteration.Python boilerplate: Python gstreamer apps get very large very fast. Pythia abstracts
away common stuff and lets you focus on your application.Quick prototyping: Sometimes you just want to check the performance of a new model
(eg after exporting fromNvidia TAO), or verify
the environment. Pythia comes with ready-to-run demo applications, and agst-launch-like cli.pythia offers:Common metadata extraction and parsing utilities.Workers and queues management in the background, to offload processing outside of the
buffer probe.Ready to use Docker images for both aarch64 (jetson) and x86_64 (nvidia gpu).ExamplesRunning pythia from the cligst-pylaunchYou can run familiar pipelines and attach buffer probes from simple python modules.Create FilesCreate a fileprobe.pywith:frompythiaimportobjects_per_batchdefgen_detections(batch_meta):forframe,detectioninobjects_per_batch(batch_meta):box=detection.rect_paramsyield{"frame_num":frame.frame_num,"label":detection.obj_label,"left":box.left,"top":box.top,"width":box.width,"height":box.height,"confidence":detection.confidence,}Create a filepipeline.txtwith:uridecodebin
  uri=file://{input}
! identity
  eos-after=30
! nvvideoconvert
! muxer.sink_0
nvstreammux
  name=muxer width=1280 height=720 batch-size=1
! nvinfer
  name=pgie
  config-file-path={pgie-conf}
! nvvideoconvert
! nvdsosd
! nvvideoconvert
! queue
! x264enc
! mp4mux
! filesink location={output}Running pythiarun the application with:$gst-pylaunch\-p./pipeline.txt\--pgie-conf=/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt\--input=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4\--output=/tmp/overlayed.mp4\--probe=probe.py:gen_detections@pgie.srcNote the--pgie-conf,--input, and--outputcli args were dynamically parsed and
added from the pipeline file.This command instructed pythia to do the following:Load a pipeline from a file located at./pipeline.txt, which containsgst-launch-like syntax with some parameters to be inserted (input,pgie-conf,output).Format the pipelie withinput,pgie-confandoutputfrom received parameters.
(For a more complex syntax, you can installpythia[jinja]to use jinja as a
template backend. See the documentation for more details.)Setup a buffer probe which internally calls thegen_detectionsmethod defined in theprobe.pyfile.Attach said buffer probe in thesource padof thepgie-named element of the
pipeline.Send incoming metadata to a logger which prints jsonified metadata to console.Check your outputCheck your console to see the incoming detections.Want to do something else with the detections? You can choose between several
backends: logging (stdout , stderr, file available), in-memory (deque),
kafka, redis, or implement your own streaming connector with thePYTHIA_STREAM_URIenv var. Check the documentation for more details.Develop applications based on pythiapython APIIf you want more granular control over the behavior of the application, its signals,
events, and messages, you can instead program an aplication using pythia's API.Create FilesContinuing with the same pipeline as in the previous example,Create a filemyscript.pywith:importjsonfromkafkaimportKafkaProducerfromkafka.adminimportKafkaAdminClient,NewTopicfrompythiaimportApplication,Gst,objects_per_batchclassApp(Application):def__init__(self,*a,**kw):super().__init__(*a,**kw)self.manual_kafka=KafkaProducer(bootstrap_servers="kafka:9092")defon_message_error(self,*a,**kw):err,debug=super().on_message_error(*a,**kw)self.manual_kafka.send("app_events",json.dumps({"CONDITION":"ERROR","ERR":err,"DEBUG":debug}).encode())raiseRuntimeError("Unhandled pipeline error")defon_message_eos(self,bus,message):self.manual_kafka.send("app_events",json.dumps({"CONDITION":"EOS","SENT_BY":str(message.src)}).encode())super().on_message_eos(bus,message)app=App.from_pipeline_file("pipeline.txt",params={"pgie-conf":"/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt","input":"/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4","output":"/tmp/overlayed.mp4",})@app.probe("pgie",pad_direction="src",backend_uri="kafka://kafka:9092?stream=raw_detections")defpgie_srcprobe(batch_meta):forframe,detectioninobjects_per_batch(batch_meta):frame_num=frame.frame_numbox=detection.rect_paramsyield{"frame_num":frame_num,"label":detection.obj_label,"left":box.left,"top":box.top,"width":box.width,"height":box.height,"confidence":detection.confidence,}@app.probe("muxer",pad_direction="src",)defsource_probe(pad,info):app.manual_kafka.send("app_events",json.dumps({"CONDITION":"STARTED","PAD_CAPS":pad.props.caps.to_string(),"PAD_DIRECTION":pad.props.direction,"PAD_OFFSET":pad.props.offset,}).encode())returnGst.PadProbeReturn.REMOVEif__name__=="__main__":admin=KafkaAdminClient(bootstrap_servers="kafka:9092")if"app_events"notinadmin.list_topics():admin.create_topics(new_topics=[NewTopic(name="app_events",num_partitions=1,replication_factor=1)],validate_only=False,)app()Running pythiarun the application with:$pythonmyscript.pyIn this mode, you have more control over the application behavior:Subclass applicationinstantiate a custom message handler (a kafka producer in this example)forward error and EOS messages to a custom kafka topicinterpolate the pipeline template file with python variables to construct the appuse the@app.probedecorator as a generator, letting pythia handle the messages
internallyuse the@app.probedecorator as a probe, handling manually the buffer flow and
messaging.Want to do something else while the application is running? you can run the application
withapp(background=True)instead. See the documentation for details and more
examples.Check your outputCheck the kafka topics to see the incoming detections.SetupPrerequisitesnvidia hardware (either jetson or gpu)One ofrecent docker (with support--gpus=all)nvidia-dockerinstalled,environment with deepstream 6.1.1 andthese bindingsInstallnon-dockerpip install pythiagsdockerdocker login ghcr.io(See instructionshere)docker pull ghcr.io/rmclabs-io/pythiaorghcr.io/rmclabs-io/pythia-l4tBuild your image usingFROM ghcr.io/rmclabs-io/pythiaorFROM ghcr.io/rmclabs-io/pythia-l4tNOTE:latesttag is deliberately not published.run docker with the following flags at least:$dockerrun\--gpus=all\-v/tmp/.X11-unix:/tmp/.X11-unix\-eDISPLAY\ghcr.io/rmclabs-io/pythia-dev:1.2.1Alternatively, you could useghcr.io/rmclabs-io/pythia-devorFROM ghcr.io/rmclabs-io/pythia-l4t-dev.UsageNote: If running from docker, make sure you've properly configured the container and its
environment, see thereference upstreamFor more examples and tutorials, visit theexamples sectionof the documentation.FAQCheck out ongoing and future developmenthereFAQ / Common IssuesQ: Package installation fails:A1: upgrade your pip:pip install --upgrade pip(Required "pip>=10").A2: Make sure you've installed the build prerequisites, as listed inreqs/apt.build.list.Q: My application is running slow on JetsonA: Ensure to enable jetson-clocks and maxn (Seereference)
:sudo nvpmodel -m 0sudo jetson_clocksQ: Program exits with errorUnable to get a Window, abort.A: Make sure x11 is properly configured. This is common when running through ssh
sessions. In most of the cases, this just means you need to have theDISPLAYenvironment variable correctly set. To list available displays, run thewcommand:$w09:53:38up2days,17:26,1user,loadaverage:0,36,0,33,0,23
USERTTYFROMLOGIN@IDLEJCPUPCPUWHAT
rmclabs:0:0lun16?xdm?3:06m0.02s/usr/lib/gdm3/gdm-x-session--run-script/usr/lib/gnome-session/run-systemd-sessionunity-session.target
rmclabspts/1110.100.10.7909:571.00s0.10s0.00swFrom here, choose a display corresponding to a local connection (:0in this case,
including the colon). Then, export the environment variable and run again your program:exportDISPLAY=:0# run your program hereQ: Program exits with error (from docker):X Error of failed request:  BadShmSeg (invalidshared segment parameter)Major opcode of failed request:  150 (XVideo)Minor opcode of failed request:  19 ()Segment id in failed request:  0x121Serial number of failed request:  57Current serial number in output stream:  58python: ../../src/hb-object-private.hh:154: Type* hb_object_reference(Type*) [with Type = hb_unicode_funcs_t]: Assertion `hb_object_is_valid (obj)' failed.A: Add--ipc=hostflag to docker run.Q: Python segfaults when several applications are run subsequently:A: It seems to be a race condition produced by theuridecodebinelement usingnvjpegenc(maybe others?). You can either add a timeout between runs (1 sec seems
to do it), or changenvjpegenc- seepythia.utils.gst:demote_plugin.Q: Python segfaults when several applications are run subsequently:A: It seems to be a race condition produced by theuridecodebinelement usingnvjpegenc(maybe others?). You can either add a timeout between runs (1 sec seems
to do it), or changenvjpegenc- seepythia.utils.gst:demote_plugin.Q: I am unable to build the devcontainer:A: Make sure to update thedevcontainer.jsonwith a properBASE_IMAGEandDOCKER_GROUP_ID.ContributeforkclonePull Request from new branch[Optional, recommended]: use provided devcontainer.[Optional, recommended]: runpre-commit installto validate commits.add tests, docs, code, scripts, etc[Optional] check code manually, with./scripts/format,./scripts/lint,./scripts/docs,./scripts/test, etc.Commit usingConventional commits.push, wait for ci and/or maintainers feedbackrepeat 6-8 until success!For more instructions, visit theDevelopers sectionof the documentation.