Package: trt-engine-explorer
trt-engine-explorerThis repository contains Python code (trexpackage) to explore various aspects of a TensorRT engine plan and its associated inference profiling data.An engine plan file is a serialized TensorRT engine format. It contains information about the final inference graph and can be deserialized for inference runtime execution.  An engine plan is specific to the hardware and software versions of the system used to build the engine.trexis useful for initial model performance debugging, visualization of plan graphs, and for understanding the characteristics of an engine plan.For in-depth performance analysis,Nvidia ® Nsight Systems ™is the recommended performance analysis tool.FeaturesThetrexpackage contains an API and Jupyter notebooks for viewing and inspecting TensorRT engine-plan files and profiling data.An engine plan graph (JSON) is loaded to a Pandas dataframe which allows slicing, querying, filtering, viewing and diagraming.An engine plan graph can be visualized as SVG/PNG files.Layer linters are an API for flagging potential performance hazards (preview feature).Four Jupyter notebooks provide:An introduction totrextutorial.trexAPI examples.Detailed engine plan performance, characteristics and structure analysis.Comparison of two or more engine plans.Becausetrexoperates on JSON input files, it does not require a GPU.CaveatsWhentrtexectimes individual layers, the total engine latency (computed by summing the average latency of each layer) is higher than the latency reported for the entire engine. This is due to per-layer measurement overheads.To measure per-layer execution times, whentrtexecenqueues kernel layers for execution in a stream, it places CUDA event objects between the layers to monitor the start and completion of each layer. These CUDA events add a small overhead which is more noticeable with smaller networks (shallow and narrow networks or networks with small activation data).Supported TensorRT VersionsStarting with TensorRT 8.2, engine-plan graph and profiling data can be exported to JSON files.trexsupports TensortRT 8.2 and 8.4.trexhas only been tested on Ubuntu 18.04, 20.04, 22.04, with Python 3.8.trexdoes not require a GPU, but generating the input JSON file(s) does require a GPU.InstallationThe instructions below detail how to use a Python3 virtualenv for installing and using trex (Python 3.8+ is required).1. Clone the trex code repository from TensorRT OSS repository$ git clone https://github.com/NVIDIA/TensorRT.git2. Create and activate a Python virtual environmentThe commands listed below create and activate a Python virtual environment namedenv_trexwhich is stored in a directory by the same name, and configures the current shell to use it as the default python environment.$ cd TensorRT/tools/experimental/trt-engine-explorer
$ python3 -m virtualenv env_trex
$ source env_trex/bin/activate3. Install trex in development mode and the Jupyter extensions required for the notebooks$ python3 -m pip install -e .
$ jupyter nbextension enable widgetsnbextension --user --py4. Install GraphvizGenerating dot and SVG graphs requires Graphviz, an open source graph visualization software:$ sudo apt-get --yes install graphvizWorkflowThe typicaltrexworkflow is depicted below:Convertan external model to a TensorRTINetworkDefinition.Builda TensorRT engine.Profilethe engine while creating the necessary JSON files.Explorethe engine by loading the JSON files in atrexnotebook.The Python scriptutils/process_engine.pyimplements this workflow for ONNX models:Usetrtexecto import an ONNX model and create an engine.Load the engine and create an engine-graph JSON file.Usetrtexecto profile the engine's inference execution and store the results in an engine profiling JSON file.Create an engine graph diagram in SVG format.For more information seeTensorRT Engine Inspectorand theTutorialnotebook.Jupyter ServerLaunch the Jupyter notebook server as detailed below and open your browser athttp://localhost:8888orhttp://<your-ip-address>:8888$ jupyter-notebook --ip=0.0.0.0 --no-browserLicenseThe TensorRT Engine Explorer license can be found in theLICENSEfile.