Package: sit4onnx
sit4onnxTools for simple inference testing using TensorRT, CUDA and OpenVINO CPU/GPU and CPU providers.SimpleInferenceTest forONNX.https://github.com/PINTO0309/simple-onnx-processing-toolsToDoAdd an interface to allow arbitrary test data to be specified as input parameters.numpy.ndarraynumpy fileAllow static fixed shapes to be specified when dimensions other than batch size are undefined.Returns numpy.ndarray of the last inference result as a return value when called from a Python script.Add--output_numpy_fileoption. Output the final inference results to a numpy file.Add--non_verboseoption.1. Setup1-1. HostPC### option$echoexportPATH="~/.local/bin:$PATH">>~/.bashrc\&&source~/.bashrc### run$pipinstall-Uonnx\&&pipinstall-Usit4onnx1-2. Dockerhttps://github.com/PINTO0309/simple-onnx-processing-tools#docker2. CLI Usage$ sit4onnx -h

usage:
  sit4onnx [-h]
  -if INPUT_ONNX_FILE_PATH
  [-b BATCH_SIZE]
  [-fs DIM0 [DIM1 DIM2 ...]]
  [-tlc TEST_LOOP_COUNT]
  [-oep {tensorrt,cuda,openvino_cpu,openvino_gpu,cpu}]
  [-iont INTRA_OP_NUM_THREADS]
  [-ifp INPUT_NUMPY_FILE_PATHS_FOR_TESTING]
  [-ofp]
  [-n]

optional arguments:
  -h, --help
      show this help message and exit.

  -if, --input_onnx_file_path INPUT_ONNX_FILE_PATH
      Input onnx file path.

  -b, --batch_size BATCH_SIZE
      Value to be substituted if input batch size is undefined.
      This is ignored if the input dimensions are all of static size.
      Also ignored if input_numpy_file_paths_for_testing
      or numpy_ndarrays_for_testing or fixed_shapes is specified.

  -fs, --fixed_shapes DIM0 [DIM1 DIM2 ...]
      Input OPs with undefined shapes are changed to the specified shape.
      This parameter can be specified multiple times depending on
      the number of input OPs in the model.
      Also ignored if input_numpy_file_paths_for_testing is specified.
      e.g.
      --fixed_shapes 1 3 224 224
      --fixed_shapes 1 5
      --fixed_shapes 1 1 224 224

  -tlc, --test_loop_count TEST_LOOP_COUNT
      Number of times to run the test.
      The total execution time is divided by the number of times the test is executed,
      and the average inference time per inference is displayed.

  -oep, --onnx_execution_provider {tensorrt,cuda,openvino_cpu,openvino_gpu,cpu}
      ONNX Execution Provider.

  -iont, --intra_op_num_threads INTRA_OP_NUM_THREADS
      Sets the number of threads used to parallelize the execution within nodes.
      Default is 0 to let onnxruntime choose.

  -ifp, --input_numpy_file_paths_for_testing INPUT_NUMPY_FILE_PATHS_FOR_TESTING
      Use an external file of numpy.ndarray saved using np.save as input data for testing.
      This parameter can be specified multiple times depending on the number of input OPs
      in the model.
      If this parameter is specified, the value specified for batch_size and fixed_shapes
      are ignored.
      e.g.
      --input_numpy_file_paths_for_testing aaa.npy
      --input_numpy_file_paths_for_testing bbb.npy
      --input_numpy_file_paths_for_testing ccc.npy

  -ofp, --output_numpy_file
      Outputs the last inference result to an .npy file.

  -n, --non_verbose
      Do not show all information logs. Only error logs are displayed.3. In-script Usage>>>fromsit4onnximportinference>>>help(inference)Helponfunctioninferenceinmodulesit4onnx.onnx_inference_test:inference(input_onnx_file_path:str,batch_size:Union[int,NoneType]=1,fixed_shapes:Union[List[int],NoneType]=None,test_loop_count:Union[int,NoneType]=10,onnx_execution_provider:Union[str,NoneType]='tensorrt',intra_op_num_threads:Optional[int]=0,input_numpy_file_paths_for_testing:Union[List[str],NoneType]=None,numpy_ndarrays_for_testing:Union[List[numpy.ndarray],NoneType]=None,output_numpy_file:Union[bool,NoneType]=False,non_verbose:Union[bool,NoneType]=False)->List[numpy.ndarray]Parameters----------input_onnx_file_path:strInputonnxfilepath.batch_size:Optional[int]Valuetobesubstitutedifinputbatchsizeisundefined.Thisisignorediftheinputdimensionsareallofstaticsize.Alsoignoredifinput_numpy_file_paths_for_testingornumpy_ndarrays_for_testingisspecified.Default:1fixed_shapes:Optional[List[int]]InputOPswithundefinedshapesarechangedtothespecifiedshape.ThisparametercanbespecifiedmultipletimesdependingonthenumberofinputOPsinthemodel.Alsoignoredifinput_numpy_file_paths_for_testingornumpy_ndarrays_for_testingisspecified.e.g.[[1,3,224,224],[1,5],[1,1,224,224],]Default:Nonetest_loop_count:Optional[int]Numberoftimestorunthetest.Thetotalexecutiontimeisdividedbythenumberoftimesthetestisexecuted,andtheaverageinferencetimeperinferenceisdisplayed.Default:10onnx_execution_provider:Optional[str]ONNXExecutionProvider."tensorrt"or"cuda"or"openvino_cpu"or"openvino_gpu"or"cpu"Default:"tensorrt"intra_op_num_threads:Optional[int]Setsthenumberofthreadsusedtoparallelizetheexecutionwithinnodes.Defaultis0toletonnxruntimechoose.input_numpy_file_paths_for_testing:Optional[List[str]]Useanexternalfileofnumpy.ndarraysavedusingnp.saveasinputdatafortesting.Ifthisparameterisspecified,thevaluespecifiedforbatch_sizeandfixed_shapesareignored.numpy_ndarray_for_testingCannotbespecifiedatthesametime.FormodelswithmultipleinputOPs,specifymultiplenumpyfilepathsinlistformat.e.g.['aaa.npy','bbb.npy','ccc.npy']Default:Nonenumpy_ndarrays_for_testing:Optional[List[np.ndarray]]Specifythenumpy.ndarraytobeusedforinferencetesting.Ifthisparameterisspecified,thevaluespecifiedforbatch_sizeandfixed_shapesareignored.input_numpy_file_paths_for_testingCannotbespecifiedatthesametime.FormodelswithmultipleinputOPs,specifymultiplenumpy.ndarraysinlistformat.e.g.[np.asarray([[[1.0],[2.0],[3.0]]],dtype=np.float32),np.asarray([1],dtype=np.int64),]Default:Noneoutput_numpy_file:Optional[bool]Outputsthelastinferenceresulttoan.npyfile.Default:Falsenon_verbose:Optional[bool]Donotshowallinformationlogs.Onlyerrorlogsaredisplayed.Default:FalseReturns-------final_results:List[np.ndarray]LastReasoningResults.4. CLI Execution$sit4onnx\--input_onnx_file_pathosnet_x0_25_msmt17_Nx3x256x128.onnx\--batch_size10\--test_loop_count10\--onnx_execution_providertensorrt5. In-script Executionfromsit4onnximportinferenceinference(input_onnx_file_path="osnet_x0_25_msmt17_Nx3x256x128.onnx",batch_size=10,test_loop_count=10,onnx_execution_provider="tensorrt",)6. Sample$dockerrun--gpusall-it--rm\-v`pwd`:/home/user/workdir\ghcr.io/pinto0309/openvino2tensorflow:latest

$sudopipinstall-Usimple-onnx-processing-tools
$sit4onnx\--input_onnx_file_pathosnet_x0_25_msmt17_Nx3x256x128.onnx\--batch_size10\--test_loop_count10\--onnx_execution_providertensorrt$dockerrun--gpusall-it--rm\-v`pwd`:/home/user/workdir\ghcr.io/pinto0309/openvino2tensorflow:latest

$sudopipinstall-Usimple-onnx-processing-tools
$sit4onnx\--input_onnx_file_pathsci_NxHxW.onnx\--fixed_shapes1003224224\--onnx_execution_providertensorrthttps://github.com/daquexian/onnx-simplifier/issues/178

$dockerrun--gpusall-it--rm\-v`pwd`:/home/user/workdir\ghcr.io/pinto0309/openvino2tensorflow:latest

$sudopipinstall-Usimple-onnx-processing-tools
$sit4onnx\--input_onnx_file_pathhitnet_xl_sf_finalpass_from_tf_720x1280_cast.onnx\--onnx_execution_providertensorrt7. Referencehttps://github.com/onnx/onnx/blob/main/docs/Operators.mdhttps://docs.nvidia.com/deeplearning/tensorrt/onnx-graphsurgeon/docs/index.htmlhttps://github.com/NVIDIA/TensorRT/tree/main/tools/onnx-graphsurgeonhttps://github.com/PINTO0309/simple-onnx-processing-toolshttps://github.com/PINTO0309/PINTO_model_zoo8. Issueshttps://github.com/PINTO0309/simple-onnx-processing-tools/issues