Package: onnx2tf
onnx2tfSelf-Created Tools to convert ONNX files (NCHW) to TensorFlow/TFLite/Keras format (NHWC). The purpose of this tool is to solve the massive Transpose extrapolation problem inonnx-tensorflow(onnx-tf). I don't need a Star, but give me a pull request. Since I am adding challenging model optimizations and fixing bugs almost daily, I frequently embed potential bugs that would otherwise break through CI's regression testing. Therefore, if you encounter new problems, I recommend that you try a package that is a few versions older, or try the latest package that will be released in a few days.Model Conversion Statushttps://github.com/PINTO0309/onnx2tf/wiki/model_statusSupported layershttps://github.com/onnx/onnx/blob/main/docs/Operators.md:heavy_check_mark:: Supported　:white_check_mark:: Partial supportHelp wanted: Pull Request are welcomeSee the list of supported layersOPStatusAbs:heavy_check_mark:Acosh:heavy_check_mark:Acos:heavy_check_mark:Add:heavy_check_mark:And:heavy_check_mark:ArgMax:heavy_check_mark:ArgMin:heavy_check_mark:Asinh:heavy_check_mark:Asin:heavy_check_mark:Atanh:heavy_check_mark:Atan:heavy_check_mark:AveragePool:heavy_check_mark:BatchNormalization:heavy_check_mark:Bernoulli:heavy_check_mark:BitShift:heavy_check_mark:BitwiseAndHelp wantedBitwiseNotHelp wantedBitwiseOrHelp wantedBitwiseXorHelp wantedCast:heavy_check_mark:Ceil:heavy_check_mark:Celu:heavy_check_mark:CenterCropPadHelp wantedClip:heavy_check_mark:Col2Im:white_check_mark:Compress:heavy_check_mark:ConcatFromSequence:heavy_check_mark:Concat:heavy_check_mark:ConstantOfShape:heavy_check_mark:Constant:heavy_check_mark:Conv:heavy_check_mark:ConvInteger:white_check_mark:ConvTranspose:heavy_check_mark:Cosh:heavy_check_mark:Cos:heavy_check_mark:CumSum:heavy_check_mark:DeformConvHelp wantedDepthToSpace:heavy_check_mark:Det:heavy_check_mark:DequantizeLinear:heavy_check_mark:DFTHelp wantedDiv:heavy_check_mark:Dropout:heavy_check_mark:DynamicQuantizeLinear:heavy_check_mark:Einsum:heavy_check_mark:Elu:heavy_check_mark:Equal:heavy_check_mark:Erf:heavy_check_mark:Expand:heavy_check_mark:Exp:heavy_check_mark:EyeLike:heavy_check_mark:Flatten:heavy_check_mark:Floor:heavy_check_mark:FusedConv:heavy_check_mark:GatherElements:heavy_check_mark:GatherND:heavy_check_mark:Gather:heavy_check_mark:Gelu:heavy_check_mark:Gemm:heavy_check_mark:GlobalAveragePool:heavy_check_mark:GlobalLpPool:heavy_check_mark:GlobalMaxPool:heavy_check_mark:GreaterOrEqual:heavy_check_mark:Greater:heavy_check_mark:GridSample:white_check_mark:GroupNormalizationHelp wantedGRU:heavy_check_mark:HammingWindow:white_check_mark:HannWindow:white_check_mark:Hardmax:heavy_check_mark:HardSigmoid:heavy_check_mark:HardSwish:heavy_check_mark:Identity:heavy_check_mark:If:heavy_check_mark:Input:heavy_check_mark:InstanceNormalization:heavy_check_mark:Inverse:heavy_check_mark:IsInf:heavy_check_mark:IsNaN:heavy_check_mark:LayerNormalization:heavy_check_mark:LeakyRelu:heavy_check_mark:LessOrEqual:heavy_check_mark:Less:heavy_check_mark:Log:heavy_check_mark:LogSoftmax:heavy_check_mark:LoopHelp wantedLpNormalization:heavy_check_mark:LRN:heavy_check_mark:LSTM:heavy_check_mark:MatMul:heavy_check_mark:MatMulInteger:heavy_check_mark:MaxPool:heavy_check_mark:Max:heavy_check_mark:MaxRoiPoolHelp wantedMaxUnpool:heavy_check_mark:Mean:heavy_check_mark:MeanVarianceNormalization:heavy_check_mark:MelWeightMatrix:heavy_check_mark:Min:heavy_check_mark:Mish:heavy_check_mark:Mod:heavy_check_mark:Mul:heavy_check_mark:Multinomial:heavy_check_mark:Neg:heavy_check_mark:NonMaxSuppression:heavy_check_mark:NonZero:heavy_check_mark:OptionalHelp wantedOptionalGetElement:heavy_check_mark:OptionalHasElement:heavy_check_mark:Not:heavy_check_mark:OneHot:heavy_check_mark:Or:heavy_check_mark:Pad:heavy_check_mark:Pow:heavy_check_mark:PRelu:heavy_check_mark:QLinearAdd:heavy_check_mark:QLinearConcat:heavy_check_mark:QLinearConv:heavy_check_mark:QLinearLeakyRelu:heavy_check_mark:QLinearMatMul:heavy_check_mark:QLinearMul:heavy_check_mark:QLinearSigmoid:heavy_check_mark:QLinearSoftmax:heavy_check_mark:QuantizeLinear:heavy_check_mark:RandomNormalLike:heavy_check_mark:RandomNormal:heavy_check_mark:RandomUniformLike:heavy_check_mark:RandomUniform:heavy_check_mark:Range:heavy_check_mark:Reciprocal:heavy_check_mark:ReduceL1:heavy_check_mark:ReduceL2:heavy_check_mark:ReduceLogSum:heavy_check_mark:ReduceLogSumExp:heavy_check_mark:ReduceMax:heavy_check_mark:ReduceMean:heavy_check_mark:ReduceMin:heavy_check_mark:ReduceProd:heavy_check_mark:ReduceSum:heavy_check_mark:ReduceSumSquare:heavy_check_mark:Relu:heavy_check_mark:Reshape:heavy_check_mark:Resize:heavy_check_mark:ReverseSequence:heavy_check_mark:RNN:heavy_check_mark:RoiAlign:heavy_check_mark:Round:heavy_check_mark:ScaleAndTranslate:heavy_check_mark:Scatter:heavy_check_mark:ScatterElements:heavy_check_mark:ScatterND:heavy_check_mark:ScanHelp wantedSelu:heavy_check_mark:SequenceAt:heavy_check_mark:SequenceConstruct:heavy_check_mark:SequenceEmpty:heavy_check_mark:SequenceErase:heavy_check_mark:SequenceInsert:heavy_check_mark:SequenceLength:heavy_check_mark:Shape:heavy_check_mark:Shrink:heavy_check_mark:Sigmoid:heavy_check_mark:Sign:heavy_check_mark:Sinh:heavy_check_mark:Sin:heavy_check_mark:Size:heavy_check_mark:Slice:heavy_check_mark:Softmax:heavy_check_mark:Softplus:heavy_check_mark:Softsign:heavy_check_mark:SpaceToDepth:heavy_check_mark:Split:heavy_check_mark:SplitToSequence:heavy_check_mark:Sqrt:heavy_check_mark:Squeeze:heavy_check_mark:STFT:white_check_mark:StringNormalizer:white_check_mark:Sub:heavy_check_mark:Sum:heavy_check_mark:Tanh:heavy_check_mark:Tan:heavy_check_mark:TfIdfVectorizerHelp wantedThresholdedRelu:heavy_check_mark:Tile:heavy_check_mark:TopK:heavy_check_mark:Transpose:heavy_check_mark:Trilu:heavy_check_mark:Unique:heavy_check_mark:Unsqueeze:heavy_check_mark:Upsample:heavy_check_mark:Where:heavy_check_mark:Xor:heavy_check_mark:DemoVideo speed is adjusted approximately 50 times slower than actual speed.EnvironmentLinux / Windowsonnx==1.15.0onnxruntime==1.16.3onnx-simplifier==0.4.33 or 0.4.30(onnx.onnx_cpp2py_export.shape_inference.InferenceError: [ShapeInferenceError] (op_type:Slice, node name: /xxxx/Slice): [ShapeInferenceError] Inferred shape and existing shape differ in rank: (x) vs (y))onnx_graphsurgeonsimple_onnx_processing_toolstensorflow==2.15.0, Note:#515, Special bugs:#436psutil==5.9.5ml_dtypes==0.2.0flatbuffers-compiler (Optional, Only when using the-coionoption. Executable file namedflatc.)# Custom flatc v23.5.26 binary for Ubuntu 20.04+# https://github.com/PINTO0309/onnx2tf/issues/196wgethttps://github.com/PINTO0309/onnx2tf/releases/download/1.16.31/flatc.tar.gz\&&tar-zxvfflatc.tar.gz\&&sudochmod+xflatc\&&sudomvflatc/usr/bin/Sample Usage1. InstallNote: If you are using TensorFlow v2.13.0 or earlier, use a version older than onnx2tf v1.17.5. onnx2tf v1.17.6 or later will not work properly due to changes in TensorFlow's API.See:#515HostPCWhen using GHCR, seeAuthenticating to the Container registryhttps://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry# PAT authentication is required to pull from GHCR.dockerloginghcr.io

Username(xxxx):{Enter}Password:{PersonalAccessToken}LoginSucceeded

dockerrun--rm-it\-v`pwd`:/workdir\-w/workdir\ghcr.io/pinto0309/onnx2tf:1.19.11

or# Authentication is not required for pulls from Docker Hub.dockerrun--rm-it\-v`pwd`:/workdir\-w/workdir\docker.io/pinto0309/onnx2tf:1.19.11

or

pipinstall-Uonnx==1.15.0\&&pipinstall-Unvidia-pyindex\&&pipinstall-Uonnx-graphsurgeon\&&pipinstall-Uonnxruntime==1.16.3\&&pipinstall-Uonnxsim==0.4.33\&&pipinstall-Usimple_onnx_processing_tools\&&pipinstall-Uonnx2tf\&&pipinstall-Uh5py==3.7.0\&&pipinstall-Upsutil==5.9.5\&&pipinstall-Uml_dtypes==0.2.0

or

pipinstall-e.orGoogle Colaboratory Python3.10!sudo apt-get -y update
!sudo apt-get -y install python3-pip
!sudo apt-get -y install python-is-python3
!wget https://github.com/PINTO0309/onnx2tf/releases/download/1.16.31/flatc.tar.gz \
  && tar -zxvf flatc.tar.gz \
  && sudo chmod +x flatc \
  && sudo mv flatc /usr/bin/
!pip install -U pip \
  && pip install tensorflow==2.15.0 \
  && pip install -U onnx==1.15.0 \
  && python -m pip install onnx_graphsurgeon \
        --index-url https://pypi.ngc.nvidia.com \
  && pip install -U onnxruntime==1.16.3 \
  && pip install -U onnxsim==0.4.33 \
  && pip install -U simple_onnx_processing_tools \
  && pip install -U onnx2tf \
  && pip install -U protobuf==3.20.3 \
  && pip install -U h5py==3.7.0 \
  && pip install -U psutil==5.9.5 \
  && pip install -U ml_dtypes==0.2.02. Run testOnly patterns that are considered to be used particularly frequently are described. In addition, there are several other options, such as disabling Flex OP and additional options to improve inference performance. See:CLI Parameter# Float32, Float16# This is the fastest way to generate tflite,# but the accompanying saved_model will not have a signature.# "ValueError: Only support at least one signature key."# If you are having trouble with this error, please use the `-osd` option.wgethttps://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf-iresnet18-v1-7.onnx# saved_model with signaturedefs added.# Output in the form of saved_model that can be used for serving# or conversion to other frameworks. e.g. TensorFlow.js, CoreML, etc# https://github.com/PINTO0309/onnx2tf#15-conversion-to-tensorflowjs# https://github.com/PINTO0309/onnx2tf#16-conversion-to-coremlwgethttps://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf-iresnet18-v1-7.onnx-osd# In the interest of efficiency for my development and debugging of onnx2tf,# the default configuration shows a large amount of debug level logs.# However, for most users, a large number of debug logs are unnecessary.# If you want to reduce the amount of information displayed in the conversion log,# you can change the amount of information in the log by specifying the# `--verbosity` or `-v` option as follows.# Possible values are "debug", "info", "warn", and "error".wgethttps://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf-iresnet18-v1-7.onnx-vinfo# Override undefined batch size or other dimensions with static values.# If the model has undefined dimensions, rewriting them to a static size will significantly# improve the success rate of the conversion.# The `-b` option overwrites the zero-dimensional batch size with the number specified# without input OP name.# Note that if there are multiple input OPs, the zero dimension of all input OPs is# forced to be rewritten.# The `-ois` option allows undefined dimensions in all dimensions, including# the zero dimensionality, to be overwritten to a static shape, but requires# the input OP name to be specified.# e.g. -ois data1:1,3,224,224 data2:1,255 data3:1,224,6wgethttps://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf-iresnet18-v1-7.onnx-b1or
onnx2tf-iresnet18-v1-7.onnx-oisdata:1,3,224,224# Suppress automatic transposition of input OPs from NCW, NCHW, NCDHW to NWC, NHWC, NDHWC.# onnx2tf is a specification that automatically transposes the input OP to [N,H,W,C] format# before converting the model. However, since onnx2tf cannot determine from the structure of# the model whether the input data is image, audio data, or something else, it unconditionally# transposes the channels. Therefore, it is the models of STT/TTS models where the input is# not NHWC that tend to have particular problems with the automatic transposition of the# input OP.# If you do not want input OPs to be automatically transposed, you can disable automatic# transposition of input OPs by specifying the `-kat` option.wgethttps://github.com/PINTO0309/onnx2tf/releases/download/1.1.28/double_gru.onnx# INPUT OPs: "spec": float32[1,3,257,1], "states_in": float32[2,1,32]# The following command suppresses the automatic transposition of "states_in" and converts it.onnx2tf-idouble_gru.onnx-katstates_in# Keras h5 format# .h5, .json, .keras, .weights.h5, .weights.keras, .data-00000-of-00001, .indexwgethttps://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf-iresnet18-v1-7.onnx-oh5# Keras keras_v3 format (TensorFlow v2.12.0 or later only)wgethttps://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf-iresnet18-v1-7.onnx-okv3# TensorFlow v1 (.pb) formatwgethttps://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf-iresnet18-v1-7.onnx-otfv1pb# INT8 Quantization, Full INT8 Quantization# INT8 Quantization with INT16 activation, Full INT8 Quantization with INT16 activation# Dynamic Range Quantizationwgethttps://github.com/PINTO0309/onnx2tf/releases/download/1.1.1/emotion-ferplus-8.onnx# INT8 Quantization (per-channel)onnx2tf-iemotion-ferplus-8.onnx-oiqt# INT8 Quantization (per-tensor)onnx2tf-iemotion-ferplus-8.onnx-oiqt-qtper-tensor# Split the model at the middle position for debugging# Specify the output name of the OPonnx2tf-iresnet18-v1-7.onnx-onimcresnetv15_stage2_conv1_fwdresnetv15_stage2_conv2_fwd# Suppress generation of Flex OP and replace with Pseudo-Function# [Asin, Acos, Atan, Abs, PReLU, LeakyReLU, Power, GatherND, Neg, HardSwish, Erf, GeLU]# Below is a sample of replacing Erf with another set of operations.wgethttps://s3.ap-northeast-2.wasabisys.com/temp-models/onnx2tf_readme/Erf_11.onnx
onnx2tf-iErf_11.onnx-rtpoErf# High-dimensional Transpose decomposition# If you do not like FlexTranspose being generated, try `-nodaftc`.# Suppresses the generation of FlexTranspose by decomposing Transpose# to the specified number of dimensions.# In TensorFlow v2.12.0 and later, up to 6 dimensions are converted to normal Transpose;# in v2.11.0 and earlier, up to 5 dimensions are converted to normal Transpose.# Note that specifying `2` for the `-nodaftc` option causes all Transpose OPs to disappear# from the model structure.# Below is an example of decomposing a Transpose of 5 or more dimensions into a Transpose# of 4 dimensions.onnx2tf-ixxxx.onnx-nodaftc4# High-dimensional Slice(StridedSlice) decomposition# If your special circumstances do not allow you to deploy a `StridedSlice` with more than# 5 dimensions to a device, you can use the `-nodafsc` option to decompose the `StridedSlice`# into a process with 4 or fewer dimensions.# Below is an example of decomposing a `StridedSlice` of 5 or more dimensions into a# `StridedSlice` of 4 dimensions.onnx2tf-ixxxx.onnx-nodafsc4# Float16 inference doubling on devices with ARM64 ARMv8.2 or higher instruction set# Double the inference speed with Float16 precision tflite models on devices with# high-performance CPUs such as Snapdragon.# (Pixel 3a, Pixel 5a, Pixel 7, Galaxy M12 and Galaxy S22, ...)# XNNPACK float16 inference on certain ARM64 cores is 2x faster.# Unfortunately, Float16 inference cannot be accelerated when using the RaspberryPi4's# ARM64 CPU.onnx2tf-ixxxx.onnx-eatfp16# Parameter replacement (Resize,Transpose,Softmax)rmreplace.json
wgethttps://github.com/PINTO0309/onnx2tf/releases/download/1.1.27/human_segmentation_pphumanseg_2021oct.onnx
wgethttps://github.com/PINTO0309/onnx2tf/releases/download/1.1.27/replace.json
onnx2tf-ihuman_segmentation_pphumanseg_2021oct.onnx-prfreplace.json3. Accuracy checkPerform error checking of ONNX output and TensorFlow output. Verify that the error of all outputs, one operation at a time, is below a certain threshold. Automatically determines before and after which OPs the tool's automatic conversion of the model failed. Know where dimensional compression, dimensional expansion, and dimensional transposition byReshapeandTrasposeare failing. Once you have identified the problem area, you can refer to the tutorial onParameter replacementto modify the tool's behavior.After many upgrades, the need for JSON parameter correction has become much less common, but there are still some edge cases where JSON correction is required. If the PC has sufficient free space in its RAM, onnx2tf will convert the model while carefully performing accuracy checks on all OPs. Thus, at the cost of successful model conversion, the conversion speed is a little slower. If the amount of RAM required for the accuracy check is expected to exceed 80% of the total available RAM capacity of the entire PC, the conversion operation will be performed without an accuracy check. Therefore, if the accuracy of the converted model is found to be significantly degraded, the accuracy may be automatically corrected by re-conversion on a PC with a large amount of RAM. For example, my PC has 128GB of RAM, but the StableDiffusion v1.5 model is too complex in its structure and consumed about 180GB of RAM in total with 50GB of SWAP space.-oisan option to overwrite the input OP to a static size if it has undefined dimensions.-cotofoption checks the accuracy of all OPs one by one.-cotoais the error value of the threshold for determining an accuracy error. If there are undefined dimensions in the input OP, it is better to fix them to the static geometry to improve the accuracy of the accuracy measurement.Also, you can use the-cindoption to specify custom input for-cotof, instead of using the default dummy input. Otherwise, all input values will be set to 1. For more information about the-cindoption, please refer tohere.The-cotofoption only compares the original ONNX and converted TensorFlow (Keras) models at Float32 precision, not at Float16 or INT8 precision.onnx2tf -i mobilenetv2-12.onnx -ois input:1,3,224,224 -cotof -cotoa 1e-1

or

onnx2tf -i mobilenetv2-12.onnx -b 1 -cotof -cotoa 1e-1

or

onnx2tf -i mobilenetv2-12.onnx -cotof -cotoa 1e-1 -cind "input" "/your/path/x.npy"4. Match tflite input/output names and input/output order to ONNXIf you want to match tflite's input/output OP names and the order of input/output OPs with ONNX, you can use theinterpreter.get_signature_runner()to infer this after using the-coion/--copy_onnx_input_output_names_to_tfliteoption to output tflite file. See:https://github.com/PINTO0309/onnx2tf/issues/228importtorchimportonnxruntimeimportnumpyasnpimportonnx2tfimporttensorflowastffromtensorflow.lite.pythonimportinterpreterastflite_interpreterclassModel(torch.nn.Module):defforward(self,x,y):return{"add":x+y,"sub":x-y,}# Let's double check what PyTorch gives usmodel=Model()pytorch_output=model.forward(10,2)print("[PyTorch] Model Predictions:",pytorch_output)# First, export the above model to ONNXtorch.onnx.export(Model(),{"x":10,"y":2},"model.onnx",opset_version=16,input_names=["x","y"],output_names=["add","sub"],)# And check its outputsession=onnxruntime.InferenceSession("model.onnx")onnx_output=session.run(["add","sub"],{"x":np.array(10),"y":np.array(2)})print("[ONNX] Model Outputs:",[o.nameforoinsession.get_outputs()])print("[ONNX] Model Predictions:",onnx_output)# Now, let's convert the ONNX model to TFonnx2tf.convert(input_onnx_file_path="model.onnx",output_folder_path="model.tf",copy_onnx_input_output_names_to_tflite=True,non_verbose=True,)# Now, test the newer TFLite modelinterpreter=tf.lite.Interpreter(model_path="model.tf/model_float32.tflite")tf_lite_model=interpreter.get_signature_runner()inputs={'x':np.asarray([10],dtype=np.int64),'y':np.asarray([2],dtype=np.int64),}tf_lite_output=tf_lite_model(**inputs)print("[TFLite] Model Predictions:",tf_lite_output)[PyTorch] Model Predictions:
  {
    'add': 12,
    'sub': 8
  }
[ONNX] Model Outputs:
  [
    'add',
    'sub'
  ]
[ONNX] Model Predictions:
  [
    array(12, dtype=int64),
    array(8, dtype=int64)
  ]
[TFLite] Model Predictions:
  {
    'add': array([12]),
    'sub': array([8])
  }5. Rewriting of tflite input/output OP names andsignature_defsIf you do not like tflite input/output names such asserving_default_*:0orStatefulPartitionedCall:0, you can rewrite them using the following tools and procedures. It can be rewritten from any name to any name, so it does not have to beserving_default_*:0orStatefulPartitionedCall:0.https://github.com/PINTO0309/tflite-input-output-rewriter# Install custom flatcwgethttps://github.com/PINTO0309/onnx2tf/releases/download/1.7.3/flatc.tar.gz\&&tar-zxvfflatc.tar.gz\&&sudochmod+xflatc\&&sudomvflatc/usr/bin/\&&rmflatc.tar.gz# Path checkwhichflatc
/usr/bin/flatc# Install tfliteiorewriterpipinstall-UtfliteiorewriterBeforetfliteiorewriter\-ixxxx.tflite\-rserving_default_input_1:0aaa\-rStatefulPartitionedCall:0bbbAfter6. Embed metadata in tfliteIf you want to embed label maps, quantization parameters, descriptions, etc. into your tflite file, you can refer to the official tutorial and try it yourself. For now, this tool does not plan to implement the ability to append metadata, as I do not want to write byte arrays to the tflite file that are not essential to its operation.Adding metadata to TensorFlow Lite modelshttps://www.tensorflow.org/lite/models/convert/metadata7. If the accuracy of the INT8 quantized model degrades significantlyIt is a matter of model structure. The activation function (SiLU/Swish), kernel size and stride forPooling, and kernel size and stride forConvshould be completely revised. See:https://github.com/PINTO0309/onnx2tf/issues/244#issuecomment-1475128445, andhttps://github.com/PINTO0309/onnx2tf/issues/269If you want to see the difference in quantization error betweenSiLUandReLU, please check this Gist by@motokimurawho helped us in our research. Thanks Motoki!Gist:Quantization error simulation of SiLU (Swish) activationThe accuracy error rates after quantization for different activation functions are shown in the figure below. The graph plots the distribution of absolute error, so a position with a higher value on the horizontal axis indicates a larger error. The vertical axis is the number of samples.SiLU (Swish)produces catastrophic errors after INT8 quantization.e.g. YOLOX-Nanohttps://github.com/motokimura/yolox-ti-lite_tflitehttps://github.com/TexasInstruments/edgeai-yoloxBeforeAfterSwish/SiLUReLUDepthwiseConv2DConv2DMaxPool, kernel_size=5x5,9x9,13x13MaxPool, kernel_size=3x3### Float32 - YOLOX-Nano
(1, 52, 52, 85)
array([[[
    [ 0.971787,  0.811184,  0.550566, ..., -5.962632, -7.403673, -6.735206],
    [ 0.858804,  1.351296,  1.231673, ..., -6.479690, -8.277064, -7.664936],
    [ 0.214827,  1.035119,  1.458006, ..., -6.291425, -8.229385, -7.761562],
        ...,
    [ 0.450116,  1.391900,  1.533354, ..., -5.672194, -7.121591, -6.880231],
    [ 0.593133,  2.112723,  0.968755, ..., -6.150078, -7.370633, -6.874294],
    [ 0.088263,  1.985220,  0.619998, ..., -5.507928, -6.914980, -6.234259]]]]),

### INT8 - YOLOX-Nano
(1, 52, 52, 85)
array([[[
    [ 0.941908,  0.770652,  0.513768, ..., -5.993958, -7.449634, -6.850238],
    [ 0.856280,  1.284420,  1.198792, ..., -6.507727, -8.391542, -7.792146],
    [ 0.256884,  0.941908,  1.455676, ..., -6.336471, -8.305914, -7.877774],
        ...,
    [ 0.342512,  1.370048,  1.541304, ..., -5.737075, -7.192750, -7.107122],
    [ 0.513768,  2.226327,  1.027536, ..., -6.165215, -7.449634, -7.021494],
    [ 0.085628,  2.055072,  0.685024, ..., -5.480191, -7.021494, -6.422099]]]]),Other recommended replacement OPBeforeAfterHardSwishReLUReLU6Paper:A Quantization-Friendly Separable Convolution for MobileNetshttps://arxiv.org/pdf/1803.08607.pdfReLUQuantization range collapse due to non-zero constant paddingIf padding is performed with a constant other than zero, the padding value may destroy the quantization range of the input tensor. For example, the pattern is shown in the figure below. TheMaxPool2Dis done after padding the 4 sides of the input tensor with the minimum value of Float32. It seems that if INT8 quantization is performed with this structure, the quantization range is determined byMaxPool2Dduring quantization, including the values padded to the tensor. See:#444Therefore, the following two similar examples are equally likely to result in divergent output values for the model after INT8 quantization, with all output values being Nan or zero.Pattern with fixed value-255.0padded on 4 sides of tensorPattern with fixed value-128.0padded on 4 sides of tensor8. Calibration data creation for INT8 quantizationCalibration data (.npy) for INT8 quantization (-cind) is generated as follows. This is a sample when the data used for training is image data. See:https://github.com/PINTO0309/onnx2tf/issues/222https://www.tensorflow.org/lite/performance/post_training_quantizationimportcv2importglobimportnumpyasnp# Not used during data generation ################################# You will need to do the calculations yourself using the test dataMEAN=np.asarray([[[[0.485,0.456,0.406]]]],dtype=np.float32)# [1,1,1,3]STD=np.asarray([[[[0.229,0.224,0.225]]]],dtype=np.float32)# [1,1,1,3]# Not used during data generation ################################files=glob.glob("data/*.png")img_datas=[]foridx,fileinenumerate(files):bgr_img=cv2.imread(file)rgb_img=cv2.cvtColor(bgr_img,cv2.COLOR_BGR2RGB)resized_img=cv2.resize(rgb_img,dsize=(200,112))extend_batch_size_img=resized_img[np.newaxis,:]normalized_img=extend_batch_size_img/255.0# 0.0 - 1.0print(f'{str(idx+1).zfill(2)}. extend_batch_size_img.shape:{extend_batch_size_img.shape}')# [1,112,200,3]img_datas.append(extend_batch_size_img)calib_datas=np.vstack(img_datas)print(f'calib_datas.shape:{calib_datas.shape}')# [10,112,200,3]np.save(file='data/calibdata.npy',arr=calib_datas)loaded_data=np.load('data/calibdata.npy')print(f'loaded_data.shape:{loaded_data.shape}')# [10,112,200,3]"""-cind INPUT_NAME NUMPY_FILE_PATH MEAN STDint8_calib_datas = (loaded_data - MEAN) / STD # -1.0 - 1.0e.g. How to specify calibration data in CLI or Script respectively.1. CLI-cind "pc_dep" "data/calibdata.npy" "[[[[0.485,0.456,0.406]]]]" "[[[[0.229,0.224,0.225]]]]"-cind "feat" "data/calibdata2.npy" "[[[[0.123,...,0.321]]]]" "[[[[0.112,...,0.451]]]]"2. Scriptcustom_input_op_name_np_data_path=[["pc_dep", "data/calibdata.npy", [[[[0.485,0.456,0.406]]]], [[[[0.229,0.224,0.225]]]]],["feat", "data/calibdata2.npy", [[[[0.123,...,0.321]]]], [[[[0.112,...,0.451]]]],]"""9. INT8 quantization of models with multiple inputs requiring non-image dataIf you do not need to perform INT8 quantization with this tool alone, the following method is the easiest.The-osdoption will output asaved_model.pbin thesaved_modelfolder with the full size required for quantization. That is, a default signature namedserving_defaultis embedded in.pb. The-boption is used to convert the batch size by rewriting it as a static integer.Note: INT8 TFLite generated by following this procedure as is will result in a model with significantly degraded accuracy. This tutorial only demonstrates the INT8 quantization procedure; if you wish to correct for accuracy, please refer toParameter replacementto correct for transposition errors in the operation.# Ref: https://github.com/onnx/models/tree/main/text/machine_comprehension/bert-squadwgethttps://s3.ap-northeast-2.wasabisys.com/temp-models/onnx2tf_248/bertsquad-12.onnx

onnx2tf-ibertsquad-12.onnx-b1-osd-cotofUse thesaved_model_clicommand to check thesaved_modelsignature. INT8 quantization calibration using signatures allows correct control of the input order of data for calibration. Therefore, calibration with signatures is recommended for INT8 quantization of models with multiple inputs.saved_model_clishow--dirsaved_model/--tag_setserve--signature_defserving_default

ThegivenSavedModelSignatureDefcontainsthefollowinginput(s):inputs['input_ids_0']tensor_info:dtype:DT_INT64shape:(1,256)name:serving_default_input_ids_0:0inputs['input_mask_0']tensor_info:dtype:DT_INT64shape:(1,256)name:serving_default_input_mask_0:0inputs['segment_ids_0']tensor_info:dtype:DT_INT64shape:(1,256)name:serving_default_segment_ids_0:0inputs['unique_ids_raw_output___9_0']tensor_info:dtype:DT_INT64shape:(1)name:serving_default_unique_ids_raw_output___9_0:0Calibrate by specifying the input OP name displayed ininputs. Thenp.ones([xxx], dtype=np.int64)part must be replaced with the correct calibration test data. In practice, several pieces of data used for training are extracted and used.importtensorflowastfimportnumpyasnpdefrepresentative_dataset():unique_ids=np.ones([10,256],dtype=np.int64)segment_ids=np.ones([10,256],dtype=np.int64)input_masks=np.ones([10,256],dtype=np.int64)input_ids=np.ones([10],dtype=np.int64)forunique_id,segment_id,input_mask,input_id\inzip(unique_ids,segment_ids,input_masks,input_ids):yield{"unique_ids_raw_output___9_0":unique_id,"segment_ids_0":segment_id,"input_mask_0":input_mask,"input_ids_0":input_id,}converter=tf.lite.TFLiteConverter.from_saved_model('saved_model')converter.optimizations=[tf.lite.Optimize.DEFAULT]converter.representative_dataset=representative_datasetconverter.target_spec.supported_ops=[tf.lite.OpsSet.TFLITE_BUILTINS_INT8]converter.inference_input_type=tf.int8# or tf.uint8converter.inference_output_type=tf.int8# or tf.uint8tflite_quant_model=converter.convert()withopen('saved_model/int8_model.tflite','wb')asw:w.write(tflite_quant_model)https://www.tensorflow.org/lite/performance/post_training_quantizationSee:https://github.com/PINTO0309/onnx2tf/issues/24810. Fixing the output of NonMaxSuppression (NMS)PyTorch'sNonMaxSuppression (torchvision.ops.nms)and ONNX'sNonMaxSuppressionare not fully compatible. TorchVision's NMS is very inefficient. Therefore, it is inevitable that converting ONNX using NMS in object detection models and other models will be very redundant and will be converted with a structure that is difficult for TensorFlow.js and TFLite models to take advantage of in devices. This is due to the indefinite number of tensors output by the NMS. In this chapter, I share how to easily tune the ONNX generated using TorchVision's redundant NMS to generate an optimized NMS.There are multiple issues with TorchVision's NMS. First, the batch size specification is not supported; second, themax_output_boxes_per_classparameter cannot be specified. Please see the NMS sample ONNX part I generated. Themax_output_boxes_per_classhas been changed to896instead of-Infinity. The biggest problem with TorchVision NMS is that it generates ONNX withmax_output_boxes_per_classset to-Infinityor9223372036854775807 (Maximum value of INT64), resulting in a variable number of NMS outputs from zero to infinite. Thus, by rewriting-Infinityor9223372036854775807 (Maximum value of INT64)to a constant value, it is possible to output an NMS that can be effortlessly inferred by TFJS or TFLite.Here you will find committed ONNX components optimized for various devices.https://github.com/PINTO0309/components_of_onnx/tree/main/components_of_onnx/opsIn the following example, themax_output_boxes_per_classof NMS in the post-processing generated by YOLOv7 is changed from-Infinityor9223372036854775807 (Maximum value of INT64)to20, as shown in the figure below. The namemain01_max_output_boxes_per_classhas been rewritten by me for clarity, but it originally appears asmax_output_boxes_per_class.Simply execute the following command. The command rewrites the specified attribute value of the OP specified by ONNX.pipinstallsam4onnx

sam4onnx\--op_namemain01_nonmaxsuppression11\--input_onnx_file_pathyolov7.onnx\--output_onnx_file_pathnms_yolov7_update.onnx\--input_constantsmain01_max_output_boxes_per_classint64[20]A tutorial on one of my ONNX modification tools,sam4onnx, can be found here.https://github.com/PINTO0309/sam4onnxMany detailed tutorials are provided below, so if you are interested, please play with them.https://github.com/PINTO0309/PINTO_model_zoo/tree/main/307_YOLOv7/post_process_gen_toolsFinally, simply convert ONNX to TFLite or saved_model or TFJS using onnx2tf. onnx2tf performs an internal operation to automatically optimize the NMS output to a fixed shape ifmax_output_boxes_per_classis set to a value other than-Infinityand9223372036854775807 (Maximum value of INT64). Specify--output_nms_with_dynamic_tensoror-onwdtif you do not want to optimize for a fixed shape.onnx2tf -i nms_yolov7_update.onnx -osd -cotofI would be happy if this is a reference for Android + Java or TFJS implementations. There are tons more tricky model optimization techniques described in my blog posts, so you'll have to find them yourself. I don't dare to list the URL here because it is annoying to see so manyissuesbeing posted. And unfortunately, all articles are in Japanese.11. RNN (RNN, GRU, LSTM) Inference AccelerationTensorFlow's RNN has a speedup option calledunroll. The network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences. onnx2tf allows you to deploy RNNs into memory-intensive operations by specifying the--enable_rnn_unrollor-eruoptions. The--enable_rnn_unrolloption is available forRNN,GRU, andLSTM.Kerashttps://keras.io/api/layers/recurrent_layers/lstm/TensorFlowhttps://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTMAn example ofBidirectionalLSTMconversion with the--enable_rnn_unrolloption is shown below. Please ignore that the shapes of the input and output tensors do not match, since the samples are shown by picking up separate models.ONNXLSTM (Bidirectional)BidirectionalLSTMwith--enable_rnn_unrolloption unspecifiedRecurrent layer is implemented from scratch.BidirectionalLSTMwith--enable_rnn_unrolloption12. If the accuracy of the Float32 model degrades significantlyThe pattern of accuracy degradation of the converted model does not only occur when INT8 quantization is performed. A special edge case is when there is a problem with the implementation of a particular OP on the TFLite runtime side. Below, I will reproduce the problem by means of a very simple CNN model and further explain its workaround. Here is the issue that prompted me to add this explanation.[Conv-TasNet] Facing issue in converting Conv-TasNet model #447Download a sample model for validation.curl\-Lhttps://github.com/PINTO0309/onnx2tf/files/12367312/prelu_check.onnx.zip\-oprelu_check.onnx.zip

unzipprelu_check.onnx.zipThe part of the downloaded model where the problem occurs is thePRelupart in the figure below.ONNXReproduce the problem. The following command converts an ONNX file to a TFLite file.onnx2tf-iprelu_check.onnx-cotofThe conversion was successful and, as shown in the figure below, the inference test results from ONNX and the inference results for the Float32 model in TensorFlow (Keras) match perfectly. It is important to note that the comparison of inference results between ONNX and TensorFlow transformed models is comparing ONNX models with TensorFlow (Keras) models, not ONNX models with TFLite models.Conversion resultstfliteNow, let's try inference with the TFLite runtime instead of the TensorFlow runtime.test.pyimporttimeimportnumpyasnpnp.random.seed(0)importtensorflowastf# Load TFLite modelinterpreter=tf.lite.Interpreter(model_path="./saved_model/prelu_check_float32.tflite")interpreter.allocate_tensors()tensor_shape=(256,20)input_data={'waveform':np.random.randn(*tensor_shape).astype(np.float32)}# Load and preprocessinput_details=interpreter.get_input_details()input_shape=input_details[0]['shape']print(input_shape)# Run inferenceinterpreter.set_tensor(input_details[0]['index'],input_data["waveform"])separate_time=time.time()interpreter.invoke()print("Done!{:.3f}s".format(time.time()-separate_time))output_details=interpreter.get_output_details()output_data=interpreter.get_tensor(output_details[0]['index'])output_data=[]foroutput_detailinoutput_details:output_data.append(interpreter.get_tensor(output_detail['index']))print(output_data)Oddly enough, the output value ofPReLUcontains multiplenan. However, as can be seen by converting the ONNX model to the middle of the model using the-onimcoption,nandoes not occur until just beforePReLU. Thus, it is clear that thePReLUOP in the TFLite runtime has a problem with divergent inference results.TFLite inference resultsThe following is a work-around to avoid this problem. Use the-rtpooption to replacePReLUwith a similar primitive operation when transforming a model, and then perform the model transformation.onnx2tf-iprelu_check.onnx-cotof-rtpoPReLUAs before, the inference results from ONNX and TensorFlow (Keras) match perfectly.Conversion resultsHowever,-rtpo PReLUwill generate a .tflite file with thePReluOP replaced by a primitive OP combination.tfliteAgain, run the test code to check the inference results. The figure below shows that nonanoccurs when inference is performed by replacing thePReLUOP with only combinations of primitive operations. In other words, it is important to know that large arithmetic errors are not only due to the broken structure of the model, but can also be caused by internal implementations such as the TFLite runtime. I have implemented the-rtpooption to replace operators as a work-around to avoid such runtime problems.TFLite inference results13. Problem of extremely large calculation error inInstanceNormalizationEven if the conversion is successful,InstanceNormalizationtends to have very large errors. This is an ONNX specification.See.1:https://discuss.pytorch.org/t/understanding-instance-normalization-2d-with-running-mean-and-running-var/144139See.2:https://github.com/pytorch/pytorch/issues/72057I verified this with a very simple sample model. There are more than 8 million elements, and the calculation error reached1e-2.14. Inference with dynamic tensors in TFLiteFor some time now, TFLite runtime has supported inference by dynamic tensors. However, the existence of this important function is not widely recognized. In this chapter, I will show how I can convert an ONNX file that contains dynamic geometry in batch size directly into a TFLite file that contains dynamic geometry and then further infer it in variable batch conditions. The issue that inspired me to add this tutorial is here.[Dynamic batch / Dynamic shape] onnx model with dynamic input is converted to tflite with static input 1 #441, orCannot use converted model with dynamic input shape #521First, download the sample ONNX file.wget https://s3.ap-northeast-2.wasabisys.com/temp-models/onnx2tf_441/osnet_x0_25_msmt17.onnxThis model calculates the similarity of features by cosine similarity. The batch size dimension of the input tensor isbatch, allowing various numbers of images to be input simultaneously. This is often used, for example, to achieve tracking by calculating the similarity of people or objects reflected between successive video frames. However, the total number of objects to be tracked changes rapidly with each video frame because the number of people and objects in the image constantly increases and decreases. Therefore, there is a very significant use case for generating models with variable settings for the number of input images (batch size) of the model.Convert the downloadedOSNettotfliteandsaved_modelas a variable batch. If you do not specify the-bor-oisoptions, onnx2tf does not change the batch size asN. The only important point is to convert the model with the-osdand-coionoptions. Note that if you use the-coionoption, you must installflatbuffers-compilerwithapt-get install, run the commands for building the environment described first in thisREADME, or use aDocker container.onnx2tf -i osnet_x0_25_msmt17.onnx -osd -coion.tfliteWhen viewing tflite in Netron, the batch size appears to be fixed at1.saved_modelHowever, checking the structure ofsaved_model, the batch size is correctly set to-1.saved_model_clishow--dirsaved_model/--all

MetaGraphDefwithtag-set:'serve'containsthefollowingSignatureDefs:

signature_def['__saved_model_init_op']:ThegivenSavedModelSignatureDefcontainsthefollowinginput(s):ThegivenSavedModelSignatureDefcontainsthefollowingoutput(s):outputs['__saved_model_init_op']tensor_info:dtype:DT_INVALIDshape:unknown_rankname:NoOpMethodnameis:

signature_def['serving_default']:ThegivenSavedModelSignatureDefcontainsthefollowinginput(s):inputs['images']tensor_info:dtype:DT_FLOATshape:(-1,256,128,3)name:serving_default_images:0ThegivenSavedModelSignatureDefcontainsthefollowingoutput(s):outputs['output']tensor_info:dtype:DT_FLOATshape:(-1,512)name:PartitionedCall:0Methodnameis:tensorflow/serving/predictTo prove that the tflite structure has been converted correctly, I will convert the tflite to JSON and look at the structure.dockerrun--rm-it\-v`pwd`:/home/user/workdir\ghcr.io/pinto0309/tflite2json2tflite:latest

./flatc-t\--strict-json\--defaults-json\-oworkdir\./schema.fbs--workdir/saved_model/osnet_x0_25_msmt17_float32.tflite

ls-lworkdir

-rw-rw-r--1useruser921564Aug410:24osnet_x0_25_msmt17.onnx
-rw-r--r--1useruser10369524Aug410:30osnet_x0_25_msmt17_float32.json
drwxrwxr-x4useruser4096Aug410:26saved_modelosnet_x0_25_msmt17_float32.json"shape_signature"is correctly set to-1. However,"shape"is set to1. This could be a problem with TFLiteConverter, or it could be a problem with Netron's graphical display capabilities.In other words, although onnx2tf converts TFLiteConverer as specified, with the batch size of-1without any model processing, only Netron's display is broken. This is a problem I have known for quite some time. However, the inference itself does not cause the problem.If you want to infer in variable batches, you need to infer usingsignature. In such cases, the-coionoption must be specified when converting the model. Note that I have identified a problem with quantization with the-coionoption, which can corrupt tflite files.https://github.com/PINTO0309/onnx2tf/issues/429https://github.com/PINTO0309/onnx2tf#4-match-tflite-inputoutput-names-and-inputoutput-order-to-onnxtest.py- Batch size:5importnumpyasnpimporttensorflowastffrompprintimportpprintinterpreter=tf.lite.Interpreter(model_path="saved_model/osnet_x0_25_msmt17_float32.tflite")tf_lite_model=interpreter.get_signature_runner()inputs={'images':np.ones([5,256,128,3],dtype=np.float32),}tf_lite_output=tf_lite_model(**inputs)print(f"[TFLite] Model Predictions shape:{tf_lite_output['output'].shape}")print(f"[TFLite] Model Predictions:")pprint(tf_lite_output)Results[TFLite] Model Predictions shape: (5, 512)
[TFLite] Model Predictions:
{'output': array([[0.0000000e+00, 2.4730086e-04, 0.0000000e+00, ..., 1.0528549e+00,
        3.7874988e-01, 0.0000000e+00],
       [0.0000000e+00, 2.4730086e-04, 0.0000000e+00, ..., 1.0528549e+00,
        3.7874988e-01, 0.0000000e+00],
       [0.0000000e+00, 2.4730086e-04, 0.0000000e+00, ..., 1.0528549e+00,
        3.7874988e-01, 0.0000000e+00],
       [0.0000000e+00, 2.4730086e-04, 0.0000000e+00, ..., 1.0528549e+00,
        3.7874988e-01, 0.0000000e+00],
       [0.0000000e+00, 2.4730084e-04, 0.0000000e+00, ..., 1.0528525e+00,
        3.7874976e-01, 0.0000000e+00]], dtype=float32)}test.py- Batch size:3importnumpyasnpimporttensorflowastffrompprintimportpprintinterpreter=tf.lite.Interpreter(model_path="saved_model/osnet_x0_25_msmt17_float32.tflite")tf_lite_model=interpreter.get_signature_runner()inputs={'images':np.ones([3,256,128,3],dtype=np.float32),}tf_lite_output=tf_lite_model(**inputs)print(f"[TFLite] Model Predictions shape:{tf_lite_output['output'].shape}")print(f"[TFLite] Model Predictions:")pprint(tf_lite_output)Results[TFLite] Model Predictions shape: (3, 512)
[TFLite] Model Predictions:
{'output': array([[0.0000000e+00, 2.4730084e-04, 0.0000000e+00, ..., 1.0528525e+00,
        3.7874976e-01, 0.0000000e+00],
       [0.0000000e+00, 2.4730084e-04, 0.0000000e+00, ..., 1.0528525e+00,
        3.7874976e-01, 0.0000000e+00],
       [0.0000000e+00, 2.4730084e-04, 0.0000000e+00, ..., 1.0528525e+00,
        3.7874976e-01, 0.0000000e+00]], dtype=float32)}15. Conversion to TensorFlow.jsWhen converting to TensorFlow.js, process as follows.pipinstalltensorflowjs

onnx2tf-imobilenetv2-12.onnx-oisinput:1,3,224,224-osd

tensorflowjs_converter\--input_formattf_saved_model\--output_formattfjs_graph_model\saved_model\tfjs_modelSee:https://github.com/tensorflow/tfjs/tree/master/tfjs-converter16. Conversion to CoreMLWhen converting to CoreML, process as follows. The-koption is for conversion while maintaining the input channel order in ONNX's NCHW format.pipinstallcoremltools

onnx2tf-imobilenetv2-12.onnx-kinput-oisinput:1,3,224,224-osdimportcoremltoolsasctFOLDER_PATH='saved_model'model=ct.convert(model=FOLDER_PATH,source='tensorflow',)model.save(f'{FOLDER_PATH}/model.mlmodel')See:https://github.com/apple/coremltoolsCLI Parameteronnx2tf -h

usage: onnx2tf
[-h]
(-i INPUT_ONNX_FILE_PATH | -V)
[-o OUTPUT_FOLDER_PATH]
[-osd]
[-oh5]
[-okv3]
[-otfv1pb]
[-ow]
[-coion]
[-oiqt]
[-qt {per-channel,per-tensor}]
[-cind INPUT_NAME NUMPY_FILE_PATH MEAN STD]
[-ioqd {int8,uint8}]
[-nuo]
[-nuonag]
[-b BATCH_SIZE]
[-ois OVERWRITE_INPUT_SHAPE [OVERWRITE_INPUT_SHAPE ...]]
[-nlt]
[-onwdt]
[-k KEEP_NCW_OR_NCHW_OR_NCDHW_INPUT_NAMES [KEEP_NCW_OR_NCHW_OR_NCDHW_INPUT_NAMES ...]]
[-kt KEEP_NWC_OR_NHWC_OR_NDHWC_INPUT_NAMES [KEEP_NWC_OR_NHWC_OR_NDHWC_INPUT_NAMES ...]]
[-kat KEEP_SHAPE_ABSOLUTELY_INPUT_NAMES [KEEP_SHAPE_ABSOLUTELY_INPUT_NAMES ...]]
[-onimc OUTPUT_NAMES [OUTPUT_NAMES ...]]
[-dgc]
[-eatfp16]
[-ebu]
[-eru]
[-dsft]
[-nodaftc]
[-dsfs]
[-dsm]
[-nodafsc]
[-ofgd]
[-rari64 | -rarf32 | -rafi64 | -raff32]
[-fasr FUSED_ARGMAX_SCALE_RATIO]
[-rtpo REPLACE_TO_PSEUDO_OPERATORS [REPLACE_TO_PSEUDO_OPERATORS ...]]
[-me MVN_EPSILON]
[-prf PARAM_REPLACEMENT_FILE]
[-cgdc]
[-coto | -cotof]
[-coton]
[-cotor CHECK_ONNX_TF_OUTPUTS_ELEMENTWISE_CLOSE_RTOL]
[-cotoa CHECK_ONNX_TF_OUTPUTS_ELEMENTWISE_CLOSE_ATOL]
[-dms]
[-uc]
[-n]
[-v]

optional arguments:
  -h, --help
    show this help message and exit

  -i INPUT_ONNX_FILE_PATH, --input_onnx_file_path INPUT_ONNX_FILE_PATH
    Input onnx file path.

  -V, --version
    Show version and exit.

  -o OUTPUT_FOLDER_PATH, --output_folder_path OUTPUT_FOLDER_PATH
    Output folder path. Default: "saved_model"

  -osd, --output_signaturedefs
    Signature is added to the output for serving or for conversion
    to other model formats. However, this can significantly reduce the speed
    of model conversion and significant increase the size of the model.

  -oh5, --output_h5
    Output model in Keras (hdf5) format.

  -okv3, --output_keras_v3
    Output model in Keras (keras_v3) format.

  -otfv1pb, --output_tfv1_pb
    Output model in TF v1 (.pb) format.

  -ow, --output_weights
    Output weights in hdf5 format.

  -coion, --copy_onnx_input_output_names_to_tflite
    Copy the input/output OP name of ONNX to the input/output OP name of tflite.
    Due to Tensorflow internal operating specifications,
    the input/output order of ONNX does not necessarily match
    the input/output order of tflite.
    Be sure to check that the input/output OP names in the generated
    tflite file have been converted as expected.
    Also, this option generates a huge JSON file as a temporary file for processing.
    Therefore, it is strongly discouraged to use it on large models of hundreds
    of megabytes or more.

  -oiqt, --output_integer_quantized_tflite
    Output of integer quantized tflite.

  -qt {per-channel,per-tensor}, --quant_type {per-channel,per-tensor}
    Selects whether "per-channel" or "per-tensor" quantization is used.
    Default: "per-channel"

  -cind INPUT_NAME NUMPY_FILE_PATH MEAN STD, \
    --custom_input_op_name_np_data_path INPUT_NAME NUMPY_FILE_PATH MEAN STD
    Input name of OP and path of data file (Numpy) for custom input for -cotof or -oiqt,
    and mean (optional) and std (optional).

    <Usage in -cotof>
      When using -cotof, custom input defined by the user, instead of dummy data, is used.
      In this case, mean and std are omitted from the input.
      -cind {input_op_name} {numpy_file_path}
      e.g. -cind onnx::Equal_0 test_cind/x_1.npy -cind onnx::Add_1 test_cind/x_2.npy -cotof
      The input_op_name must be the same as in ONNX,
      and it may not work if the input format is different between ONNX and TF.

    <Usage in -oiqt>
      INPUT Name of OP and path of calibration data file (Numpy) for quantization
      and mean and std.
      The specification can be omitted only when the input OP is a single 4D tensor image data.
      If omitted, it is automatically calibrated using 20 normalized MS-COCO images.
      The type of the input OP must be Float32.
      Data for calibration must be pre-normalized to a range of 0 to 1.
      -cind {input_op_name} {numpy_file_path} {mean} {std}
      Numpy file paths must be specified the same number of times as the number of input OPs.
      Normalize the value of the input OP based on the tensor specified in mean and std.
      (input_value - mean) / std
      Tensors in Numpy file format must be in dimension order after conversion to TF.
      Note that this is intended for deployment on low-resource devices,
      so the batch size is limited to 1 only.

      e.g.
      The example below shows a case where there are three input OPs.
      Assume input0 is 128x128 RGB image data.
      In addition, input0 should be a value that has been divided by 255
      in the preprocessing and normalized to a range between 0 and 1.
      input1 and input2 assume the input of something that is not an image.
      Because input1 and input2 assume something that is not an image,
      the divisor is not 255 when normalizing from 0 to 1.
      "n" is the number of calibration data.

      ONNX INPUT shapes:
        input0: [n,3,128,128]
            mean: [1,3,1,1] -> [[[[0.485]],[[0.456]],[[0.406]]]]
            std:  [1,3,1,1] -> [[[[0.229]],[[0.224]],[[0.225]]]]
        input1: [n,64,64]
            mean: [1,64] -> [0.1, ..., 0.64]
            std:  [1,64] -> [0.05, ..., 0.08]
        input2: [n,5]
            mean: [1] -> [0.3]
            std:  [1] -> [0.07]
      TensorFlow INPUT shapes (Numpy file ndarray shapes):
        input0: [n,128,128,3]
            mean: [1,1,1,3] -> [[[[0.485, 0.456, 0.406]]]]
            std:  [1,1,1,3] -> [[[[0.229, 0.224, 0.225]]]]
        input1: [n,64,64]
            mean: [1,64] -> [0.1, ..., 0.64]
            std:  [1,64] -> [0.05, ..., 0.08]
        input2: [n,5]
            mean: [1] -> [0.3]
            std:  [1] -> [0.07]
      -cind "input0" "../input0.npy" "[[[[0.485,0.456,0.406]]]]" "[[[[0.229,0.224,0.225]]]]"
      -cind "input1" "./input1.npy" "[0.1,...,0.64]" "[0.05,...,0.08]"
      -cind "input2" "input2.npy" "[0.3]" "[0.07]"

    <Using -cotof and -oiqt at the same time>
      To use -cotof and -oiqt simultaneously,
      you need to enter the Input name of OP, path of data file, mean, and std all together.
      And the data file must be in Float32 format,
      and {input_op_name}, {numpy_file_path}, {mean}, and {std} must all be entered.
      Otherwise, an error will occur during the -oiqt stage.

  -ioqd {int8,uint8}, --input_output_quant_dtype {int8,uint8}
    Input and Output dtypes when doing Full INT8 Quantization.
    "int8"(default) or "uint8"

  -nuo, --not_use_onnxsim
    No optimization by onnx-simplifier is performed.
    If this option is used, the probability of a conversion error is very high.

  -nuonag, --not_use_opname_auto_generate
    Automatic generation of each OP name in the old format ONNX file
    and assignment of OP name are not performed.

  -b BATCH_SIZE, --batch_size BATCH_SIZE
    Fixes the dynamic batch size to the specified numeric batch size.
    A value of 1 or more must be specified.

  -ois OVERWRITE_INPUT_SHAPE [OVERWRITE_INPUT_SHAPE ...], \
      --overwrite_input_shape OVERWRITE_INPUT_SHAPE [OVERWRITE_INPUT_SHAPE ...]
    Overwrite the input shape.
    The format is
    "i1:dim0,...,dimN" "i2:dim0,...,dimN" "i3:dim0,...,dimN"
    When there is only one input, for example,
    "data:1,3,224,224"
    When there are multiple inputs, for example,
    "data1:1,3,224,224" "data2:1,3,112" "data3:5"
    A value of 1 or more must be specified.
    Numerical values other than dynamic dimensions are ignored.
    Ignores --batch_size if specified at the same time as --batch_size.

  -nlt, --no_large_tensor
    Suppresses constant bloat caused by Tile OP when optimizing models in onnxsim.
    See: https://github.com/daquexian/onnx-simplifier/issues/178

  -onwdt, --output_nms_with_dynamic_tensor
    The number of bounding boxes in the NMS output results is
    not fixed at the maximum number of max_output_boxes_per_class,
    but rather at the smallest possible number of dynamic tensors.
    If this option is disabled, NMS output is padded to the number
    set in the max_output_boxes_per_class attribute.
    e.g.
    disable --output_nms_with_dynamic_tensor:
        output_tensor_shape: [100, 7]
    enable --output_nms_with_dynamic_tensor:
        output_tensor_shape: [N, 7]

  -k KEEP_NCW_OR_NCHW_OR_NCDHW_INPUT_NAMES [KEEP_NCW_OR_NCHW_OR_NCDHW_INPUT_NAMES ...], \
      --keep_ncw_or_nchw_or_ncdhw_input_names KEEP_NCW_OR_NCHW_OR_NCDHW_INPUT_NAMES \
          [KEEP_NCW_OR_NCHW_OR_NCDHW_INPUT_NAMES ...]
    Holds the NCW or NCHW or NCDHW of the input shape for the specified INPUT OP names.
    If a nonexistent INPUT OP name is specified, it is ignored.
    Valid only for 3D, 4D and 5D input tensors.
    e.g. --keep_ncw_or_nchw_or_ncdhw_input_names "input0" "input1" "input2"

  -kt KEEP_NWC_OR_NHWC_OR_NDHWC_INPUT_NAMES [KEEP_NWC_OR_NHWC_OR_NDHWC_INPUT_NAMES ...], \
      --keep_nwc_or_nhwc_or_ndhwc_input_names KEEP_NWC_OR_NHWC_OR_NDHWC_INPUT_NAMES \
          [KEEP_NWC_OR_NHWC_OR_NDHWC_INPUT_NAMES ...]
    Holds the NWC or NHWC or NDHWC of the input shape for the specified INPUT OP names.
    If a nonexistent INPUT OP name is specified, it is ignored.
    If the input OP name is the same as the input OP name specified
    in the keep_ncw_or_nchw_or_ncdhw_input_names option, it is ignored.
    Valid only for 3D, 4D and 5D input tensors.
    e.g. --keep_nwc_or_nhwc_or_ndhwc_input_names "input0" "input1" "input2"

  -kat KEEP_SHAPE_ABSOLUTELY_INPUT_NAMES [KEEP_SHAPE_ABSOLUTELY_INPUT_NAMES ...], \
      --keep_shape_absolutely_input_names KEEP_SHAPE_ABSOLUTELY_INPUT_NAMES \
        [KEEP_SHAPE_ABSOLUTELY_INPUT_NAMES ...]
    Name of the INPUT that unconditionally maintains its shape.
    If a nonexistent INPUT OP name is specified, it is ignored.
    e.g. --keep_shape_absolutely_input_names "input0" "input1" "input2"

  -onimc OUTPUT_NAMES [OUTPUT_NAMES ...], \
      --output_names_to_interrupt_model_conversion OUTPUT_NAMES [OUTPUT_NAMES ...]
    Output names of ONNX that interrupt model conversion.
    Interrupts model transformation at the specified output name and outputs the
    model partitioned into subgraphs.
    e.g. --output_names_to_interrupt_model_conversion "output0" "output1" "output2"

  -dgc, --disable_group_convolution
    Disable GroupConvolution and replace it with SeparableConvolution for
    output to saved_model format.

  -eatfp16, --enable_accumulation_type_float16 ENABLE_ACCUMULATION_TYPE_FLOAT16
    Hint for XNNPACK fp16 inference on float16 tflite model.
    XNNPACK float16 inference on certain ARM64 cores is 2x faster.
    Float16 inference doubling on devices with ARM64 ARMv8.2 or higher instruction set.
    See: https://github.com/PINTO0309/onnx2tf/pull/553

  -ebu, --enable_batchmatmul_unfold
    BatchMatMul is separated batch by batch to generate a primitive MatMul.

  -eru, --enable_rnn_unroll
    Instead of increasing inference speed by expanding all symbolic loops of
    the RNN (LSTM, GRU, RNN), RAM consumption will increase because all tensors
    are expanded and embedded in the model.
    https://keras.io/api/layers/recurrent_layers/

  -dsft, --disable_suppression_flextranspose
    Disables FlexTranspose generation suppression.

  -nodaftc, --number_of_dimensions_after_flextranspose_compression
    Number of Transpose OP dimensions generated after avoiding FlexTranspose generation.
    Also suppress the creation of the Transpose itself by specifying 2.
    Default: 6

  -dsfs, --disable_suppression_flexstridedslice
    Disables FlexStridedSlice generation suppression.

  -dsm, --disable_strict_mode
    If specified, the conversion speed is greatly accelerated because the strict accuracy
    correction process is skipped, but the frequency of transposition errors increases
    and accuracy errors are more likely to occur. Strict mode is enabled by default.
    As of 2023.05.07, this is a work in progress and is an experimental feature.
    Therefore, only some OPs are converted in strict mode for accuracy correction.

  -nodafsc, --number_of_dimensions_after_flexstridedslice_compression
    Number of StridedSlice OP dimensions generated after avoiding FlexStridedSlice generation.
    Default: 5

  -ofgd, --optimization_for_gpu_delegate
    Replace operations that do not support gpu delegate with those
    that do as much as possible.

  -rari64, --replace_argmax_to_reducemax_and_indicies_is_int64
    Replace ArgMax with a ReduceMax. The returned indicies are int64.
    Only one of replace_argmax_to_reducemax_and_indicies_is_int64
    and replace_argmax_to_reducemax_and_indicies_is_float32
    and replace_argmax_to_fused_argmax_and_indicies_is_int64
    and replace_argmax_to_fused_argmax_and_indicies_is_float32 can be specified.

  -rarf32, --replace_argmax_to_reducemax_and_indicies_is_float32
    Replace ArgMax with a ReduceMax. The returned indicies are float32.
    Only one of replace_argmax_to_reducemax_and_indicies_is_int64
    and replace_argmax_to_reducemax_and_indicies_is_float32
    and replace_argmax_to_fused_argmax_and_indicies_is_int64
    and replace_argmax_to_fused_argmax_and_indicies_is_float32 can be specified.

  -rafi64, --replace_argmax_to_fused_argmax_and_indicies_is_int64
    Replace ArgMax with a Fused_ArgMax. The returned indicies are int64.
    It improves inference speed at the cost of a small sacrifice in accuracy.
    See. https://github.com/tensorflow/models/tree/master/official/projects/edgetpu/vision#argmax-fusion-to-improve-segmentation-model-latency
    Currently, only 4D tensors are supported.
    Only one of replace_argmax_to_reducemax_and_indicies_is_int64
    and replace_argmax_to_reducemax_and_indicies_is_float32
    and replace_argmax_to_fused_argmax_and_indicies_is_int64
    and replace_argmax_to_fused_argmax_and_indicies_is_float32 can be specified.

  -raff32, --replace_argmax_to_fused_argmax_and_indicies_is_float32
    Replace ArgMax with a Fused_ArgMax. The returned indicies are float32.
    It improves inference speed at the cost of a small sacrifice in accuracy.
    See. https://github.com/tensorflow/models/tree/master/official/projects/edgetpu/vision#argmax-fusion-to-improve-segmentation-model-latency
    Currently, only 4D tensors are supported.
    Only one of replace_argmax_to_reducemax_and_indicies_is_int64
    and replace_argmax_to_reducemax_and_indicies_is_float32
    and replace_argmax_to_fused_argmax_and_indicies_is_int64
    and replace_argmax_to_fused_argmax_and_indicies_is_float32 can be specified.

  -fasr FUSED_ARGMAX_SCALE_RATIO, --fused_argmax_scale_ratio FUSED_ARGMAX_SCALE_RATIO
    For Fused ArgMax.
    Scale ratio when generating Fused ArgMax.
    0.0 < fused_argmax_scale_ratio <= 1.0
    Default: 0.5

  -rtpo, --replace_to_pseudo_operators
    Replace list of operators to pseudo operators.
    Full name of the target operators should be given.
    Currently supported operators :
    Asin, Acos, Atan, Abs, PReLU, LeakyReLU, Power, GatherND, Neg, HardSwish, Erf, GeLU

  -me, --mvn_epsilon
    For MeanVarianceNormalization.
    The number to be added to the variance to avoid division by zero
    when normalizing the value.
    (input_tensor - mean) / tf.sqrt(variance + mvn_epsilon)
    Default: 0.0000000001

  -prf PARAM_REPLACEMENT_FILE, --param_replacement_file PARAM_REPLACEMENT_FILE
    Parameter replacement file path. (.json)

  -cgdc, --check_gpu_delegate_compatibility
    Run TFLite ModelAnalyzer on the generated Float16 tflite model
    to check if the model can be supported by GPU Delegate.
    e.g.
    """
    === TFLite ModelAnalyzer ===

    Your TFLite model has '1' subgraph(s). In the subgraph description below,
    T# represents the Tensor numbers. For example, in Subgraph#0, the RESHAPE op takes
    tensor #0 and tensor #6 as input and produces tensor #7 as output.

    Subgraph#0 main(T#0) -> [T#17]
      Op#0 RESHAPE(T#0, T#6[2, 8, 8, 3, 2, ...]) -> [T#7]
      Op#1 SPLIT(T#5[0], T#7) -> [T#8, T#9]
      Op#2 RESHAPE(T#8, T#1[8, 8, 3, 2, 2]) -> [T#10]
      Op#3 TRANSPOSE(T#10, T#4[0, 3, 1, 4, 2]) -> [T#11]
      Op#4 RESHAPE(T#11, T#2[1, 8, 2, 8, 2, ...]) -> [T#12]
      Op#5 RESHAPE(T#9, T#1[8, 8, 3, 2, 2]) -> [T#13]
      Op#6 TRANSPOSE(T#13, T#4[0, 3, 1, 4, 2]) -> [T#14]
      Op#7 RESHAPE(T#14, T#2[1, 8, 2, 8, 2, ...]) -> [T#15]
      Op#8 CONCATENATION(T#12, T#15) -> [T#16]
      Op#9 RESHAPE(T#16, T#3[2, 16, 16, 3]) -> [T#17]

    Tensors of Subgraph#0
      T#0(inputs_0) shape:[2, 8, 8, 12], type:FLOAT32
      T#1(model/tf.compat.v1.squeeze_2/Squeeze) shape:[5], type:INT32 RO 20 bytes, data:[8, 8, 3, 2, 2]
      T#2(model/tf.expand_dims_1/ExpandDims) shape:[6], type:INT32 RO 24 bytes, data:[1, 8, 2, 8, 2, ...]
      T#3(model/tf.reshape_1/Reshape/shape) shape:[4], type:INT32 RO 16 bytes, data:[2, 16, 16, 3]
      T#4(model/tf.compat.v1.transpose/transpose/perm) shape:[5], type:INT32 RO 20 bytes, data:[0, 3, 1, 4, 2]
      T#5(model/tf.concat/concat/axis) shape:[], type:INT32 RO 4 bytes, data:[0]
      T#6(model/tf.reshape/Reshape/shape) shape:[6], type:INT32 RO 24 bytes, data:[2, 8, 8, 3, 2, ...]
      T#7(model/tf.reshape/Reshape) shape:[2, 8, 8, 3, 2, 2], type:FLOAT32
      T#8(model/tf.split/split) shape:[1, 8, 8, 3, 2, 2], type:FLOAT32
      T#9(model/tf.split/split1) shape:[1, 8, 8, 3, 2, 2], type:FLOAT32
      T#10(model/tf.compat.v1.squeeze_1/Squeeze) shape:[8, 8, 3, 2, 2], type:FLOAT32
      T#11(model/tf.compat.v1.transpose/transpose) shape:[8, 2, 8, 2, 3], type:FLOAT32
      T#12(model/tf.expand_dims/ExpandDims) shape:[1, 8, 2, 8, 2, 3], type:FLOAT32
      T#13(model/tf.compat.v1.squeeze_2/Squeeze1) shape:[8, 8, 3, 2, 2], type:FLOAT32
      T#14(model/tf.compat.v1.transpose_1/transpose) shape:[8, 2, 8, 2, 3], type:FLOAT32
      T#15(model/tf.expand_dims_1/ExpandDims1) shape:[1, 8, 2, 8, 2, 3], type:FLOAT32
      T#16(model/tf.concat/concat) shape:[2, 8, 2, 8, 2, 3], type:FLOAT32
      T#17(Identity) shape:[2, 16, 16, 3], type:FLOAT32

    Your model looks compatibile with GPU delegate with TFLite runtime version 2.10.0.
    But it doesn't guarantee that your model works well with GPU delegate.
    There could be some runtime incompatibililty happen.
    ---------------------------------------------------------------
                  Model size:       2988 bytes
        Non-data buffer size:       2757 bytes (92.27 %)
      Total data buffer size:        231 bytes (07.73 %)
        (Zero value buffers):          4 bytes (00.13 %)

    * Buffers of TFLite model are mostly used for constant tensors.
      And zero value buffers are buffers filled with zeros.
      Non-data buffers area are used to store operators, subgraphs and etc.
      You can find more details from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/schema/schema.fbs
    """

  -coto, --check_onnx_tf_outputs_elementwise_close
    Returns "Matches" if the output of onnx and the output of TF are
    within acceptable proximity element by element.
    Returns "Unmatched" if the output of onnx and the output of TF are
    not within acceptable proximity element by element.
    If the output of onnx is 1D, it returns "Skipped" and skips the comparison
    between the output of onnx and that of TF. This is because when undefined
    dimensions are present, a situation often arises where very large index
    values are compared, causing OutOfMemory.
    Only the output content of the models final output OP is checked.

  -cotof, --check_onnx_tf_outputs_elementwise_close_full
    Returns "Matches" if the output of onnx and the output of TF are
    within acceptable proximity element by element.
    Check the output of all OPs in sequence from the beginning,
    including all but the final output OP of the model.
    Returns "Unmatched" if the output of onnx and the output of TF are
    not within acceptable proximity element by element.
    If the output of onnx is 1D, it returns "Skipped" and skips the comparison
    between the output of onnx and that of TF. This is because when undefined
    dimensions are present, a situation often arises where very large index
    values are compared, causing OutOfMemory.
    It is very time consuming because it performs as many inferences as
    there are operations.

  -coton, --check_onnx_tf_outputs_sample_data_normalization
    norm: Validate using random data normalized to the range 0.0 to 1.0
    denorm: Validate using random data in the range 0.0 to 255.0
    If there is a normalization layer at the model's entry point, or
    if the model was trained on denormalized data, "denorm" must be specified.
    Default: "norm"

  -cotor CHECK_ONNX_TF_OUTPUTS_ELEMENTWISE_CLOSE_RTOL,\
    --check_onnx_tf_outputs_elementwise_close_rtol CHECK_ONNX_TF_OUTPUTS_ELEMENTWISE_CLOSE_RTOL
    The relative tolerance parameter.
    Default: 0.0

  -cotoa CHECK_ONNX_TF_OUTPUTS_ELEMENTWISE_CLOSE_ATOL,\
    --check_onnx_tf_outputs_elementwise_close_atol CHECK_ONNX_TF_OUTPUTS_ELEMENTWISE_CLOSE_ATOL
    The absolute tolerance parameter.
    Default: 1e-4

  -dms, --disable_model_save
    Does not save the converted model. For CIs RAM savings.

  -n, --non_verbose
    Shorthand to specify a verbosity of "error".

  -v, --verbosity
    Change the level of information printed.
    Values are "debug", "info", "warn", and "error".
    Default: "debug" (for backwards compatability)In-script Usage>>>fromonnx2tfimportconvert>>>help(convert)Helponfunctionconvertinmoduleonnx2tf:convert(input_onnx_file_path:Union[str,NoneType]='',onnx_graph:Union[onnx.onnx_ml_pb2.ModelProto,NoneType]=None,output_folder_path:Union[str,NoneType]='saved_model',output_signaturedefs:Optional[bool]=False,output_h5:Optional[bool]=False,output_keras_v3:Optional[bool]=False,output_tfv1_pb:Optional[bool]=False,output_weights:Optional[bool]=False,copy_onnx_input_output_names_to_tflite:Optional[bool]=False,output_integer_quantized_tflite:Optional[bool]=False,quant_type:Optional[str]='per-channel',custom_input_op_name_np_data_path:Optional[List]=None,input_output_quant_dtype:Optional[str]='int8',not_use_onnxsim:Optional[bool]=False,not_use_opname_auto_generate:Optional[bool]=False,batch_size:Union[int,NoneType]=None,overwrite_input_shape:Union[List[str],NoneType]=None,no_large_tensor:Optional[bool]=False,output_nms_with_dynamic_tensor:Optional[bool]=False,keep_ncw_or_nchw_or_ncdhw_input_names:Union[List[str],NoneType]=None,keep_nwc_or_nhwc_or_ndhwc_input_names:Union[List[str],NoneType]=None,keep_shape_absolutely_input_names:Optional[List[str]]=None,output_names_to_interrupt_model_conversion:Union[List[str],NoneType]=None,disable_group_convolution:Union[bool,NoneType]=False,enable_batchmatmul_unfold:Optional[bool]=False,enable_rnn_unroll:Optional[bool]=False,disable_suppression_flextranspose:Optional[bool]=False,number_of_dimensions_after_flextranspose_compression:Optional[int]=6,disable_suppression_flexstridedslice:Optional[bool]=False,disable_strict_mode:Optional[bool]=False,number_of_dimensions_after_flexstridedslice_compression:Optional[int]=5,optimization_for_gpu_delegate:Optional[bool]=False,replace_argmax_to_reducemax_and_indicies_is_int64:Union[bool,NoneType]=False,replace_argmax_to_reducemax_and_indicies_is_float32:Union[bool,NoneType]=False,replace_argmax_to_fused_argmax_and_indicies_is_int64:Union[bool,NoneType]=False,replace_argmax_to_fused_argmax_and_indicies_is_float32:Union[bool,NoneType]=False,fused_argmax_scale_ratio:Union[float,NoneType]=0.5,replace_to_pseudo_operators:List[str]=None,mvn_epsilon:Union[float,NoneType]=0.0000000001,param_replacement_file:Optional[str]='',check_gpu_delegate_compatibility:Optional[bool]=False,check_onnx_tf_outputs_elementwise_close:Optional[bool]=False,check_onnx_tf_outputs_elementwise_close_full:Optional[bool]=False,check_onnx_tf_outputs_sample_data_normalization:Optional[str]='norm',check_onnx_tf_outputs_elementwise_close_rtol:Optional[float]=0.0,check_onnx_tf_outputs_elementwise_close_atol:Optional[float]=1e-4,disable_model_save:Union[bool,NoneType]=False,non_verbose:Union[bool,NoneType]=False,verbosity:Optional[str]='debug')->keras.engine.training.ModelConvertONNXtoTensorFlowmodels.Parameters----------input_onnx_file_path:Optional[str]Inputonnxfilepath.Eitherinput_onnx_file_pathoronnx_graphmustbespecified.onnx_graph:Optional[onnx.ModelProto]onnx.ModelProto.Eitherinput_onnx_file_pathoronnx_graphmustbespecified.onnx_graphIfspecified,ignoreinput_onnx_file_pathandprocessonnx_graph.output_folder_path:Optional[str]Outputtensorflowmodelfolderpath.Default:"saved_model"output_signaturedefs:Optional[bool]Signatureisaddedtotheoutputforservingorforconversiontoothermodelformats.However,thiscansignificantlyreducethespeedofmodelconversionandsignificantincreasethesizeofthemodel.output_h5:Optional[bool]OutputmodelinKerasH5format.output_keras_v3:Optional[bool]OutputmodelinKeras(keras_v3)format.output_tfv1_pb:Optional[bool]OutputmodelinTFv1(.pb)format.output_weights:Optional[bool]Outputweightsinhdf5format.copy_onnx_input_output_names_to_tflite:Optional[bool]Copytheinput/outputOPnameofONNXtotheinput/outputOPnameoftflite.DuetoTensorflowinternaloperatingspecifications,theinput/outputorderofONNXdoesnotnecessarilymatchtheinput/outputorderoftflite.Besuretocheckthattheinput/outputOPnamesinthegeneratedtflitefilehavebeenconvertedasexpected.Also,thisoptiongeneratesahugeJSONfileasatemporaryfileforprocessing.Therefore,itisstronglydiscouragedtouseitonlargemodelsofhundredsofmegabytesormore.output_integer_quantized_tflite:Optional[bool]Outputofintegerquantizedtflite.quant_type:Optional[str]Selectswhether"per-channel"or"per-tensor"quantizationisused.Default:"per-channel"custom_input_op_name_np_data_path:Optional[List]--custom_input_op_name_np_data_pathINPUT_NAMENUMPY_FILE_PATHMEANSTDInputnameofOPandpathofdatafile(Numpy)forcustominputfor-cotofor-oiqt,andmean(optional)andstd(optional).<Usagein-cotof>Whenusing-cotof,custominputdefinedbytheuser,insteadofdummydata,isused.Inthiscase,meanandstdareomittedfromtheinput.-cind{input_op_name}{numpy_file_path}e.g.-cindonnx::Equal_0test_cind/x_1.npy-cindonnx::Add_1test_cind/x_2.npy-cotofTheinput_op_namemustbethesameasinONNX,anditmaynotworkiftheinputformatisdifferentbetweenONNXandTF.<Usagein-oiqt>INPUTNameofOPandpathofcalibrationdatafile(Numpy)forquantizationandmeanandstd.ThespecificationcanbeomittedonlywhentheinputOPisasingle4Dtensorimagedata.Ifomitted,itisautomaticallycalibratedusing20normalizedMS-COCOimages.ThetypeoftheinputOPmustbeFloat32.Dataforcalibrationmustbepre-normalizedtoarangeof0to1.-cind{input_op_name}{numpy_file_path}{mean}{std}NumpyfilepathsmustbespecifiedthesamenumberoftimesasthenumberofinputOPs.NormalizethevalueoftheinputOPbasedonthetensorspecifiedinmeanandstd.(input_value-mean)/stdTensorsinNumpyfileformatmustbeindimensionorderafterconversiontoTF.Notethatthisisintendedfordeploymentonlow-resourcedevices,sothebatchsizeislimitedto1only.e.g.TheexamplebelowshowsacasewheretherearethreeinputOPs.Assumeinput0is128x128RGBimagedata.Inaddition,input0shouldbeavaluethathasbeendividedby255inthepreprocessingandnormalizedtoarangebetween0and1.input1andinput2assumetheinputofsomethingthatisnotanimage.Becauseinput1andinput2assumesomethingthatisnotanimage,thedivisorisnot255whennormalizingfrom0to1."n"isthenumberofcalibrationdata.ONNXINPUTshapes:input0:[n,3,128,128]mean:[1,3,1,1]->[[[[0.485]],[[0.456]],[[0.406]]]]std:[1,3,1,1]->[[[[0.229]],[[0.224]],[[0.225]]]]input1:[n,64,64]mean:[1,64]->[[0.1,...,0.64]]std:[1,64]->[[0.05,...,0.08]]input2:[n,5]mean:[1]->[0.3]std:[1]->[0.07]TensorFlowINPUTshapes(Numpyfilendarrayshapes):input0:[n,128,128,3]mean:[1,1,1,3]->[[[[0.485,0.456,0.406]]]]std:[1,1,1,3]->[[[[0.229,0.224,0.225]]]]input1:[n,64,64]mean:[1,64]->[[0.1,...,0.64]]std:[1,64]->[[0.05,...,0.08]]input2:[n,5]mean:[1]->[0.3]std:[1]->[0.07]cind=[["input0","../input0.npy",[[[[0.485,0.456,0.406]]]],[[[[0.229,0.224,0.225]]]]],["input1","./input1.npy",[0.1,...,0.64],[0.05,...,0.08]],["input2","input2.npy",[0.3],[0.07]],]<Using-cotofand-oiqtatthesametime>Touse-cotofand-oiqtsimultaneously,youneedtoentertheInputnameofOP,pathofdatafile,mean,andstdalltogether.AndthedatafilemustbeinFloat32format,and{input_op_name},{numpy_file_path},{mean},and{std}mustallbeentered.Otherwise,anerrorwilloccurduringthe-oiqtstage.input_output_quant_dtype:Optional[str]InputandOutputdtypeswhendoingFullINT8Quantization."int8"(default)or"uint8"not_use_onnxsim:Optional[bool]Nooptimizationbyonnx-simplifierisperformed.Ifthisoptionisused,theprobabilityofaconversionerrorisveryhigh.not_use_opname_auto_generate:Optional[bool]AutomaticgenerationofeachOPnameintheoldformatONNXfileandassignmentofOPnamearenotperformed.batch_size:Optional[int]Fixesthedynamicbatchsizetothespecifiednumericbatchsize.Avalueof1ormoremustbespecified.overwrite_input_shape:Optional[List[str]]Overwritetheinputshape.Theformatis['i1:dim0,dim1,...,dimN','i2:dim0,dim1,...,dimN','i3:dim0,dim1,...,dimN']Whenthereisonlyoneinput,forexample,['data:1,3,224,224']Whentherearemultipleinputs,forexample,['data1:1,3,224,224','data2:1,3,112','data3:5']Avalueof1ormoremustbespecified.Numericalvaluesotherthandynamicdimensionsareignored.Ignoresbatch_sizeifspecifiedatthesametimeasbatch_size.no_large_tensor:Optional[bool]SuppressesconstantbloatcausedbyTileOPwhenoptimizingmodelsinonnxsim.See:https://github.com/daquexian/onnx-simplifier/issues/178output_nms_with_dynamic_tensor:Optional[bool]ThenumberofboundingboxesintheNMSoutputresultsisnotfixedatthemaximumnumberofmax_output_boxes_per_class,butratheratthesmallestpossiblenumberofdynamictensors.Ifthisoptionisdisabled,NMSoutputispaddedtothenumbersetinthemax_output_boxes_per_classattribute.e.g.disable--output_nms_with_dynamic_tensor:output_tensor_shape:[100,7]enable--output_nms_with_dynamic_tensor:output_tensor_shape:[N,7]keep_ncw_or_nchw_or_ncdhw_input_names:Optional[List[str]]HoldstheNCWorNCHWorNCDHWoftheinputshapeforthespecifiedINPUTOPnames.IfanonexistentINPUTOPnameisspecified,itisignored.Validonlyfor3D,4Dand5Dinputtensors.e.g.keep_ncw_or_nchw_or_ncdhw_input_names=['input0','input1','input2']keep_nwc_or_nhwc_or_ndhwc_input_names:Optional[List[str]]HoldstheNWCorNHWCorNDHWCoftheinputshapeforthespecifiedINPUTOPnames.IfanonexistentINPUTOPnameisspecified,itisignored.IftheinputOPnameisthesameastheinputOPnamespecifiedinthekeep_ncw_or_nchw_or_ncdhw_input_namesoption,itisignored.Validonlyfor3D,4Dand5Dinputtensors.e.g.keep_nwc_or_nhwc_or_ndhwc_input_names=['input0','input1','input2']keep_shape_absolutely_input_names:Optional[List[str]]NameoftheINPUTthatunconditionallymaintainsitsshape.IfanonexistentINPUTOPnameisspecified,itisignored.e.g.keep_shape_absolutely_input_names=['input0','input1','input2']output_names_to_interrupt_model_conversion:Optional[List[str]]OutputnamesofONNXthatinterruptmodelconversion.Interruptsmodeltransformationatthespecifiedoutputnameandoutputsthemodelpartitionedintosubgraphs.e.g.output_names_to_interrupt_model_conversion=['output0','output1','output2']disable_group_convolution:Optional[bool]DisableGroupConvolutionandreplaceitwithSeparableConvolutionforoutputtosaved_modelformat.enable_accumulation_type_float16:Optional[bool]HintforXNNPackfp16inferenceonfloat16tflitemodel.XNNPACKfloat16inferenceoncertainARM64coresis2xfaster.Float16inferencedoublingondeviceswithARM64ARMv8.2orhigherinstructionset.https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/xnnpack/README.md#floating-point-ieee-fp16-operatorsenable_batchmatmul_unfold:Optional[bool]BatchMatMulisseparatedbatchbybatchtogenerateaprimitiveMatMul.enable_rnn_unroll:Optional[bool]InsteadofincreasinginferencespeedbyexpandingallsymbolicloopsoftheRNN(LSTM,GRU,RNN),RAMconsumptionwillincreasebecausealltensorsareexpandedandembeddedinthemodel.https://keras.io/api/layers/recurrent_layers/disable_suppression_flextranspose:Optional[bool]DisablesFlexTransposegenerationsuppression.number_of_dimensions_after_flextranspose_compression:Optional[int]NumberofTransposeOPdimensionsgeneratedafteravoidingFlexTransposegeneration.AlsosuppressthecreationoftheTransposeitselfbyspecifying2.Default:6disable_suppression_flexstridedslice:Optional[bool]DisablesFlexStridedSlicegenerationsuppression.disable_strict_mode:Optional[bool]Ifspecified,theconversionspeedisgreatlyacceleratedbecausethestrictaccuracycorrectionprocessisskipped,butthefrequencyoftranspositionerrorsincreasesandaccuracyerrorsaremorelikelytooccur.Strictmodeisenabledbydefault.Asof2023.05.07,thisisaworkinprogressandisanexperimentalfeature.Therefore,onlysomeOPsareconvertedinstrictmodeforaccuracycorrection.number_of_dimensions_after_flexstridedslice_compression:Optional[int]NumberofStridedSliceOPdimensionsgeneratedafteravoidingFlexStridedSlicegeneration.Default:5optimization_for_gpu_delegate:Optional[bool]Replaceoperationsthatdonotsupportgpudelegatewiththosethatdoasmuchaspossible.replace_argmax_to_reducemax_and_indicies_is_int64:Optional[bool]ReplaceArgMaxwithaReduceMax.Thereturnedindiciesareint64.Onlyoneofreplace_argmax_to_reducemax_and_indicies_is_int64andreplace_argmax_to_reducemax_and_indicies_is_float32andreplace_argmax_to_fused_argmax_and_indicies_is_int64andreplace_argmax_to_fused_argmax_and_indicies_is_float32canbespecified.Default:Falsereplace_argmax_to_reducemax_and_indicies_is_float32:Optional[bool]ReplaceArgMaxwithaReduceMax.Thereturnedindiciesarefloat32.Onlyoneofreplace_argmax_to_reducemax_and_indicies_is_int64andreplace_argmax_to_reducemax_and_indicies_is_float32andreplace_argmax_to_fused_argmax_and_indicies_is_int64andreplace_argmax_to_fused_argmax_and_indicies_is_float32canbespecified.Default:Falsereplace_argmax_to_fused_argmax_and_indicies_is_int64:Optional[bool]ReplaceArgMaxwithaReduceMax.Thereturnedindiciesareint64.Itimprovesinferencespeedatthecostofasmallsacrificeinaccuracy.See.https://github.com/tensorflow/models/tree/master/official/projects/edgetpu/vision#argmax-fusion-to-improve-segmentation-model-latencyCurrently,only4Dtensorsaresupported.Onlyoneofreplace_argmax_to_reducemax_and_indicies_is_int64andreplace_argmax_to_reducemax_and_indicies_is_float32andreplace_argmax_to_fused_argmax_and_indicies_is_int64andreplace_argmax_to_fused_argmax_and_indicies_is_float32canbespecified.Default:Falsereplace_argmax_to_fused_argmax_and_indicies_is_float32:Optional[bool]ReplaceArgMaxwithaReduceMax.Thereturnedindiciesarefloat32.Itimprovesinferencespeedatthecostofasmallsacrificeinaccuracy.See.https://github.com/tensorflow/models/tree/master/official/projects/edgetpu/vision#argmax-fusion-to-improve-segmentation-model-latencyCurrently,only4Dtensorsaresupported.Onlyoneofreplace_argmax_to_reducemax_and_indicies_is_int64andreplace_argmax_to_reducemax_and_indicies_is_float32andreplace_argmax_to_fused_argmax_and_indicies_is_int64andreplace_argmax_to_fused_argmax_and_indicies_is_float32canbespecified.Default:Falsefused_argmax_scale_ratio:Optional[float]ForFusedArgMax.ScaleratiowhengeneratingFusedArgMax.0.0<fused_argmax_scale_ratio<=1.0Default:0.5replace_to_pseudo_operators:List[str]Replacelistofoperatorstopseudooperators.Fullnameofthetargetoperatorsshouldbegiven.Currentlysupportedoperators:Asin,Acos,Atan,Abs,PReLU,LeakyReLU,Power,GatherND,Neg,HardSwish,Erf,GeLUmvn_epsilon:Optional[float]ForMeanVarianceNormalization.Thenumbertobeaddedtothevariancetoavoiddivisionbyzerowhennormalizingthevalue.(input_tensor-mean)/tf.sqrt(variance+mvn_epsilon)Default:0.0000000001param_replacement_file:Optional[str]Parameterreplacementfilepath.(.json)check_gpu_delegate_compatibility:Optional[bool]RunTFLiteModelAnalyzeronthegeneratedFloat16tflitemodeltocheckifthemodelcanbesupportedbyGPUDelegate.e.g."""=== TFLite ModelAnalyzer ===Your TFLite model has '1' subgraph(s). In the subgraph description below,T# represents the Tensor numbers. For example, in Subgraph#0, the RESHAPE op takestensor #0 and tensor #6 as input and produces tensor #7 as output.Subgraph#0 main(T#0) -> [T#17]Op#0 RESHAPE(T#0, T#6[2, 8, 8, 3, 2, ...]) -> [T#7]Op#1 SPLIT(T#5[0], T#7) -> [T#8, T#9]Op#2 RESHAPE(T#8, T#1[8, 8, 3, 2, 2]) -> [T#10]Op#3 TRANSPOSE(T#10, T#4[0, 3, 1, 4, 2]) -> [T#11]Op#4 RESHAPE(T#11, T#2[1, 8, 2, 8, 2, ...]) -> [T#12]Op#5 RESHAPE(T#9, T#1[8, 8, 3, 2, 2]) -> [T#13]Op#6 TRANSPOSE(T#13, T#4[0, 3, 1, 4, 2]) -> [T#14]Op#7 RESHAPE(T#14, T#2[1, 8, 2, 8, 2, ...]) -> [T#15]Op#8 CONCATENATION(T#12, T#15) -> [T#16]Op#9 RESHAPE(T#16, T#3[2, 16, 16, 3]) -> [T#17]Tensors of Subgraph#0T#0(inputs_0) shape:[2, 8, 8, 12], type:FLOAT32T#1(model/tf.compat.v1.squeeze_2/Squeeze) shape:[5], type:INT32 RO 20 bytes, data:[8, 8, 3, 2, 2]T#2(model/tf.expand_dims_1/ExpandDims) shape:[6], type:INT32 RO 24 bytes, data:[1, 8, 2, 8, 2, ...]T#3(model/tf.reshape_1/Reshape/shape) shape:[4], type:INT32 RO 16 bytes, data:[2, 16, 16, 3]T#4(model/tf.compat.v1.transpose/transpose/perm) shape:[5], type:INT32 RO 20 bytes, data:[0, 3, 1, 4, 2]T#5(model/tf.concat/concat/axis) shape:[], type:INT32 RO 4 bytes, data:[0]T#6(model/tf.reshape/Reshape/shape) shape:[6], type:INT32 RO 24 bytes, data:[2, 8, 8, 3, 2, ...]T#7(model/tf.reshape/Reshape) shape:[2, 8, 8, 3, 2, 2], type:FLOAT32T#8(model/tf.split/split) shape:[1, 8, 8, 3, 2, 2], type:FLOAT32T#9(model/tf.split/split1) shape:[1, 8, 8, 3, 2, 2], type:FLOAT32T#10(model/tf.compat.v1.squeeze_1/Squeeze) shape:[8, 8, 3, 2, 2], type:FLOAT32T#11(model/tf.compat.v1.transpose/transpose) shape:[8, 2, 8, 2, 3], type:FLOAT32T#12(model/tf.expand_dims/ExpandDims) shape:[1, 8, 2, 8, 2, 3], type:FLOAT32T#13(model/tf.compat.v1.squeeze_2/Squeeze1) shape:[8, 8, 3, 2, 2], type:FLOAT32T#14(model/tf.compat.v1.transpose_1/transpose) shape:[8, 2, 8, 2, 3], type:FLOAT32T#15(model/tf.expand_dims_1/ExpandDims1) shape:[1, 8, 2, 8, 2, 3], type:FLOAT32T#16(model/tf.concat/concat) shape:[2, 8, 2, 8, 2, 3], type:FLOAT32T#17(Identity) shape:[2, 16, 16, 3], type:FLOAT32Your model looks compatibile with GPU delegate with TFLite runtime version 2.10.0.But it doesn't guarantee that your model works well with GPU delegate.There could be some runtime incompatibililty happen.---------------------------------------------------------------Model size:       2988 bytesNon-data buffer size:       2757 bytes (92.27 %)Total data buffer size:        231 bytes (07.73 %)(Zero value buffers):          4 bytes (00.13 %)* Buffers of TFLite model are mostly used for constant tensors.And zero value buffers are buffers filled with zeros.Non-data buffers area are used to store operators, subgraphs and etc.You can find more details from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/schema/schema.fbs"""check_onnx_tf_outputs_elementwise_close:Optional[bool]Returns"Matches"iftheoutputofonnxandtheoutputofTFarewithinacceptableproximityelementbyelement.Returns"Unmatched"iftheoutputofonnxandtheoutputofTFarenotwithinacceptableproximityelementbyelement.Iftheoutputofonnxis1D,itreturns"Skipped"andskipsthecomparisonbetweentheoutputofonnxandthatofTF.Thisisbecausewhenundefineddimensionsarepresent,asituationoftenariseswhereverylargeindexvaluesarecompared,causingOutOfMemory.OnlytheoutputcontentofthemodelsfinaloutputOPischecked.check_onnx_tf_outputs_elementwise_close_full:Optional[bool]Returns"Matches"iftheoutputofonnxandtheoutputofTFarewithinacceptableproximityelementbyelement.ChecktheoutputofallOPsinsequencefromthebeginning,includingallbutthefinaloutputOPofthemodel.Returns"Unmatched"iftheoutputofonnxandtheoutputofTFarenotwithinacceptableproximityelementbyelement.Iftheoutputofonnxis1D,itreturns"Skipped"andskipsthecomparisonbetweentheoutputofonnxandthatofTF.Thisisbecausewhenundefineddimensionsarepresent,asituationoftenariseswhereverylargeindexvaluesarecompared,causingOutOfMemory.Itisverytimeconsumingbecauseitperformsasmanyinferencesasthereareoperations.check_onnx_tf_outputs_sample_data_normalization:Optional[str]norm:Validateusingrandomdatanormalizedtotherange0.0to1.0denorm:Validateusingrandomdataintherange0.0to255.0Ifthereisanormalizationlayeratthemodelsentrypoint,orifthemodelwastrainedondenormalizeddata,"denorm"mustbespecified.Default:"norm"check_onnx_tf_outputs_elementwise_close_rtol:Optional[float]Therelativetoleranceparameter.Default:0.0check_onnx_tf_outputs_elementwise_close_atol:Optional[float]Theabsolutetoleranceparameter.Default:1e-4disable_model_save:Optional[bool]Doesnotsavetheconvertedmodel.ForCIsRAMsavings.Default:Falsenon_verbose:Optional[bool]Shorthandtospecifyaverbosityof"error".Default:Falseverbosity:Optional[str]Changethelevelofinformationprinted.Valuesare"debug","info","warn",and"error".Default:"debug"(forbackwardscompatability)Returns----------model:tf.keras.ModelModelParameter replacementThis tool is used to convertNCWtoNWC,NCHWtoNHWC,NCDHWtoNDHWC,NCDDHWtoNDDHWC,NCDDDDDDHWtoNDDDDDDHWC. Therefore, as stated in the Key Concepts, the conversion will inevitably break down at some point in the model. You need to look at the entire conversion log to see which OP transpositions are failing and correct them yourself. I dare to explain very little because I know that no matter how much detail I put in the README, you guys will not read it at all.attributeorINPUT constantorINPUT Initializercan be replaced with the specified value.Starting fromv1.3.0, almost all OPs except for some special OPs support pre- and post-transposition bypre_process_transposeandpost_process_transpose."A conversion error occurs.""Output results are wrong."Do not submit an issue that only contains an amount of information that cannot be reproduced.convert option--param_replacement_file param_replacement.json

or

-prf param_replacement.jsonparam_replacement.jsonSee a sample of replacement JSON{"format_version":1,"operations":[{"op_name":"StatefulPartitionedCall/Tile_4","param_target":"inputs",# attributes or inputs"param_name":"const_fold_opt__677","values":[1,1,17]# Disable parameter transposition or overwrite parameters},{"op_name":"StatefulPartitionedCall/Cast_3","param_target":"attributes",# attributes or inputs"param_name":"to","values":1# Disable parameter transposition or overwrite "to" parameters},{"op_name":"Resize__697","param_target":"inputs","param_name":"Concat__696:0","values":[26,26]# Replacement of unk__x (Resize OP, sizes height/width parameter)},{"op_name":"Transpose__927","param_target":"attributes","param_name":"perm","values":[0,1,2,3]# Disable parameter transposition or overwrite "perm" parameters},{"op_name":"StatefulPartitionedCall/functional_1/max_unpooling2d_2/Reshape_1","param_target":"inputs","param_name":"const_fold_opt__911","values":[4,131072]# Overwrite "shape" parameters},{"op_name":"Reshape_25","param_target":"outputs","param_name":"onnx::InstanceNormalization_270","post_process_transpose_perm":[0,2,1]# Extrapolate 3D Transpose after Reshape},{"op_name":"Reshape_30","param_target":"outputs","param_name":"onnx::Mul_275","post_process_transpose_perm":[0,2,3,1]# Extrapolate 4D Transpose after Reshape},{"op_name":"flatten_1127","param_target":"inputs","param_name":"dropout0","pre_process_transpose_perm":[0,3,1,2]},{"op_name":"/Slice","param_target":"op","begin":[0,0,1,0],"end":[0,0,0,0],"end_mask":15},{"op_name":"/Slice_1","param_target":"op","begin":[0,0,0,0],"end":[0,0,39,0],"end_mask":11},{"op_name":"/backbone/backbone.1/Unsqueeze_1","param_target":"op","new_shape":[1,15,15,1]}]}Replacement Supported OPsSee list of replacement specificationsNo.OP typeRemarks1Add1. "param_target": "inputs"pre_process_transpose_perm: Transpose is applied to the tensor before the Add operation with the perm specified as pre-processing.2. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Add operation with the perm specified as post-processing.2CastTypeValuesTypeValuesfloat1610int83float321int165float6411int326bool9int647uint82uint164uint3212uint64133Concat1. "param_target": "attributes"axis: Value ofaxis2. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Concat operation with the perm specified as post-processing.4ConvTransposeConvTransposeimplements special replacements separately ignore all automatic conversions and generatetf.nn.conv1d_transposeortf.nn.conv2d_transposeortf.nn.conv3d_transposedirectly by specifying all parameters.https://www.tensorflow.org/api_docs/python/tf/nn/conv1d_transposehttps://www.tensorflow.org/api_docs/python/tf/nn/conv2d_transposehttps://www.tensorflow.org/api_docs/python/tf/nn/conv3d_transpose1. "param_target": "op"output_shape: Value ofoutput_shapestrides: Value ofstridespadding: Value ofpaddingdilations: Value ofdilations5Div1. "param_target": "inputs"values: Value ofinputpre_process_transpose_perm: Transpose is applied to the tensor before the Div operation with the perm specified as pre-processing.2. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Div operation with the perm specified as post-processing.6Expand1. "param_target": "inputs"values: Value ofshapepre_process_transpose_perm: Transpose is applied to the tensor before the Expand operation with the perm specified as pre-processing.2. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Expand operation with the perm specified as post-processing.7Flatten1. "param_target": "attributes"axis: Value ofaxis2. "param_target": "inputs"pre_process_transpose_perm: Transpose is applied to the tensor before the Flatten operation with the perm specified as pre-processing.3. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Flatten operation with the perm specified as post-processing.8Gemm9Gather1. "param_target": "attributes"axis: Value ofaxis2. "param_target": "inputs"values: Value ofindicespre_process_transpose_perm: Transpose is applied to the tensor before the Gather operation with the perm specified as pre-processing.3. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Gather operation with the perm specified as post-processing.10MatMul1. "param_target": "inputs"pre_process_transpose_perm: Transpose is applied to the tensor before the MatMul operation with the perm specified as pre-processing.2. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the MatMul operation with the perm specified as post-processing.11Mul1. "param_target": "inputs"values: Value ofinputpre_process_transpose_perm: Transpose is applied to the tensor before the Mul operation with the perm specified as pre-processing.2. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Mul operation with the perm specified as post-processing.12NonMaxSuppression13ReduceL1ReduceL2ReduceLogSumReduceLogSumExpReduceMaxReduceMeanReduceMinReduceProdReduceSumReduceSumSquare1. "param_target": "attributes"axes: Value ofaxeskeepdims: Value ofkeepdims2. "param_target": "inputs"pre_process_transpose_perm: Transpose is applied to the tensor before the ReduceXX operation with the perm specified as pre-processing.3. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the ReduceXX operation with the perm specified as post-processing.14Unsqueeze1. "param_target": "inputs"pre_process_transpose_perm: Transpose is applied to the tensor before the Unsqueeze operation with the perm specified as pre-processing.2. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Unsqueeze operation with the perm specified as post-processing.3. "param_target": "op"new_shape: Specifies directly the shape after Unsqueeze processing.{"op_name": "/backbone/backbone.1/Unsqueeze_1","param_target": "op","new_shape": [1,15,15,1]}15Reshape1. "param_target": "inputs"values: Value ofshapepre_process_transpose_perm: Transpose is applied to the tensor before the Reshape operation with the perm specified as pre-processing.2. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Reshape operation with the perm specified as post-processing.16Resize1. "param_target": "attributes"coordinate_transformation_mode: Value ofcoordinate_transformation_modeextrapolation_value: Value ofextrapolation_valuemode: Value ofmode2. "param_target": "inputs"values: Value ofroiorscalesorsizes.scales=[scale_h,scale_w],sizes=[h,w]pre_process_transpose_perm: Transpose is applied to the tensor before the Resize operation with the perm specified as pre-processing.3. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Resize operation with the perm specified as post-processing.17SliceSliceimplements special replacements separately ignore all automatic conversions and generatetf.strided_slicedirectly by specifying all parameters oftf.strided_slicedirectly.https://www.tensorflow.org/api_docs/python/tf/strided_sliceSeereplace_slice.jsonfor a sample description.1. "param_target": "op"begin: Value ofbeginend: Value ofendstrides: Value ofstridesbegin_mask: Value ofbegin_maskend_mask: Value ofend_maskellipsis_mask: Value ofellipsis_masknew_axis_mask: Value ofnew_axis_maskshrink_axis_mask: Value ofshrink_axis_mask{"op_name": "/Slice","param_target": "op","begin": [0,0,1,0],"end": [0,0,0,0],"end_mask": 15}18Softmax1. "param_target": "attributes"axis: Value ofaxis. The transpositions corresponding to the specified axis are extrapolated before and afterSoftmax.2. "param_target": "inputs"values: Value oftensor19Split1. "param_target": "inputs"values: Value ofsplit2. "param_target": "attributes"axis: Value ofaxis.num_outputs: Value ofnum_outputs.20Sub1. "param_target": "inputs"values: Value ofinputpre_process_transpose_perm: Transpose is applied to the tensor before the Sub operation with the perm specified as pre-processing.2. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Sub operation with the perm specified as post-processing.21Tile1. "param_target": "inputs"values: Value ofinputpre_process_transpose_perm: Transpose is applied to the tensor before the Tile operation with the perm specified as pre-processing.2. "param_target": "outputs"post_process_transpose_perm: Transpose is applied to the tensor after the Tile operation with the perm specified as post-processing.22Transpose1. "param_target": "attributes"perm: Value ofperm2. "param_target": "inputs"values: Value oftensorGenerated ModelYOLOv7-tiny with Post-Process (NMS) ONNX to TFLite Float32https://github.com/PINTO0309/onnx2tf/releases/download/0.0.33/yolov7_tiny_head_0.768_post_480x640.onnxSee the structure of the modelonnx2tfonnx-tensorflow(Super redundant + Broken)YOLACT-Edge MobileNetV2 with Post-Process (MultiClass-NMS) ONNX to TFLite Float32https://github.com/PINTO0309/onnx2tf/releases/download/1.0.11/yolact_edge_mobilenetv2_550x550.onnxSee the structure of the modelMoveNet MultiPose ONNX to TFLite Float32 (CastandTrueDivstandard OP support)https://github.com/PINTO0309/onnx2tf/releases/download/1.0.24/movenet_multipose_lightning_192x256_p6.onnxSee the structure of the modelValidated models (without replacement.json)ONNX file for testing.https://github.com/PINTO0309/onnx2tf/releases/tag/1.1.28See a list of verified modelsNo.ModelPass1age_googlenet.onnx:heavy_check_mark:2alike_t_opset11_192x320.onnx:heavy_check_mark:3arcfaceresnet100-8.onnx:heavy_check_mark:4baseline_simplified.onnx:heavy_check_mark:5big_slice_11.onnx:heavy_check_mark:6bvlcalexnet-12.onnx:heavy_check_mark:7caffenet-12.onnx:heavy_check_mark:8convtranspose_3_1_5_2.onnx:heavy_check_mark:9convtranspose_4_5_2_2.onnx:heavy_check_mark:10convtranspose_5_5_6_1.onnx:heavy_check_mark:11convtranspose_6_5_5_8.onnx:heavy_check_mark:12convtranspose_7_1_3_4.onnx:heavy_check_mark:13damoyolo_tinynasL20_T_192x192_post.onnx:heavy_check_mark:14deeplabv3_mobilenet_v3_large.onnx:heavy_check_mark:15densenet-12.onnx:heavy_check_mark:16depth_to_spase_17.onnx:heavy_check_mark:17double_gru.onnx:heavy_check_mark:18digits.onnx:heavy_check_mark:19detr_demo.onnx:heavy_check_mark:20efficientformer_l1.onnx:heavy_check_mark:21efficientdet_lite2_detection_1.onnx:heavy_check_mark:22efficientnet-lite4-11_nchw.onnx:heavy_check_mark:23effnet_opset11_dynamic_axis.onnx:heavy_check_mark:24emotion-ferplus-8_rename.onnx:heavy_check_mark:25face_detection_yunet_2022mar.onnx:heavy_check_mark:26face_recognition_sface_2021dec-act_int8-wt_int8-quantized.onnx:heavy_check_mark:27face_recognition_sface_2021dec.onnx:heavy_check_mark:28faster_rcnn-10.onnx:heavy_check_mark:29fastestdet.onnx:heavy_check_mark:30fused_conv_clip.onnx:heavy_check_mark:31fused_conv_hardsigmoid.onnx:heavy_check_mark:32fused_conv_leakyrelu.onnx:heavy_check_mark:33fused_conv_relu.onnx:heavy_check_mark:34fused_conv_sigmoid.onnx:heavy_check_mark:35fused_conv_tanh.onnx:heavy_check_mark:36gender_googlenet.onnx:heavy_check_mark:37gmflow-scale1-mixdata-train320x576-4c3a6e9a_1x3x480x640_bidir_flow_sim.onnx:heavy_check_mark:38handpose_estimation_mediapipe_2022may.onnx:heavy_check_mark:39htnet_1x17x2_without_norm.onnx:heavy_check_mark:40iat_llie_180x320.onnx:heavy_check_mark:41if_p1_11.onnx:heavy_check_mark:42if_p2_11.onnx:heavy_check_mark:43if_p3_11.onnx:heavy_check_mark:44imageclassifier.onnx:heavy_check_mark:45inception-v2-9.onnx:heavy_check_mark:46inverse11.onnx:heavy_check_mark:47mhformer_NxFxKxXY_1x27x17x2.onnx:heavy_check_mark:48mnist.onnx:heavy_check_mark:49mnist-12.onnx:heavy_check_mark:50mobilenetv2-12.onnx:heavy_check_mark:51mosaic_11.onnx:heavy_check_mark:52mosaic-9.onnx:heavy_check_mark:53movenet_multipose_lightning_192x256_p6.onnx:heavy_check_mark:54nanodet-plus-m_416.onnx:heavy_check_mark:55object_tracking_dasiamrpn_kernel_cls1_2021nov.onnx:heavy_check_mark:56object_tracking_dasiamrpn_kernel_r1_2021nov.onnx:heavy_check_mark:57object_tracking_dasiamrpn_model_2021nov.onnx:heavy_check_mark:58pidnet_S_cityscapes_192x320.onnx:heavy_check_mark:59ppmattingv2_stdc1_human_480x640.onnx:heavy_check_mark:60qlinear_conv_tensor_test.onnx:heavy_check_mark:61rcnn-ilsvrc13-9.onnx:heavy_check_mark:62regnet_x_400mf.onnx:heavy_check_mark:63ResNet101-DUC-12.onnx:heavy_check_mark:64resnet18-v1-7.onnx:heavy_check_mark:65resnet50-v1-12.onnx:heavy_check_mark:66resnet50-v2-7.onnx:heavy_check_mark:67retinanet-9.onnx:heavy_check_mark:68sinet_320_op.onnx:heavy_check_mark:69squeezenet1.0-12.onnx:heavy_check_mark:70super-resolution-10.onnx:heavy_check_mark:71swinir-m_64x64_12.onnx:heavy_check_mark:72text_recognition_CRNN_EN_2021sep.onnx:heavy_check_mark:73tinyyolov2-8.onnx:heavy_check_mark:74version-RFB-640.onnx:heavy_check_mark:75vit-b-32_textual.onnx:heavy_check_mark:76vit-b-32_visual.onnx:heavy_check_mark:77yolact_edge_mobilenetv2_550x550.onnx:heavy_check_mark:78yolact_regnetx_600mf_d2s_31classes_512x512.onnx:heavy_check_mark:79yolact_regnetx_800mf_20classes_512x512.onnx:heavy_check_mark:80yolo_free_nano_crowdhuman_192x320_post.onnx:heavy_check_mark:81yolov7_tiny_head_0.768_post_480x640.onnx:heavy_check_mark:82yolox_nano_192x192.onnx:heavy_check_mark:83yolox_nano_416x416.onnx:heavy_check_mark:84yolox_s.onnx:heavy_check_mark:85yolox_x_crowdhuman_mot17_bytetrack.onnx:heavy_check_mark:86zero_dce_640_dele.onnx:heavy_check_mark:87zfnet512-12.onnx:heavy_check_mark:Key conceptList of Key conceptonnx-tensorflowis a very useful tool, but the performance of the generated TensorFlow models is significantly degraded due to the extrapolation of a large number ofTransposeOPs before and after each OP during the format conversion fromNCHWtoNHWC. Therefore, I will make this tool myself as a derivative tool ofonnx-tensorflowwithout extrapolatingTranspose.Most of the internal processing of the tool is full-scratch, but some of the more complex OPs have been adapted fromonnx-tensorflow. I am very grateful to the engineers at International Business Machines Corporation / LeapMind / Microsoft / IBM for developingonnx-tensorflow.I have incorporated all my knowledge of model optimization to other models such as TFLite, EdgeTPU, TensorFlow.js and Myriad based on my years of experience implementingopenvino2tensorflowandtflite2tensorflow. It probably has the best model optimization performance and conversion efficiency of any tool I have created in the past, and the lowest rate of conversion errors.Supported layers list.Supported layersIf you are having trouble with conversion errors, searching forresolved or open issueswill almost always solve your problems. Issues are knowledge for engineers around the world.Contributors to this repository should first readContribution Guide.https://user-images.githubusercontent.com/33194443/197319770-e7ef7174-66cd-4bc2-84be-59e1a251151d.mp4All OPs are decomposed into primitive operations as much as possible. This is beneficial for lateral deployment of models to frameworks other than TFLite. Therefore, OPs belonging totf.keras.layersare almost never used, and the tool consists only oftf.xxx. (except for a very few OPs)As I do not want to add more dependent packages, I do not usetensorflow_addons (tfa), but replace it with the standard OP of tensorflow.Not only does it handle conversions of 4-dimensional inputs, such asNCHWtoNHWC, but also the number of input dimensions in 3, 5, or even more dimensions. For example,NCDHWtoNDHWC, etc. However, since 1-D, 2-D, 3-D and 6-D input may produce patterns that are mechanically difficult to convert, it should be possible to give parameters to externally modify the tool's behavior. SeeParameter replacementIf there are undefined dimensions in the input OP, the model structure is not fully optimized and conversion errors are very likely to occur.Immediately following aReshapeOP with dimensional compression and dimensional decompression, there is a 95% probability that the model transformation operation will be disrupted and errors will occur. For example, patterns such as[1,200,200,5]->[1,200,-1]or[10,20,30,40,50]->[10,2,10,30,10,4,50]orFlatten. See#8 Not able to reshape input in replace.json, or#15 Conv layer shape wrong, or#18 Question about channel_transpose in common_functions.py, or#105 [MobileFormer]Converted model outputs values mismatch with original ones., or#133 When Onnx Matmul inputs have different dimension.TensorFlow's Convolution does not have an equivalent operation to ONNX's Padding operation. Therefore, aPadOP is inserted immediately before a Convolution with Padding of size greater than 1.Support conversion to TensorFlow saved model and TFLite (Float32/Float16/INT8).Files exceeding the Protocol Buffers file size limit of 2GB are not supported. Therefore, the external format is not supported at the initial stage of tool creation.If there are ONNX OPs that are not supported by TensorFlow, usesimple-onnx-processing-toolsto replace them with harmless OPs in advance and then use this tool to convert them. In other words, you can convert any model with your efforts.ONNX splitting, merging, generating OPs, rewriting OP attributes, BGR<->RGB conversion, converting to JSON and editing in the IDE, batch size changes for undefined dimensions, and various other processing can be done with thesimple-onnx-processing-tools. Therefore, it is recommended that models with very complex structures be converted to TFLite after modifying the structure beforehand.BatchNormalizationsupports only inference mode.LayerNormalizationsupports only inference mode.Only foropset=11or higherIf you do not like the generated TFLite OP name, edit it usingtflite2json2tflite.The generated Keras models cannot be used for retraining. If you want to train, you must build your own model.When converting to TensorFlow.js, CoreML, etc., please generate saved_model with the--output_signaturedefsoption and use the generated saved_model to convert with various converters.tensorflowjs_converter,coremltools,edgetpu_compilier, etc... If this option is not enabled, saved_model records only the minimum necessary information and its size is minimized. When this option is enabled, saved_model records the maximum amount of information, and instead of being maximized in size, the output is in a format that supports conversion to other frameworks. It can also be used for serving.There are many OPs on ONNX that do not support TFLite/EdgeTPU/TFJS/CoreML/TensorRT. Therefore, if you need to generate an EdgeTPU model, please specify--replace_to_pseudo_operatorsto convert your model. onnx2tf will attempt to replace the OP with an TFLite/EdgeTPU/TFJS/CoreML/TensorRT-compatible OP whenever possible.The main factors that cause accuracy degradation after model conversion are as followsdifferences in Padding specificationsdifference in Python division specification in the process of model transformation (error due to even rounding)Divide epsilon without considerationdeprecated TrueDivisionsupport difference of powersdifferences in interpolation operation specifications during resizingDifference in arithmetic precision supported by each operationCalculation error due to scaling up or down by specifying ascalewhen resizing imagesThe above differences often cannot be dealt with by simply converting the model in a straightforward manner. Therefore, you need to replace the model yourself in advance with an operation that is less prone to errors.Support forINT8 Quantization,Full INT8 Quantization,INT8 Quantization with INT16 activation,Full INT8 Quantization with INT16 activationandDynamic Range Quantization.Support forPer-Channel QuantizationandPer-Tensor Quantization.Support forGroupConvolution.TFLite does not supportTrueDiv(INT), soTrueDivis avoided if possible.Implement theResizeprocess for the 5D tensor.Add process to replaceAsinwithpseudo-Asin.Add process to replaceAcoswithpseudo-Acos.Add process to replaceAtanwithpseudo-Atan.Add process to replaceAbswithpseudo-Abs.Add process to replaceGatherNDwithpseudo-GatherND.Add process to replaceHardSwishwithpseudo-HardSwish.Add process to replaceGridSamplewithpseudo-GridSample.Add process to replacePReluwithpseudo-PRelu.Add process to replaceLeakyReluwithpseudo-LeakyRelu.Add process to replacePowerwithpseudo-Power.Add process to replaceNegwithpseudo-Neg.Add process to replaceArgMaxwithpseudo-ArgMax.Add process to replaceErfwithpseudo-Erf.Add process to replaceGeLUwithpseudo-GeLU.Added option to fix dynamic batch sizeNto a specified number.Added option to overwrite dynamic shape input OPs with static shape.--overwrite_input_shapeOutput in Keras H5 format.Automatically runonnx-simplifier(onnxsim) backend and optimize onnx files before model transformation.Added the ability to automatically generate each OP name and assign OP names to ONNX files in the old format.Supports model splitting. Interrupts model transformation at the specified output name and outputs the model partitioned into subgraphs.Related toolstflite2tensorflowopenvino2tensorflowtflite2json2tflitetensorflowjs_convertercoremltoolssimple-onnx-processing-toolstflite-input-output-rewriteronnx-simplifieronnx_graphsurgeononnxonnx-tensorflowonnx2kerasTinyNeuralNetworknobucoonnx2torchAcknowledgementhttps://github.com/onnx/modelshttps://github.com/opencv/opencv_zoohttps://pytorch.org/vision/stable/models.htmlhttps://tfhub.dev/https://www.kaggle.com/modelshttps://github.com/TexasInstruments/edgeai-modelzooContributorsMade withcontrib.rocks.