Package: tpu-mlir
TPU-MLIRFor Chinese version:README.TPU-MLIR is an open-source machine-learning compiler based on MLIR for TPU. This project provides a complete toolchain, which can convert pre-trained neural networks from different frameworks into binary filesbmodelthat can be efficiently operated on TPUs.SOPHGO aims to become a leading global provider of general-purpose computing power. SOPHGO focuses on the research, development, and promotion of computing products such as Deep Learning and RISC-V processors, and has built a comprehensive application matrix covering the 'cloud, edge, and endpoint' scenarios with its self-developed products. SOPHGO provides computing products and integrated solutions for applications such as smart cities, intelligent computing centers, smart security, intelligent transportation, safety production, industrial quality inspection, and intelligent terminals. The company has research and development centers in more than 10 cities in China including Beijing, Shanghai, Shenzhen, Qingdao, and Xiamen, as well as in the United States and Singapore.Currently, supported Deep Learning frameworks are PyTorch, ONNX, TFLite and Caffe. Models from other frameworks need to be converted to ONNX models.ResourcesHere are some resources to help you better understand the project:IndexDocuments01TPU-MLIR paper02TPU-MLIR Technical Reference Manual03TPU-MLIR Quick StartIndexSharing Sessions01TPU-MLIR Paper02LayerGroupIndexTopicVideo Links01What is Depp Learning Compiler?Depp Learning Compiler Intro02MLIR IntroBasic Syntax (1),Basic Syntax (2),Basic Syntax (3),Dialect Conversion,Pattern Rewriting03TPU-MLIR IntroOverview,Front-end Conversion,Lowering04QuantizationOverview,Formula Derivation,Calibration,QAT05TPU MemoryEp1,Ep206TPU-MLIR PracticeTo Onnx Format,Graph Optimization,Operator Support,Model Support,Fuse Preprocess,Accuracy ValidationIn addition, we also published a series of tasks for any of you interested in our project and would like to develop it with us:IndexTasks01Rewrite Patterns for PermuteOp02Shape Inference Implement03mlir2onnx tool OptimizeFor More tasks please checkIssue.If you have any questions while doing the tasks above, you can ask or check the existing answers in ourQ&A Platform.How to BuildAfter cloning the code of this project, it needs to be compiled in docker.Download the required image fromdockerhub.dockerpullsophgo/tpuc_dev:latest# myname1234 is just an example, you can set your own namedockerrun--privileged--namemyname1234-v$PWD:/workspace-itsophgo/tpuc_dev:latestAfter the container is created, the directory of the code in docker should be/workspace/tpu-mlir.BuildingRun the following command in the project directory:cdtpu-mlirsource./envsetup.sh
./build.shUsageIntroduce the usage ofTPU-MLIRby a simple example of compilingyolov5s.onnxand running it on the BM1684X TPU platform.The model comes from the official website of yolov5:https://github.com/ultralytics/yolov5/releases/download/v6.0/yolov5s.onnx.It has been placed in project pathregression/model/yolov5s.onnx.PreparationFirstly, create amodel_yolov5sdirectory at the same level directory with this project. Then put both model and image files into it.The operation is as follows:mkdirmodel_yolov5s&&cdmodel_yolov5s
cp${REGRESSION_PATH}/model/yolov5s.onnx.
cp-rf${REGRESSION_PATH}/dataset/COCO2017.
cp-rf${REGRESSION_PATH}/image.
mkdirworkspace&&cdworkspaceModel to MLIRIf the model takes images as input, we need to learn its preprocessing before transforming. No preprocessing needs to be considered if the input is npz file. The preprocessing process is formulated as follows:$$
y = （x - mean） \times scale,
$$wherexrepresents the input.The input of the official yolov5 is RGB image. Each value will be multiplied by1/255. Mean and scale are0.0, 0.0, 0.0and0.0039216, 0.0039216, 0.0039216respectively.The model conversion command:model_transform.py\--model_nameyolov5s\--model_def../yolov5s.onnx\--input_shapes[[1,3,640,640]]\--mean0.0,0.0,0.0\--scale0.0039216,0.0039216,0.0039216\--keep_aspect_ratio\--pixel_formatrgb\--output_names350,498,646\--test_input../image/dog.jpg\--test_resultyolov5s_top_outputs.npz\--mliryolov5s.mlirThe arguments ofmodel_transform.py:ArgumentRequired？Descriptionmodel_nameYesModel namemodel_defYesModel definition file (.onnx,.pt,.tfliteor.prototxt)model_dataNoSpecify the model weight file, required when it is caffe model (corresponding to the '.caffemodel' file)input_shapesNoThe shape of the input, such as[[1,3,640,640]](a two-dimensional array), which can support multiple inputsresize_dimsNoThe size of the original image to be adjusted to. If not specified, it will be resized to the input size of the modelkeep_aspect_ratioNoWhether to maintain the aspect ratio when resize. False by default. It will pad 0 to the insufficient part when settingmeanNoThe mean of each channel of the image. The default is 0.0,0.0,0.0scaleNoThe scale of each channel of the image. The default is 1.0,1.0,1.0pixel_formatNoImage type, can be rgb, bgr, gray or rgbdoutput_namesNoThe names of the output. Use the output of the model if not specified, otherwise use the specified names as the outputtest_inputNoThe input file for validation, which can be an image, npy or npz. No validation will be carried out if it is not specifiedtest_resultNoOutput file to save validation resultexceptsNoNames of network layers that need to be excluded from validation. Separated by commadebugNoif open debug, immediate model file will keep; or will remove after conversion donemlirYesThe output mlir file name (including path)After converting to mlir file, a${model_name}_in_f32.npzfile containing preprocessed input will be generated.MLIR to F16 bmodelConvert the mlir file to the F16 bmodel by the following command:model_deploy.py\--mliryolov5s.mlir\--quantizeF16\--processorbm1684x\--test_inputyolov5s_in_f32.npz\--test_referenceyolov5s_top_outputs.npz\--modelyolov5s_1684x_f16.bmodelThe arguments ofmodel_deploy.py:ArgumentRequired？DescriptionmlirYesMlir filequantizeYesQuantization type (F32/F16/BF16/INT8)processorYesThe platform that the model will use. Currently only bm1684x is supported. More TPU platforms will be supported in the futurecalibration_tableNoThe quantization table path. Required when it is INT8 quantizationtoleranceNoTolerance for the minimum similarity between MLIR quantized and MLIR fp32 inference resultscorrectnetssNoTolerance for the minimum similarity between simulator and MLIR quantized inference results. 0.99,0.90 by defaultexceptsNoNames of network layers that need to be excluded from validation. Separated by commadebugNoif open debug, immediate model file will keep; or will remove after conversion donemodelYesName of output model file (including path)dynamicNodynamic codegen for to support dynamic shapeMLIR to INT8 bmodelBefore converting to the INT8 model, you need to run calibration to get the calibration table. The number of input data is about 100 to 1000 according to the situation.Then use the calibration table to generate a symmetric int8 bmodel. It is generally not recommended to use the asymmetric one if the symmetric one already meets the requirements, because
the performance of the asymmetric model will be slightly worse than the symmetric model.Here is an example of the existing 100 images from COCO2017 to perform calibration:run_calibration.pyyolov5s.mlir\--dataset../COCO2017\--input_num100\-oyolov5s_cali_tableExecute the following command to convert to the INT8 symmetric quantized model:model_deploy.py\--mliryolov5s.mlir\--quantizeINT8\--calibration_tableyolov5s_cali_table\--processorbm1684x\--test_inputyolov5s_in_f32.npz\--test_referenceyolov5s_top_outputs.npz\--tolerance0.85,0.45\--modelyolov5s_1684x_int8.bmodelResults ComparisonThis project has a yolov5 sample written in python (path:python/samples/detect_yolov5.py) for object detection. Read the code to learn how the model is used:preprocess the inputmodel inference to get outputpost-process the outputThe following code is used to verify the output of onnx/f32/int8 model respectively:ONNX model:detect_yolov5.py\--input../image/dog.jpg\--model../yolov5s.onnx\--outputdog_origin.jpgF16 bmodel:detect_yolov5.py\--input../image/dog.jpg\--modelyolov5s_1684x_f16.bmodel\--outputdog_f16.jpgINT8symmetric quantizedbmodel:detect_yolov5.py\--input../image/dog.jpg\--modelyolov5s_1684x_int8.bmodel\--outputdog_int8.jpgOutputs of different models are compared below:Auxiliary ToolsModel Inference Toolmodel_runner.pySupports bmodel/mlir/pytorch/onnx/tflite/caffe.model_runner.py\--inputresnet18_in_f32.npz\--modelresnet18_1684x_f32.bmodel\--outputresnet18_output.npzTool forbmodelThebmodelfile can be viewed and edited bymodel_tool:model_tool
    --info model_file : show brief model info
    --print model_file : show detailed model info
    --extract model_file : extract one multi-net bmodel to multi one-net bmodels
    --combine file1 .. fileN -o new_file: combine bmodels to one bmodel by filepath
    --combine_dir dir1 .. dirN -o new_dir: combine bmodels to one bmodel by directory path
    --dump model_file start_offset byte_size out_file: dump binary data to file from bmodelFor example, to get basic information ofbmodel:model_tool--inforesnet18_1684x_f32.bmodelRelated LinksOfficial NetworkDiscourseVideos