Package: mnn-meter
Tools for quickly building operator
latency tables and for accurately predicting
model latency (based on Pytorch and MNN)中文版1.InstallationMMT is used in both server-side and inference-side situations:on the server side, the operator list is generated according to the specified operator space;
the delay of a given model is predicted according to the operator delay table.On the inference side, test the operator delay according to the operator list to obtain the operator latency table.The server side must installPytorchandMNN(C++)at the same time,
and the inference side must installMNN(C++)Note: Be sure to add thebuildfolder generated by compilingMNNto the environment variable!After configuring the above dependencies, install MMTpip install mmn-meter2.Start2.1 Modify your modelsFor your custom model(layer), please overriderepr() with unique representation of the parameters, for example:def__init__(self,...)self.name="ResNetBasicBlock-%d-%d-%d-%d-"%(in_channels,out_channels,stride,kernel)...def__repr__(self):returnself.nameIf the results returned by__repr__()cannot be differentiated for the same type of operator input with different parameters,
it is very easy to cause running errors or measurement errors!See how to modify your model2.2 Export the operatorsAfter themmt=2.xversion, bothdescription file generationandfunction generationare supported.2.2.1 Method 1: Write an operator description fileThe parameters that determine the specific delay of
an operator include (operator type, operator
instantiation parameters, input shape).
The specific operator space
needs to be expressed in the following way:resnet18:ResNetBasicBlock:in_channels:[64,128,256,512]out_channels:[64,128,256,512]stride:[1]kernel:[3,5,7]input_shape:[[1,64,112,112],[1,128,56,56],[1,256,28,28],[1,512,14,14]]torch.nn:Conv2d:in_channels:[3]out_channels:[64]kernel_size:[7]stride:[2]padding:[3]input_shape:[[1,3,224,224]]BatchNorm2d:num_features:[64]input_shape:[[1,64,112,112]]ReLU:no_params:trueinput_shape:[[1,64,112,112]]Refer to how to describe your operatorThen use the following command to create a list of operators and export the operators to mnn format.frommmt.converterimportgenerate_ops_listgenerate_ops_list("ops.yaml","/path/ops_folder")ops.yamlis the operator description file,pathops_folderis the directory where
the operator is saved, and the correspondingmeta.pklwill be generated in this directory
to save the metadata information of the operator.2.2.1 Method 2: Functional GenerationHighly similar to Mode 1, it is directly registered and generated by using themmt.registerfunction, and supports multiple registration of operators of the same type to reduce redundant operators caused by unnecessary combinations (the disadvantage of Mode 1),
forexamplefrommmt.converterimportregisterimporttorch.nnasnnfp="./mbv3_ops"reg=lambdaops,**kwargs:register(ops,fp,**kwargs)reg(nn.Linear,in_features=[576,1024],out_features=[1024,1000],bias=[True],input_shape=[[1,576],[1,1024]],)Method 1:torch.nn:Linear:in_features:[576,1024]out_features:[1024,1000]bias:[True]input_shape:[[1,576],[1,1024]]The corresponding operator can be generated by directly running the written file. For more details,
please refer toExamplemmtsupports training a machine learning model with a
small number of model real and predicted delays as training
samples for more accurate predictors.Here we use the functionexport_modelsto save model and convert it to the
`mnn' format. You could create models with different configs, and save these models
in one folder.frommmt.converterimportexport_modelsforiinrange(16):cfg_=generate_cfg(cfgs)# generate confignet=MobileNetV3(cfg_,id=i,mode="small")# create a modelexport_models(net,[1,3,224,224],"mbv3")# save this model at "./mbv3" and convert it to .mnn2.3 Record operator delays on the deployment side, and build an operator latency tablefrommmt.meterimportmeter_opsmeter_ops("./ops",times=100)opsis the folder where the operator andmeta.pklare saved,timesrepresents the number of repeated tests,
run the modified program, the delay of the operator
will be calculated, and the operator latency table will be
saved as.ops/meta_latency.pkl. This file
specifically records the metadata and corresponding
latency of all operators.Similarly, you can directly test the model in the folder and record the delay.frommmt.meterimportmeter_modelsmeter_models("mbv3")2.4 Predicting model latency on the server sidefrommmt.parserimportpredict_latency...model=ResNet18()pred_latency=predict_latency(model,path,[1,3,224,224],verbose=False)pathis the path corresponding tometa_latency.pkl.
Note that the shape of the input tensor must be
the same as theinput_shapeset in the operator
description.Build more accurate predictors with machine learning models：frommmt.converterimportvalidationfrommmt.predictorimportlatency_predictorvalidation("mbv3","mbv3_ops/meta_latency.pkl",save_path="train_error.csv")lp=latency_predictor("mbv3_ops","train_error.csv")pred_latency=lp(model,path,[1,3,224,224])3 Test the prediction error of MMTSpecific referenceMobileNetV3 testModelNumerr(%)deviceMobileNet3344.1%*(39%)40  Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz