Package: nn-meter
nn-Meteris a novel and efficient system to accurately predict the inference latency of DNN models on diverse edge devices. The key idea is dividing a whole model inference into kernels, i.e., the execution units of fused operators on a device, and conduct kernel-level prediction. We currently evaluate four popular platforms on a large dataset of 26k models. It achieves 99.0% (mobile CPU), 99.1% (mobile Adreno 640 GPU), 99.0% (mobile Adreno 630 GPU), and 83.4% (Intel VPU) prediction accuracy.The current supported hardware and inference frameworks:DeviceFrameworkProcessor+-10%  AccuracyHardware namePixel4TFLite v2.1CortexA76 CPU99.0%cortexA76cpu_tflite21Mi9TFLite v2.1Adreno 640 GPU99.1%adreno640gpu_tflite21Pixel3XLTFLite v2.1Adreno 630 GPU99.0%adreno630gpu_tflite21Intel Movidius NCS2OpenVINO2019R2Myriad VPU83.4%myriadvpu_openvino2019r2nn-Meter has achieved theMobisys 21 Best Paper Award!For more details, please check out paper:nn-Meter: towards accurate latency prediction of deep-learning model inference on diverse edge devicesWho should consider using nn-MeterThose who want to get the DNN inference latency on mobile and edge devices withno deployment efforts on real devices.Those who want to runhardware-aware NAS withNNI.Those who want tobuild latency predictors for their own devices(Documentsof nn-Meter builder).Those who want to use the 26k latencybenchmark dataset.InstallationCurrently nn-Meter has been tested on Linux and Windows system. Windows 10, Ubuntu 16.04 and 20.04 with python 3.6.10 are tested and supported. Please first installpython3before nn-Meter installation. Then nn-Meter could be installed by running:pipinstallnn-meterIf you want to try latest code, please install nn-Meter from source code. First git clone nn-Meter package to local:gitclonegit@github.com:microsoft/nn-Meter.gitcdnn-MeterThen simply run the following pip install in an environment that haspython >= 3.6. The command will complete the automatic installation of all necessary dependencies and nn-Meter.pipinstall.nn-Meter is a latency predictor of models with type of Tensorflow, PyTorch, Onnx, nn-meter IR graph andNNI IR graph. To use nn-Meter for specific model type, you also need to install corresponding required packages. The well tested versions are listed below:Testing Model TypeRequirementsTensorflowtensorflow==2.6.0Torchtorch==1.9.0,torchvision==0.10.0, (alternative)[onnx==1.9.0,onnx-simplifier==0.3.6] or [nni>=2.4][1]Onnxonnx==1.9.0nn-Meter IR graph---NNI IR graphnni>=2.4[1] Please refer tonn-Meter Usagefor more information.Please also check the versions ofnumpyandscikit_learn. The different versions may change the prediction accuracy of kernel predictors.The stable version of wheel binary package will be released soon.UsageTo apply for hardware latency prediction, nn-Meter provides two types of interfaces：command linenn-meterafternn-meterinstallation.Python binding provided by the modulenn_meterHere is a summary of supported inputs of the two methods.Testing Model TypeCommand SupportPython BindingTensorflowCheckpoint file dumped bytf.saved_model()and end with.pbCheckpoint file dumped bytf.saved_modeland end with.pbTorchModels intorchvision.modelsObject oftorch.nn.ModuleOnnxCheckpoint file dumped bytorch.onnx.export()oronnx.save()and end with.onnxCheckpoint file dumped byonnx.save()or model loaded byonnx.load()nn-Meter IR graphJson file in the format ofnn-Meter IR Graphdictobject following the format ofnn-Meter IR GraphNNI IR graph-NNI IR graph objectIn both methods, users could appoint predictor name and version to target a specific hardware platform (device). Currently, nn-Meter supports prediction on the following four configs:Predictor (device_inferenceframework)Processor CategoryVersioncortexA76cpu_tflite21CPU1.0adreno640gpu_tflite21GPU1.0adreno630gpu_tflite21GPU1.0myriadvpu_openvino2019r2VPU1.0Users can get all predefined predictors and versions by running# to list all predefined predictorsnn-meter--list-predictorsPredict latency of saved CNN modelAfter installation, a command namednn-meteris enabled. To predict the latency for a CNN model with a predefined predictor in command line, users can run the following commands (sample models can be downloadedhere)# for Tensorflow (*.pb) filenn-meterpredict--predictor<hardware>[--predictor-version<version>]--tensorflow<pb-file_or_folder># Example Usagenn-meterpredict--predictorcortexA76cpu_tflite21--predictor-version1.0--tensorflowmobilenetv3small_0.pb# for ONNX (*.onnx) filenn-meterpredict--predictor<hardware>[--predictor-version<version>]--onnx<onnx-file_or_folder>#Example Usagenn-meterpredict--predictorcortexA76cpu_tflite21--predictor-version1.0--onnxmobilenetv3small_0.onnx# for torch model from torchvision model zoo (str)nn-meterpredict--predictor<hardware>[--predictor-version<version>]--torchvision<model-name><model-name>...#Example Usagenn-meterpredict--predictorcortexA76cpu_tflite21--predictor-version1.0--torchvisionmobilenet_v2# for nn-Meter IR (*.json) filenn-meterpredict--predictor<hardware>[--predictor-version<version>]--nn-meter-ir<json-file_or_folder>#Example Usagenn-meterpredict--predictorcortexA76cpu_tflite21--predictor-version1.0--nn-meter-irmobilenetv3small_0.json--predictor-version <version>arguments is optional. When the predictor version is not specified by users, nn-meter will use the latest version of the predictor.nn-Meter can support batch mode prediction. To predict latency for multiple models in the same model type once, user should collect all models in one folder and state the folder after--[model-type]liked argument.It should also be noted that for PyTorch model, nn-meter can only support existing models in torchvision model zoo. The string followed by--torchvisionshould be exactly one or more string indicating name(s) of some existing torchvision models. To apply latency prediction for torchvision model in command line,onnxandonnx-simplifierpackages are required.Convert to nn-Meter IR GraphFurthermore, users may be interested to convert tensorflow pb-file or onnx file to nn-Meter IR graph. Users could convert nn-Meter IR graph and save to.jsonfile be running# for Tensorflow (*.pb) filenn-meterget_ir--tensorflow<pb-file>[--output<output-name>]# for ONNX (*.onnx) filenn-meterget_ir--onnx<onnx-file>[--output<output-name>]Output name is default to be/path/to/input/file/<input_file_name>_<model-type>_ir.jsonif not specified by users.Use nn-Meter in your python codeAfter installation, users can import nn-Meter in python codefromnn_meterimportload_latency_predictorpredictor=load_latency_predictor(hardware_name,hardware_predictor_version)# case insensitive in backend# build your model (e.g., model instance of torch.nn.Module)model=...lat=predictor.predict(model,model_type)# the resulting latency is in unit of msBy callingload_latency_predictor, user selects the target hardware and loads the corresponding predictor. nn-Meter will try to find the right predictor file in~/.nn_meter/data. If the predictor file doesn't exist, it will download from the Github release.Inpredictor.predict(), the allowed items of the parametermodel_typeinclude["pb", "torch", "onnx", "nnmeter-ir", "nni-ir"], representing model types of tensorflow, torch, onnx, nn-meter IR graph and NNI IR graph, respectively.<span id="torch-model-converters">For Torch models, the shape of feature maps is unknown merely based on the given network structure, which is, however, significant parameters in latency prediction. Therefore, torch model requires a shape of input tensor for inference as a input ofpredictor.predict(). Based on the given input shape, a random tensor according to the shape will be generated and used. Another thing for Torch model prediction is that users can install theonnxandonnx-simplifierpackages for latency prediction (referred to as Onnx-based latency prediction for torch model), or alternatively install thennipackage (referred to as NNI-based latency prediction for torch model). Note that thennioption does not support command line calls. In addition, if users usennifor latency prediction, the PyTorch modules should be defined by thenninterface from NNIimport nni.retiarii.nn.pytorch as nn(viewNNI docfor more information), and the parameterapply_nnishould be set asTruein the functionpredictor.predict(). Here is an example of NNI-based latency prediction for Torch model:importnni.retiarii.nn.pytorchasnnfromnn_meterimportload_latency_predictorpredictor=load_latency_predictor(...)# build your model using nni.retiarii.nn.pytorch as nnmodel=nn.Module...input_shape=(1,3,224,224)lat=predictor.predict(model,model_type='torch',input_shape=input_shape,apply_nni=True)The Onnx-based latency prediction for torch model is stable but slower, while the NNI-based latency prediction for torch model is unstable as it could fail in some case but much faster compared to the Onnx-based model. The Onnx-based model is set as the default one for Torch model latency prediction in nn-Meter. Users could choose which one they preferred to use according to their needs.Users could view the information all built-in predictors bylist_latency_predictorsor view the config file innn_meter/configs/predictors.yaml.Users could get a nn-Meter IR graph by applyingmodel_file_to_graphandmodel_to_graphby calling the model name or model object and specify the model type. The supporting model types ofmodel_file_to_graphinclude "onnx", "pb", "torch", "nnmeter-ir" and "nni-ir", while the supporting model types ofmodel_to_graphinclude "onnx", "torch" and "nni-ir".nn-Meter Buildernn-Meter builder is an open source tool for users to build latency predictor on their own devices. There are three main parts in nn-Meter builder:backend: the module of connecting backends;backend_meta: the meta tools related to backend. Here we provide the fusion rule tester to detect fusion rules for users' backend;kernel_predictor_builder: the tool to build different kernel latency predictors.Users could get access to nn-Meter builder by callingnn_meter.builder. For more details to use nn-Meter builder, please check the document ofnn-Meter builder.Hardware-aware NAS by nn-Meter and NNITo empower affordable DNN on the edge and mobile devices, hardware-aware NAS searches both high accuracy and low latency models. In particular, the search algorithm only considers the models within the target latency constraints during the search process.Currently we provide two examples of hardware-aware NAS, including end-to-endmulti-trial NASwhich is arandom search algorithmonSPOS NASsearch space, and the popularProxylessNAS, which is a one-shot NAS algorithm with hardware-efficient loss function. More examples of other widely-used hardware-aware NAS and model compression algorithms are coming soon.Multi-trial SPOS DemoTo run multi-trail SPOS demo, NNI should be installed through source code by followingNNI Docpythonsetup.pydevelopThen run multi-trail SPOS demo:python${NNI_ROOT}/examples/nas/oneshot/spos/multi_trial.pyHow the demo worksRefer toNNI Docfor how to perform NAS by NNI.To support hardware-aware NAS, you first need aStrategythat supports filtering the models by latency. We provide such a filter namedLatencyFilterin NNI and initialize aRandomstrategy with the filter:simple_strategy=strategy.Random(model_filter=LatencyFilter(threshold=100,predictor=base_predictor))LatencyFilterwill predict the models' latency by using nn-Meter and filter out the models whose latency with the given predictor are larger than the threshold (i.e.,100in this example).
You can also build your own strategies and filters to support more flexible NAS such as sorting the models according to latency.Then, pass this strategy toRetiariiExperiment:exp=RetiariiExperiment(base_model,trainer,strategy=simple_strategy)exp_config=RetiariiExeConfig('local')...exp_config.dummy_input=[1,3,32,32]exp.run(exp_config,port)Inexp_config,dummy_inputis required for tracing shape info.ProxylessNAS DemoTo run the one-shot ProxylessNAS demo, users can run the NNI ProxylessNAS training demo:python${NNI_ROOT}/examples/nas/oneshot/proxylessnas/main.py--applied_hardware<hardware>--reference_latency<referencelatency(ms)>How the demo worksRefer toNNI Docfor how to perform NAS by NNI.ProxylessNAS currently builds a lookup table, that stores the measured latency of each candidate building block in the search space. The latency sum of all building blocks in a candidate model will be treated as the model inference latency. With leveraging nn-Meter in NNI, users can apply ProxylessNAS to search efficient DNN models on more types of edge devices. In NNI implementation, aHardwareLatencyEstimatorpredicts expected latency for the mixed operation based on the path weight ofProxylessLayerChoice. To call nn-Meter in NNI ProxylessNAS, users can add the arguments of "--applied_hardware <hardware> --reference_latency <reference latency (ms)>" in theexample.Benchmark DatasetTo evaluate the effectiveness of a prediction model on an arbitrary DNN model, we need a representative dataset that covers a large prediction scope. nn-Meter collects and generates 26k CNN models. (Please refer the paper for the dataset generation method.)We release the dataset, and provide an interface ofnn_meter.datasetfor users to get access to the dataset. Users can also download the data from theDownload Linkon their own.ContributingThis project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visithttps://cla.opensource.microsoft.com.When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor
contactopencode@microsoft.comwith any additional questions or comments.LicenseThe entire codebase is underMIT licenseThe dataset is underOpen Use of Data AgreementCitationIf you find that nn-Meter helps your research, please consider citing it:@inproceedings{nnmeter,
    author = {Zhang, Li Lyna and Han, Shihao and Wei, Jianyu and Zheng, Ningxin and Cao, Ting and Yang, Yuqing and Liu, Yunxin},
    title = {nn-Meter: Towards Accurate Latency Prediction of Deep-Learning Model Inference on Diverse Edge Devices},
    year = {2021},
    publisher = {ACM},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3458864.3467882},
    doi = {10.1145/3458864.3467882},
    booktitle = {Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services},
    pages = {81–93},
}

@misc{nnmetercode,
    author = {Microsoft Research nn-Meter Team},
    title = {nn-Meter: Towards Accurate Latency Prediction of Deep-Learning Model Inference on Diverse Edge Devices},
    year = {2021},
    url = {https://github.com/microsoft/nn-Meter},
}