Package: mlia
ML Inference Advisor - IntroductionThe ML Inference Advisor (MLIA) helps AI developers design and optimize
neural network models for efficient inference on Arm® targets (seesupported targets). MLIA provides
insights on how the ML model will perform on Arm early in the model
development cycle. By passing a model file and specifying an Arm hardware target,
users get an overview of possible areas of improvement and actionable advice.
The advice can cover operator compatibility, performance analysis and model
optimization (e.g. pruning and clustering). With the ML Inference Advisor,
we aim to make the Arm ML IP accessible to developers at all levels of abstraction,
with differing knowledge on hardware optimization and machine learning.Inclusive language commitmentThis product conforms to Arm's inclusive language policy and, to the best of
our knowledge, does not contain any non-inclusive language.If you find something that concerns you, emailterms@arm.com.ReleasesRelease notes can be found inMLIA releases.Getting supportIn case you need support or want to report an issue, give us feedback or
simply ask a question about MLIA, please send an email tomlia@arm.com.Alternatively, use theAI and ML forumto get support by marking your post with theMLIAtag.Reporting vulnerabilitiesInformation on reporting security issues can be found inReporting vulnerabilities.LicenseML Inference Advisor is licensed underApache License 2.0.Trademarks and copyrightsArm®, Arm® Ethos™-U, Arm® Cortex®-A, Arm® Cortex®-M, Arm® Corstone™ are
registered trademarks or trademarks of Arm® Limited (or its subsidiaries) in
the U.S. and/or elsewhere.TensorFlow™ is a trademark of Google® LLC.Keras™ is a trademark by François Chollet.Linux® is the registered trademark of Linus Torvalds in the U.S. and
elsewhere.Python® is a registered trademark of the PSF.Ubuntu® is a registered trademark of Canonical.Microsoft and Windows are trademarks of the Microsoft group of companies.General usagePrerequisites and dependenciesIt is recommended to use a virtual environment for MLIA installation, and a
typical setup requires:Ubuntu® 20.04.03 LTS (other OSs may work, the ML Inference Advisor has been
tested on this one specifically)Python® >= 3.8.1Ethos™-U Vela dependencies (Linux® only)For more details, please refer to theprerequisites of VelaInstallationMLIA can be installed withpipusing the following command:pipinstallmliaIt is highly recommended to create a new virtual environment for the installation.First stepsAfter the installation, you can check that MLIA is installed correctly by
opening your terminal, activating the virtual environment and typing the
following command that should print the help text:mlia--helpThe ML Inference Advisor works with sub-commands, i.e. in general a command
would look like this:mlia[sub-command][arguments]Where the following sub-commands are available:"check": perform compatibility or performance checks on the model"optimize": apply specified optimizationsDetailed help about the different sub-commands can be shown like this:mlia[sub-command]--helpThe following sections go into further detail regarding the usage of MLIA.Sub-commandsThis section gives an overview of the available sub-commands for MLIA.checkcompatibilityLists the model's operators with information about their compatibility with
the specified target.Examples:# List operator compatibility with Ethos-U55 with 256 MACmliacheck~/models/mobilenet_v1_1.0_224_quant.tflite--target-profileethos-u55-256# List operator compatibility with Cortex-Amliacheck~/models/mobilenet_v1_1.0_224_quant.tflite--target-profilecortex-a# Get help and further informationmliacheck--helpperformanceEstimates the model's performance on the specified target and prints out
statistics.Examples:# Use default parametersmliacheck~/models/mobilenet_v1_1.0_224_quant.tflite\--target-profileethos-u55-256\--performance# Explicitly specify the target profile and backend(s) to use# with --backend optionmliacheck~/models/ds_cnn_large_fully_quantized_int8.tflite\--target-profileethos-u65-512\--performance\--backend"vela"\--backend"corstone-300"# Get help and further informationmliacheck--helpoptimizeThis sub-command applies optimizations to a Keras model (.h5 or SavedModel) or
a TensorFlow Lite model and shows the performance improvements compared to
the original unoptimized model.There are currently three optimization techniques available to apply:pruning: Sets insignificant model weights to zero until the specified
sparsity is reached.clustering: Groups the weights into the specified number of clusters and
then replaces the weight values with the cluster centroids.More information about these techniques can be found online in the TensorFlow
documentation, e.g. in theTensorFlow model optimization guides.rewrite: Replaces certain subgraph/layer of the pre-trained model with
candidates from the rewrite library, with or without training using a
small portion of the training data, to achieve local performance gains.Note:AKeras model(.h5 or SavedModel) is required as input to
perform pruning and clustering. ATensorFlow Lite modelis required as input
to perform a rewrite.Examples:# Custom optimization parameters: pruning=0.6, clustering=16mliaoptimize~/models/ds_cnn_l.h5\--target-profileethos-u55-256\--pruning\--pruning-target0.6\--clustering\--clustering-target16# Get help and further informationmliaoptimize--help# An example for using rewritemliaoptimize~/models/ds_cnn_large_fp32.tflite\--target-profileethos-u55-256\--rewrite\--datasetinput.tfrec\--rewrite-targetfully_connected\--rewrite-startMobileNet/avg_pool/AvgPool\--rewrite-endMobileNet/fc1/BiasAddTarget profilesThe targets currently supported are described in the sections below.
All sub-commands require a target profile as input parameter.
That target profile can be either a name of a built-in target profile
or a custom file. MLIA saves the target profile that was used for a run
in the output directory.The support of the above sub-commands for different targets is provided via
backends that need to be installed separately, seeBackend installationsection.Ethos-UThere are a number of predefined profiles for Ethos-U with the following
attributes:+--------------------------------------------------------------------+
| Profile name  | MAC | System config               | Memory mode    |
+=====================================================================
| ethos-u55-256 | 256 | Ethos_U55_High_End_Embedded | Shared_Sram    |
+---------------------------------------------------------------------
| ethos-u55-128 | 128 | Ethos_U55_High_End_Embedded | Shared_Sram    |
+---------------------------------------------------------------------
| ethos-u65-512 | 512 | Ethos_U65_High_End          | Dedicated_Sram |
+---------------------------------------------------------------------
| ethos-u65-256 | 256 | Ethos_U65_High_End          | Dedicated_Sram |
+--------------------------------------------------------------------+Example:mliacheck~/model.tflite--target-profileethos-u65-512--performanceEthos-U is supported by these backends:Corstone-300Corstone-310VelaCortex-AThe profilecortex-acan be used to get the information about supported
operators for Cortex-A CPUs when using the Arm NN TensorFlow Lite Delegate.
Please, find more details in the section for thecorresponding backend.TOSAThe target profiletosacan be used for TOSA compatibility checks of your
model. It requires theTOSA Checkerbackend.For more information, see TOSA Checker's:repositorypypi.org pageCustom target profilesFor thecustom target profiles, the configuration file for a custom
target profile is passed as path and needs to conform to the TOML file format.
Each target in MLIA has a pre-defined set of parameters which need to be present
in the config file. The built-in target profiles (insrc/mlia/resources/target_profiles)
can be used to understand what parameters apply for each target.Example:# for custom profilesmliaops--target-profile~/my_custom_profile.tomlsample_model.tfliteBackend installationThe ML Inference Advisor is designed to use backends to provide different
metrics for different target hardware. Some backends come pre-installed,
but others can be added and managed using the commandmlia-backend, that
provides the following functionality:installuninstalllistExamples:# List backends installed and available for installationmlia-backendlist# Install Corstone-300 backend for Ethos-Umlia-backendinstallCorstone-300--path~/FVP_Corstone_SSE-300/# Uninstall the Corstone-300 backendmlia-backenduninstallCorstone-300# Get help and further informationmlia-backend--helpNote:Some, but not all, backends can be automatically downloaded, if no
path is provided.Available backendsThis section lists available backends. As not all backends work on any platform
the following table shows some compatibility information:+----------------------------------------------------------------------------+
| Backend       | Linux                  | Windows        | Python           |
+=============================================================================
| Arm NN        |                        |                |                  |
| TensorFlow    | x86_64                 | Windows 10     | Python>=3.8      |
| Lite Delegate |                        |                |                  |
+-----------------------------------------------------------------------------
| Corstone-300  | x86_64                 | Not compatible | Python>=3.8      |
+-----------------------------------------------------------------------------
| Corstone-310  | x86_64                 | Not compatible | Python>=3.8      |
+-----------------------------------------------------------------------------
| TOSA checker  | x86_64 (manylinux2014) | Not compatible | 3.7<=Python<=3.9 |
+-----------------------------------------------------------------------------
| Vela          | x86_64                 | Windows 10     | Python~=3.7      |
+----------------------------------------------------------------------------+Arm NN TensorFlow Lite DelegateThis backend provides general information about the compatibility of operators
with the Arm NN TensorFlow Lite Delegate for Cortex-A. It comes pre-installed.For version 23.05 the classic delegate is used.For more information see:Arm NN TensorFlow Lite Delegate documentationCorstone-300Corstone-300 is a backend that provides performance metrics for systems based
on Cortex-M55 and Ethos-U. It is only available on the Linux platform.Examples:# Download and install Corstone-300 automaticallymlia-backendinstallCorstone-300# Point to a local version of Corstone-300 installed using its installation scriptmlia-backendinstallCorstone-300--pathYOUR_LOCAL_PATH_TO_CORSTONE_300For further information about Corstone-300 please refer to:https://developer.arm.com/Processors/Corstone-300Corstone-310Corstone-310 is a backend that provides performance metrics for systems based
on Cortex-M85 and Ethos-U. It is available as Arm Virtual Hardware (AVH) only,
i.e. it can not be downloaded automatically.For access to AVH for Corstone-310 please refer to:https://developer.arm.com/Processors/Corstone-310Please use the examples of MLIA using Corstone-310 here to get started:https://github.com/ARM-software/open-iot-sdkTOSA CheckerThe TOSA Checker backend provides operator compatibility checks against the
TOSA specification.Please, install it into the same environment as MLIA using this command:mlia-backendinstalltosa-checkerAdditional resources:Source code:https://review.mlplatform.org/admin/repos/tosa/tosa_checkerPyPi packagehttps://pypi.org/project/tosa-checker/VelaThe Vela backend provides performance metrics for Ethos-U based systems. It
comes pre-installed.Additional resources:https://pypi.org/project/ethos-u-vela/