Package: ethos-u-vela
VelaThis tool is used to compile aTensorFlow Lite for Microcontrollersneural network model into an optimised version that can run on an embedded
system containing anArm Ethos-U NPU.In order to be accelerated by the Ethos-U NPU the network operators must be
quantised to either 8-bit (unsigned or signed) or 16-bit (signed).The optimised model will contain TensorFlow Lite Custom operators for those
parts of the model that can be accelerated by the Ethos-U NPU.  Parts of the
model that cannot be accelerated are left unchanged and will instead run on the
Cortex-M series CPU using an appropriate kernel (such as theArmoptimisedCMSIS-NNkernels).After compilation the optimised model can only be run on an Ethos-U NPU
embedded system.The tool will also generate performance estimates (EXPERIMENTAL) for the
compiled model.The tool has limited functionality for compiling aTOSAneural network
(EXPERIMENTAL).TensorFlow SupportVela is tested by comparing the bit exact numerical behaviour of the Ethos-U
optimised operators against that of the corresponding TensorFlow Lite reference
kernels (or TensorFlow Lite for Microcontrollers reference kernels in the case
of the UNIDIRECTIONAL_SEQUENCE_LSTM operator).  The following list indicates
which version is used for comparison:Vela 3.11.0 to current supports TensorFlow 2.15Vela 3.10.0 supports TensorFlow 2.14Vela 3.9.0 supports TensorFlow 2.12Vela 3.8.0 supports TensorFlow 2.11Vela 3.6.0 to 3.7.0 supports TensorFlow 2.10Vela 3.5.0 supports TensorFlow 2.9Vela 3.4.0 supports TensorFlow 2.8Vela 3.3.0 supports TensorFlow 2.7Vela 3.1.0 to 3.2.0 supports TensorFlow 2.5Vela 2.1.0 to 3.0.0 supports TensorFlow 2.4Vela 2.0.0 to 2.0.1 supports TensorFlow 2.3Vela 0.1.0 to 1.2.0 supports TensorFlow 2.1Python Version SupportThe majority of Vela's testing is done using a single version of Python, as
indicated by the first version in the list below.  However, some additional
testing is also performed across a range of newer versions starting at the
minimum version (pyproject.toml:project.requires-python) indicated in the
brackets:Vela 3.10.0 to current supports Python 3.10 (3.9)Vela 3.9.0 supports Python 3.10 (3.8)Vela 3.8.0 supports Python 3.9 (3.8)Vela 3.4.0 to 3.7.0 supports Python 3.7 (3.8)Vela 3.3.0 supports Python 3.8 (3.7)Vela 0.1.0 to 3.2.0 supports Python 3.6 (3.7)EnvironmentVela runs on Linux and Microsoft Windows 10 operating systems.PrerequisitesThe following should be installed prior to the installation of Vela:Python 3.10 or compatibleDevelopment version containing the Python/C API header filese.g.apt install python3.10-devoryum install python310-develPip3C99 capable compiler and associated toolchainFor Linux operating systems, a GNU toolchain is recommended.For Microsoft Windows 10, the Microsoft Visual C++ 14.2 Build Tools are recommended.
Seehttps://wiki.python.org/moin/WindowsCompilersInstallationVela is available to install as a package fromPyPi, or as
source code fromML Platform.
Both methods will automatically install all the required dependencies.PyPiInstall Vela from PyPi using the following command:pip3installethos-u-velaML PlatformFirst obtain the source code by either downloading the desired TGZ file from:https://review.mlplatform.org/plugins/gitiles/ml/ethos-u/ethos-u-velaOr by cloning the git repository:gitclonehttps://review.mlplatform.org/ml/ethos-u/ethos-u-vela.gitOnce you have the source code, Vela can be installed using the following
command from the root directory of the repository:pip3install.Advanced Installation for DevelopersIf you plan to modify the Vela codebase then it is recommended to install Vela
as an editable package to avoid the need to re-install after every modification.
This is done by adding the-eoption to the install command like so:pip3install-e.[dev]If you plan to contribute to the Vela project (highly encouraged!) then it is
recommended to install Vela with the development dependencies (seeVela Testingfor more details).RunningVela is run with an input.tfliteor.tosa(EXPERIMENTAL) file passed on the
command line. This file contains the neural network to be compiled. The tool then
outputs an optimised.tflitefile with a_velasuffix in the file name, along
with performance estimate (EXPERIMENTAL) CSV files, all to the output directory.
It also prints a performance estimation summary back to the console, seeVela Performance Estimation Summary.Example usage:Compile the networkmy_model.tflite.  The optimised version will be output
to./output/my_network_vela.tflite.velamy_model.tfliteCompile the network/path/to/my_model.tfliteand specify the output to go
in the directory./results_dir/.vela--output-dir./results_dir/path/to/my_model.tfliteCompile a network targeting a particular Ethos-U NPU.  The following command
selects an Ethos-U65 NPU accelerator configured with 512 MAC units.vela--accelerator-configethos-u65-512my_model.tfliteCompile a network while minimizing peak SRAM usage, prioritising lower SRAM
usage over runtime performance.vela--optimiseSizemy_model.tfliteCompile a network to have maximum performance, i.e. the fastest inference time.
This prioritises a higher runtime performance over a lower peak SRAM usage.vela--optimisePerformancemy_model.tfliteCompile a network while optimising for the fastest inference time possible,
with an upper bound for the SRAM usage. The memory limit is set in bytes, i.e.
run the following example if one requires a limit of 300KB.vela--optimisePerformance--arena-cache-size300000my_model.tfliteCompile a network using a particular embedded system configuration defined in
Vela's configuration file.  The following command selects theMy_Sys_Configsystem configuration along with theMy_Mem_Modememory mode from thevela.iniconfiguration file located in the config_files directory.vela--configArm/vela.ini--system-configMy_Sys_Config--memory-modeMy_Mem_Modemy_model.tfliteTo get a list of all available configuration files in the config_files directory:vela--list-config-filesTo get a list of all available options (see CLI Options section below):vela--helpWarningsWhen running the Vela compiler it may report a number of warning messages to the
console. These should all be thoroughly reviewed as they will indicate decisions
that the compiler has made in order to create the optimised network.Example NetworksSome example networks that contain quantised operators which can be compiled by
Vela to run on the Ethos-U NPU can be found at:https://tfhub.dev/s?deployment-format=lite&q=quantizedKnown Issues1. NumPy C API version changeOnce ethos-u-vela is installed, the user might want to install a different NumPy
version that is still within the dependency constraints defined in pyproject.toml.In some scenarios, doing so might prevent ethos-u-vela from functioning as
expected due to incompatibilities between the installed NumPy C headers used in
the mlw_codec and the current version of NumPy.Example scenario:In the ethos-u-vela source directory, run:virtualenv-p3.10venv
.venv/bin/activate
pipinstallethos-u-velaNext, install a different NumPy version (e.g. 1.21.3)pipinstallnumpy==1.21.3--forceFinally, run ethos-u-vela. You might get an error similar to this:ImportError: NumPy C API version mismatch
(Build-time version: 0x10, Run-time version: 0xe)
This is a known issue most likely caused by a change in the API version in
NumPy after installing ethos-u-vela.SolutionIn order for ethos-u-vela to work with an older version of NumPy that uses
different C APIs, you will need to install the desired NumPy version first, and
then build ethos-u-vela with that specific NumPy version:Uninstall ethos-u-vela and install the desired version of NumPypip uninstall ethos-u-vela
pip install numpy==1.21.3 --forceInstall required build dependenciespip install "setuptools_scm[toml]<6" wheelInstall ethos-u-vela without build isolation. Not using build isolation
ensures that the correct version of NumPy is used when copying the C headers
in mlw_codec during the build process.pip install ethos-u-vela --no-build-isolation --no-cache-dirAPIsPlease seeVela External APIs.Bug ReportingPlease seeVela Community Bug Reportingfor a description of how to
report bugs.ContributionsPlease seeVela Contributions.Debug DatabasePlease seeVela Debug Database.Inclusive language commitmentThis product conforms to Arm’s inclusive language policy and, to the best of
our knowledge, does not contain any non-inclusive language. If you find
something that concerns you, emailterms@arm.com.OptionsPlease seeVela CLI Options.  This includes a description of the
system configuration file format.PerformancePlease seeVela Performance Estimation Summary.ReleasesPlease seeVela Releases.ResourcesAdditional useful information:Arm Products: Ethos-U55 NPUArm Products: Ethos-U65 NPUArm Developer: Ethos-U55 NPUArm Developer: Ethos-U65 NPUSecurityPlease seeVela Security.Supported OperatorsPlease seeVela Supported Operatorsfor the list of
operators supported in this release.TestingPlease seeVela Testing.LicenseVela is licensed underApache License 2.0.