TensorFlow Lite Micro
By: Derrick Mwiti
Tags: Tinyml, TensorFlow, Heartbeat, Machine Learning

According to Statistica, 25.6 billion units of microcontrollers were shipped in 2019. There are over 250 billion microcontrollers in the world and this number is projected to grow over the coming years. As a result of this, deep learning on embedded devices is one of the fastest-growing fields. This area is popularly known as tiny machine learning (TinyML). That said, embedded devices pose a couple of challenges, key among them being their low processing power and limited memory. Machine learning models must therefore be able to work on just a few kilobytes of memory. They must also be able to perform inferencing with the low processing power available in embedded systems. In this piece, we’ll look at TensorFlow Lite Micro (TF Micro) whose aim is to run deep learning models on embedded systems. TF Micro is an open-source ML inference framework that has been fronted by researchers from Google and Harvard University. It addresses the resource constraints faced with running deep learning models on embedded systems.

Applications of TinyML

The authors of the above TinyML article start by highlighting some of the applications of TinyML technology. Key among them being:

Wakeword detection — waking up a device with a certain phrase, e.g., ‘Ok, Google.’

Predictive maintenance — resulting from analysis and modeling of signals from microphones, sensors, and accelerometers, just to mention a few.

Acoustic-anomaly detection.

Visual object detection.

Human-activity recognition.

Challenges Facing TinyML

Next, the authors highlight the major challenges faced when implementing machine learning in embedded devices. These include:

There is a lack of a unified TinyML framework for embedded devices.

Deploying models on embedded devices is slow.

Vendors produce different hardware types, hence making it difficult to evaluate hardware performance.

There is a lack of basic features such as memory management and library support.

The proposed TensorFlow Lite Micro (TF Micro) is aimed at solving these challenges.

TensorFlow Lite Micro (TF Micro) Design

The TF Micro framework has been designed with the following principles:

Minimize Feature Scope for Portability

This means that an embedded machine learning framework assumes that the model, input data, and output arrays are in memory. The framework should handle ML computations based on those values, meaning that the library will omit features such as loading models from a filesystem or accessing peripherals for inputs. This design principle is crucial because many embedded platforms don’t have a memory management mechanism as well as library support.

Enable Vendor Contributions to Span Ecosystem

As a result of device fragmentation, developers can’t build software that runs well on different embedded platforms. In order to tackle this challenge, the authors ensure that optimizing the core library operations is easy. They aim for ensuring significant technical support for developers. The authors also encourage submissions to a library repository.

Reuse TensorFlow Tools for Scalability

Exporting a model comes with its own set of challenges. Key among them being export errors that result from developers converting a model to a format that can run on an embedded device. Another challenge is the lack of support for some operations on the target device. It is also important to note that most models are trained on floating operations and these operations consume more space and memory and would therefore be unsuitable for an embedded device. This can be addressed by converting them to a quantized representation. This, however, increases the exporter complexity. To address these challenges, a converter is built on top of the existing TensorFlow Lite toolchain. This has been extended for deeply embedded machine learning systems.

Build System for Heterogeneous Support

This feature aims at achieving a flexible build environment that doesn’t rely on any one platform. This would definitely encourage the adoption of TF Micro by developers.

TensorFlow Lite Micro (TF Micro) Implementation

Developing a TF Micro application involves a couple of steps. The first one is to create a live neural-network-model object in memory. The application developer starts by producing an operator resolver object via the Client API. The operator resolver controls which operations link to the final binary in order to reduce the size of the executable.

The second step is to provide a contiguous memory array referred to as the arena. The arena holds intermediate results and variables that are needed by the interpreter.

The third step is to create an interpreter instance. The instance arguments are the model, operator resolver, and the arena. During the initialization phase, the interpreter allocates all required memory from the arena. All communications between the interpreter and operators are handled by a C API. This ensures that operator implementations are modular and independent of the interpreter’s implementation.

The fourth step is the model execution. At this point, the application retrieves pointers to the memory regions that represent the model inputs. It then populates them with values mainly obtained from sensors or user-supplied data. With the inputs available, the interpreter is invoked so that it can perform model calculations.

Finally, the interpreter returns control to the application after it has evaluated all the operations.