# Embedded Artificial Intelligence for IoT Applications
Using the MAX78000

Martina Maria Balbi Antunes, Lance Doherty, Thomas Watteyne

► To cite this version:

Martina Maria Balbi Antunes, Lance Doherty, Thomas Watteyne. Embedded Artificial Intelligence for IoT Applications Using the MAX78000. IEEE Access, 2025, 13, pp.38979-39005. 10.1109/AC-CESS.2025.3546557. [hal-04982025](https://hal-04982025)

HAL Id: hal-04982025

https://inria.hal.science/hal-04982025v1

Submitted on 7 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Embedded Artificial Intelligence for IoT Applications using the MAX78000

Martina Balbi<fnref n="1" />,<fnref n="2" />, Lance Doherty<fnref n="1" />, Thomas Watteyne<fnref n="1" />,<fnref n="2" />

Corresponding author: Martina Balbi (e-mail: martina.balbi@analog.com).

**ABSTRACT** Recent advances in embedded AI and IoT have revolutionized the development of intelligent edge devices. This work provides a tutorial on developing and deploying AI models on the MAX78000, a low-power microcontroller designed specifically for AI applications. Starting with the foundational understanding of neural networks and machine learning, this tutorial explores the architecture and capabilities of the MAX78000, which integrates a CNN accelerator with an ARM Cortex-M4 core. We give practical guidance on creating, training, and quantizing AI models, detailing essential tools, frameworks, and the deployment process. Real-world examples illustrate the versatility of AI microcontrollers and their performance in various IoT applications. Emphasizing the importance of accessible development tools, this tutorial aims to increase awareness within the IoT community about low-power accelerators. This enables developers to create efficient, real-time AI solutions, highlighting the transformative potential of embedded AI in IoT.

**INDEX TERMS** Artificial Intelligence, Convolutional Neural Networks, Embedded Systems, Hardware Accelerators, Internet of Things.

## I. INTRODUCTION

Embedded Artificial Intelligence (AI), referring to the integration of AI capabilities directly into devices through hardware inference at the edge, represents a significant advancement in the Internet of Things (IoT). Historically, implementing AI within IoT systems required extensive data transmission to cloud servers for processing, which led to substantial energy consumption, increased latency, and higher operational costs. These constraints made AI integration impractical for resource-constrained devices like those in IoT. However, the development of embedded AI allows for local data processing and analysis, enabling real-time decision-making that is both efficient and cost-effective [1]. This paradigm shift towards edge computing, where data is processed locally on devices rather than being sent to centralized cloud servers [2], with embedded AI enhances energy efficiency, boosts privacy and security, and improves the reliability and resilience of IoT systems [3].

The rapid advancement of embedded AI has been propelled by significant improvements in both hardware and software [4]. Today, specialized AI chips and processors are being developed specifically for edge devices, each varying in terms of power efficiency and computational capabilities. These dedicated accelerators typically feature large on-chip memory to minimize costly off-chip memory accesses, extensive computational arrays for efficient matrix operations, dedicated data flows for optimal data reuse, and support for low precision computing [5].

<figure><img src="image_2.png" /><figcaption>FIGURE 1. Performance vs. Power of AI Accelerators.</figcaption></figure>

Fig. 1 illustrates the performance and power trade-offs among various AI accelerators, showcasing the diversity of solutions available for IoT applications. High-performance devices like the Nvidia A100 and Google TPUv3 deliver exceptional computational throughput suitable for data centers but operate at power levels that make them impractical for IoT use cases. In contrast, edge-focused devices like the Nvidia Jetson Nano and Google Coral Edge TPU strike a

<footnote n="1">¹Analog Devices</footnote>

<footnote n="2">²Inria Paris</footnote>
balance between computational power and moderate energy consumption, making them viable for robotics and smart city applications.

For ultra-low-power applications, microcontrollers such as the MAX78000, GAP8, and GAP9 are optimized to deliver AI processing capabilities while consuming only milliwatts of power. These devices are designed for resource-constrained environments, including wearable devices, remote sensors, and battery-operated IoT systems.

This work presents a tutorial on leveraging this technology, specifically focusing on the MAX78000¹, an ultra-low power device with which we have extensive experience and which is representative of other ultra low-power AI microcontrollers on the market. The key contributions of this work include:

* **Hands-on guide for the MAX78000:** A step-by-step tutorial that enables readers to implement AI-driven functionalities directly on the MAX78000, providing practical insights for leveraging this device in real-world applications.

* **Focus on ultra-low power AI accelerators:** By examining the MAX78000, this paper highlights the potential of ultra-low power devices for resource-constrained IoT environments, a growing area of interest as energy efficiency becomes more critical in embedded AI.

* **Bridging theory and practice:** This work offers a balance of theoretical background and practical implementation, making it accessible for both newcomers to embedded AI and those seeking to expand their expertise in AI hardware accelerators.

* **Broadening awareness of embedded AI solutions:** The paper aims to raise awareness of specialized AI accelerators like the MAX78000, fostering innovation and driving further adoption of embedded AI technologies in IoT systems.

Section II introduces the concept of neural networks, highlights the evolution of ML, and discusses its integration into edge devices like the MAX78000. Section III delves into key concepts and processes of ML, explaining how models are built, trained, and evaluated. Section IV explores the hardware design and capabilities of the MAX78000. Section V details the process of creating AI models tailored to this microcontroller. Section VI provides a step-by-step guide on training AI models for the MAX78000. Section VII describes the steps to deploy a trained model on the MAX78000. Section VIII presents various IoT applications using the MAX78000, illustrating its versatility and performance in real-world scenarios. Section IX discusses the future prospects of embedded AI. Finally, Section X summarizes the key takeaways and emphasizes the significance of integrating AI into edge devices for future innovations.

## II. BACKGROUND

Understanding the distinction between human brains and computers is fundamental to appreciate ML. The human

<figure><img src="image_3.png" /><figcaption>**FIGURE 2.** Illustrating the connections between neurons, highlighting excitatory signals (green) and inhibitory signals (red) as they transmit information through neural networks.</figcaption></figure>

brain, with approximately 86 billion neurons [6], processes information in a highly parallel and distributed manner, enabling sophisticated tasks such as pattern recognition and decision-making [7]. Neurons form complex networks, each connecting to thousands of others, allowing immense learning and adaptation through synaptic plasticity. This biological system contrasts with traditional computers that operate based on sequential processing and deterministic algorithms.

Computers, designed with a Central Processing Unit (CPU) and memory, excel at performing repetitive, arithmetic, and logical operations at high speed, but struggle with tasks requiring understanding or adaptation without explicit programming [8]. For instance, early attempts at voice synthesis were largely successful [9], yet voice recognition remained a persistently difficult challenge. Similarly, while early computer graphics were primitive but functional, image recognition algorithms often yielded disappointing results [10]. This highlighted the limitations of traditional computing in certain classes of problems, particularly those involving classification tasks like identifying the subject of a photograph or extracting keywords from spoken language.

A brain processes information differently than a computer, recognizing patterns based on experiences, while computers execute algorithmic steps. As represented in Fig. 2, a brain is composed of neurons with multiple inputs and a single output connecting to other neurons. When specific input conditions are met, the neuron fires, sending excitatory or inhibitory pulses to the other connected neurons. Neural networks, which are computer models inspired by the brain's structure, consist of interconnected layers of nodes (neurons), where each node processes information and passes it to the next layer, allowing the network to learn and recognize patterns in data [11].

The field of ML has evolved significantly, from early linear models and decision trees to the mentioned neural networks and deep learning [12]. In particular, embedded AI allows intelligent processing on low-power, resource-constrained devices, enabling real-time decision-making and reducing cloud connectivity dependence [13]. This trend is driving the development

<footnote n="1">¹ https://www.analog.com/en/products/max78000.html</footnote>
<figure><img src="image_4.png" /><figcaption>FIGURE 3. Architecture of the MAX78000 microcontroller.</figcaption></figure>

opment of smart, connected systems, where devices process data locally and make intelligent decisions.

Analog Devices' MAX78000 microcontroller exemplifies this concept by incorporating a Convolutional Neural Network (CNN) accelerator alongside an ARM Cortex-M4 and a RISC-V core. A CNN accelerator is a specialized hardware component designed to efficiently execute the operations of convolutional neural networks, such as convolutions, pooling, and activation functions [14]. These operations are computationally intensive, but the accelerator significantly speeds them up while minimizing energy consumption. The ARM Cortex-M4, featuring a floating-point unit and Single Instruction, Multiple Data (SIMD) instruction support, efficiently handles algorithmic tasks, while the RISC-V core complements the system with additional processing capabilities tailored for specialized functions. This configuration allows the accelerator to handle AI tasks effectively, while the Cortex-M4 and RISC-V cores manage other tasks. This design makes the MAX78000 an excellent solution for battery-operated IoT devices. Its architecture can be seen in Fig. 3.

### III. BASICS OF MACHINE LEARNING

Having established the context and the platform, we now turn our attention to the basics of machine learning, which form the foundation of AI models deployed on embedded systems like the MAX78000.

Embedded developers traditionally approach problem-solving through a systematic process: they define the problem, devise an algorithm, translate it into computer code, compile the code, and rigorously test and evaluate its performance [15]. If enhancements are needed, they refine the algorithm, update the code, recompile, and conduct iterative testing until satisfactory results are achieved. This method has been foundational in the field of embedded systems development for decades. In contrast, ML introduces a paradigm shift in problem-solving strategies.

Neurons within a neural network can be compared to logic gates in a microcontroller core. Individually, a single neuron lacks significant utility. However, when aggregated with thousands of other neurons and organized into layers – from input layers through multiple hidden layers to output layers – the neural network demonstrates remarkable capabilities.

### A. PRACTICAL EXAMPLE: ROAD SIGN RECOGNITION

To illustrate this concept, consider the problem of recognizing road signs (Fig. 4).

#### 1) Input Layer

In this example, the input layer of the neural network processes a two-dimensional image. Typically, the number of input neurons corresponds to the number of pixels in the image. However, this does not imply that each neuron is connected solely to a single input pixel. Neurons can have multiple inputs, allowing each input neuron to connect to an input pixel and its neighboring pixels within the image array (Fig. 4 (b)), with each input pixel being appropriately weighted.

High-resolution images (Fig. 4 (a)) is 640x640 pixels) are not typically used as direct inputs for neural networks due to the computational intensity and inefficiency that millions of connections would entail. Instead, it is more effective to downsample the input image to a lower resolution (Fig. 4 (c)), such as 32x32 or 64x64 pixels, thereby reducing the complexity and number of connections in the neural network. Increasing image resolution does not significantly enhance recognition accuracy, whereas downsampling allows for a more efficient allocation of neurons to the hidden layers.

#### 2) Hidden Layers

The hidden layers, shown in Fig. 5, are arrangements of neurons that receive input from the preceding layer and provide output to the subsequent layer. As data traverses the network, each neuron computes its output value by applying a specific function to its input, known as an activation function. An activation function determines whether a neuron should fire based on the input it receives, introducing non-linearity into the network, which is essential for learning complex patterns [16]. Additionally, the connections between neurons are characterized by multipliers called weights, and each neuron has an internal offset referred to as a bias. The internal components of a neuron are shown in Fig. 6.

#### 3) Output Layer

The output layer comprises another set of neurons that receive input from the final hidden layer, as shown in Fig. 7. Each neuron in the output layer corresponds to a specific possible outcome, in this case, a particular road sign. The output value of each neuron is a probability, scaled between zero and one, representing the likelihood that the image contains the associated road sign. For instance, if the neuron associated with stop signs emits a value of 0.97, it is very likely that a stop sign is present in the field of view. Conversely, if the neuron associated with yield signs emits a value of 0.03, it is unlikely that a yield sign is present. If the output values are scattered without a clear dominant value, it indicates that the network is unable to interpret the inputs and cannot determine the contents of the field of view.
<figure><img src="image_5.png" /><figcaption>FIGURE 4. Recognition of a road sign.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>FIGURE 5. Hidden layers of a road sign recognition model.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>FIGURE 6. Internal components of a neuron, including weights, bias and activation function.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>FIGURE 7. Output layer of a road sign recognition model. This model is able to recognize 12 road signs.</figcaption></figure>

## B. MACHINE LEARNING DEVELOPMENT FLOW

The arrangement of activation functions, weights, and biases at each neuron allows for the dynamic adjustment of connections within a neural network, enabling the differentiation of data through multiple layers. Two critical questions arise in neural network design: *how to determine the optimal structure of the hidden layers*, and *how to select the appropriate activation functions, weights, and biases*.

The arrangement of a neural network – including the structure of the input, hidden, and output layers – is collectively referred to as the “model”. Designing and building the model is one of the more challenging aspects of neural network engineering, as the structure largely depends on the type of input to be processed and other relevant factors.

The optimal values for weights and biases are not manually chosen; rather, they are learned through training the neural network. This process involves exposing the network to a large and diverse set of sample data, analyzing the output, and incrementally adjusting the weights and biases. For example, training a network to recognize road signs would require a dataset containing thousands of images of various sign types. This dataset is typically divided into training and validation subsets, with 80% used for training and 20% reserved for validation [17].

During training, each sample is fed into the model, and the output is compared to the expected result. Weights associated with correct outputs are strengthened, while those linked to incorrect predictions are adjusted to minimize error. This iterative optimization process is computationally demanding due to the large number of neurons and billions of internal connections, requiring significant computational resources. High-end Graphics Processing Units (GPUs), capable of performing numerous mathematical operations simultaneously, are commonly employed to handle the intensive computations required for training [18].

After training, the model's performance is evaluated using the smaller validation set. If the results meet the desired performance criteria, the model is deemed ready for deployment. Otherwise, further tuning and retraining are necessary to improve its accuracy and reliability.

This process of ML development differs fundamentally
<figure><img src="image_9.png" /><figcaption>FIGURE 8. Comparison of neural networks and traditional algorithms.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>FIGURE 9. Representation of a multichannel layer.</figcaption></figure>

from conventional microcontroller development, which involves formulating a precise solution through algorithmic development. In contrast, ML systems require creating a model and providing extensive training data. This difference is seen in Fig. 8.

### C. UNDERSTANDING HIDDEN LAYERS IN CONVOLUTIONAL NEURAL NETWORKS

CNNs are among the most effective types of networks for ML applications [19]. In a CNN, each layer may consist of many two-dimensional planes. In ML terminology, each plane is called a channel, and the output of a collection of channels in a layer is called a feature map or an activation map, as seen in Fig. 9. A hidden layer can have any number of these channels, depending on the model designer's choice. The best way to conceptualize a layer is as a three-dimensional cube, with height and width representing the dimensions we explored previously. Operations on the layer can be thought of as looking through the cube in the depth dimension. The operations that can be performed in a hidden layer of a CNN are presented below.

<figure><img src="image_11.png" /><figcaption>FIGURE 10. Convolution operation using a 3x3 kernel over 3 channels.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>FIGURE 11. Example of a convolution on an RGB image using a 3x3 kernel and generating 16 output channels.</figcaption></figure>

#### 1) Convolution

A convolution is a mathematical operation used to extract features from the input data by applying a kernel (or filter) that slides over the input data and computes a dot product between the kernel and a local region of the input [20]. This process produces an output feature map that highlights the presence of specific patterns in the input data. For instance, a 3x3 convolutional kernel might detect edges or textures in an image by examining 3x3 regions and computing a weighted sum of the pixels within the region. Considering the multichannel nature of the layers, a convolution looks at all the data elements covered by the 3x3 block in every channel within that layer, as represented in Fig. 10.

For example, consider the scenario depicted in Fig. 11. The input layer of a Face ID model takes a 160x120 pixel image with red, green, and blue primary colors, resulting in three color channels. The first hidden layer is a convolutional layer with 16 feature map channels. Thus, the input layer has 19200 pixels (160x120) multiplied by 3 color channels, totaling 57600 input elements.

To determine the number of weights required for these 57600 input elements as they transition to the first hidden layer, we start by considering nine elements in the source layer (as we are applying a 3x3 kernel) for each element in the target layer: the element that directly corresponds with the target element and its eight neighbors, resulting in nine weights. These nine weights are then moved across the entire source layer, computing a dot product for every element in the target layer.
Since we are evaluating every plane of the source layer—in this case, the three primary colors (red, green, and blue)—each element of the target layer is the sum of 27 products (9 products from each of the three planes), requiring 27 weights. Furthermore, considering that the first hidden layer consists of 16 output channels forming the feature map, the total number of weights required in the transformation from the input layer (the full-color image in RGB color space) to the first internal feature map is 432 (27 weights per channel multiplied by 16 output channels).

## 2) Pooling

Another important operation in CNNs is pooling, shown in Fig. 12. Pooling involves taking a collection of input values and creating a single output value by applying a simple operation [21]. The most common pooling operations are taking the largest input value, known as max pooling (Fig. 12 (b)), or computing the average of all input values, known as average pooling (Fig. 12 (c)). Pooling is typically performed over a 2x2 group of elements, effectively reducing the size of each channel by a factor of four.

From a traditional data analysis perspective, pooling might seem like discarding valuable data. While it does reduce the data, this reduction is not about losing important details, but about enhancing the network's ability to generalize. The purpose of neural networks is not to exactly recreate the input data, but to classify it accurately by identifying whether it belongs to a particular category. This task requires the network to be robust to variations such as spatial shifts in images or time shifts in sequences. Pooling operations, like max pooling, help achieve this robustness by making the network invariant to such shifts. For example, in max pooling, the specific location of the maximum value within the pooled region is irrelevant; the output depends only on the highest value, not where it appears. By combining convolution and pooling, the network can focus on distinguishing meaningful features while ignoring irrelevant variations, thereby improving its classification accuracy.

Other techniques can also be employed to manage data in a neural network. For example, results below a certain threshold can be discarded, or layers can be reshaped. A layer consisting of multiple 2-D channels can be transformed into a 1-D string. These methods collectively provide a robust toolkit for directing data flow from the input layer through the hidden layers to the output layer.

## 3) Activation Functions

While we've mentioned activation functions before, it's important to understand their purpose. The activation function introduces the necessary non-linearity into the model, allowing it to learn and make complex decisions [22]. This idea is inspired by biological neurons, which remain inactive until their inputs reach a certain threshold, at which point they “fire” and send signals to other neurons—a highly nonlinear process. Similarly, in an ML network, introducing non-

linearity through activation functions is essential for effective learning and classification.

Common activation functions are the sigmoid, the ReLU (Rectified Linear Unit) or the ABS. The sigmoid function (Fig. 13 (a)), characterized by its S-shaped curve, is common in natural systems and follows a simple exponential formula. The ReLU function (Fig. 13 (b)) is even more straightforward: it outputs zero if the weighted sum of the inputs is negative and the weighted sum itself for positive values. The ABS function outputs the absolute value of the input, ensuring all activations are non-negative (Fig. 13 (c)). Most conventional layers use the ReLU function, while pooling layers often do not require an activation function, as they simply output the maximum or average values of the inputs.

# D. TRAINING NEURAL NETWORKS: KEY STEPS AND TECHNIQUES

To achieve effective neural network performance, it is crucial to train it correctly. This process involves adjusting the network's weights and biases through iterative optimization [23]. Typically, networks are trained once during development and testing, after which the parameters are fixed, allowing the model to be deployed without further adjustments.

During training, the network is shown numerous labeled examples, each accompanied by a target value. Each time an example is presented, the network's parameters are optimized to align the predicted and actual values. Once this process is complete, the network can accurately predict new examples it has not seen during training.

### 1) Overfitting and Underfitting

Determining the appropriate complexity for a network, including the number of layers or the number of parameters, is a common challenge in neural network modeling. This complexity directly impacts overfitting and underfitting, which are critical to model performance [24].

Overfitting occurs when a model is too complex, with too many parameters relative to the training data. It performs well on training data but poorly on unseen test data, indicating that it has memorized the training examples rather than learning to generalize patterns. Reducing the network's complexity can mitigate overfitting.

Underfitting happens when a model is too simple to capture the underlying patterns in the data, resulting in high loss on both training and test datasets. Increasing the model's complexity, such as adding more layers or using more parameters, can help capture the data's nuances better.

### 2) Loss Function and Training Algorithms

Learning in neural networks involves a two-step process: feedforward and backpropagation.

In feedforward, the network processes an input through its layers to generate an output vector. The class with the highest score represents the network's prediction. The loss function quantifies the difference between this output vector and the actual target value, measuring how well the predictions match
<table><caption>a) Original matrix.</caption><tr><td>21</td><td>-6</td><td>-42</td><td>-4</td><td>50</td><td>-15</td><td>-14</td><td>-5</td></tr><tr><td>-37</td><td>-45</td><td>-46</td><td>-26</td><td>25</td><td>-27</td><td>20</td><td>-15</td></tr><tr><td>-2</td><td>-47</td><td>35</td><td>42</td><td>-36</td><td>39</td><td>21</td><td>-32</td></tr><tr><td>1</td><td>5</td><td>31</td><td>14</td><td>-40</td><td>16</td><td>3</td><td>23</td></tr><tr><td>43</td><td>-9</td><td>-39</td><td>-16</td><td>-35</td><td>45</td><td>16</td><td>16</td></tr><tr><td>12</td><td>-48</td><td>-23</td><td>13</td><td>-47</td><td>-4</td><td>-19</td><td>34</td></tr><tr><td>-23</td><td>40</td><td>48</td><td>5</td><td>-19</td><td>-24</td><td>-43</td><td>28</td></tr><tr><td>44</td><td>-38</td><td>47</td><td>17</td><td>-21</td><td>-33</td><td>-10</td><td>-2</td></tr></table>

<table><caption>b) Max pool.</caption><tr><td>21</td><td>-6</td><td>-42</td><td>-4</td><td>50</td><td>-15</td><td>-14</td><td>-5</td></tr><tr><td>-37</td><td>-45</td><td>-46</td><td>-26</td><td>25</td><td>-27</td><td>20</td><td>-15</td></tr><tr><td>-2</td><td>-47</td><td>35</td><td>42</td><td>-36</td><td>39</td><td>21</td><td>-32</td></tr><tr><td>1</td><td>5</td><td>31</td><td>14</td><td>-40</td><td>16</td><td>3</td><td>23</td></tr><tr><td>43</td><td>-9</td><td>-39</td><td>-16</td><td>-35</td><td>45</td><td>16</td><td>16</td></tr><tr><td>12</td><td>-48</td><td>-23</td><td>13</td><td>-47</td><td>-4</td><td>-19</td><td>34</td></tr><tr><td>-23</td><td>40</td><td>48</td><td>5</td><td>-19</td><td>-24</td><td>-43</td><td>28</td></tr><tr><td>44</td><td>-38</td><td>47</td><td>17</td><td>-21</td><td>-33</td><td>-10</td><td>-2</td></tr></table>

<table><caption>c) Average pool.</caption><tr><td>-17</td><td>-30</td><td>8</td><td>-4</td></tr><tr><td>-11</td><td>31</td><td>-5</td><td>4</td></tr><tr><td>-1</td><td>-16</td><td>-10</td><td>12</td></tr><tr><td>6</td><td>29</td><td>-24</td><td>-7</td></tr></table>

<figure><img src="image_13.png" /><figcaption>FIGURE 12. Illustration of max pooling and average pooling operations.</figcaption></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /><figcaption>FIGURE 13. Most common activation functions.</figcaption></figure>

the true labels. Backpropagation minimizes the loss function by propagating the error backward through the network after the loss is computed in the feedforward step. This process updates the network's weights and biases to reduce the error.

Training involves repeating the feedforward and backpropagation steps over multiple iterations. Each complete pass through the entire training dataset is called an epoch. The total number of epochs required depends on factors such as the dataset's size and complexity and the neural network's architecture.

The balance between the number of epochs, computational resources, and training time is crucial, as more epochs generally improve the model's performance, but with an increased risk of overfitting. Training continues until the loss value reaches an acceptable level or until the maximum number of epochs is completed.

### 3) Optimization Algorithms

Optimizing the weights and biases of a neural network involves several numerical optimization algorithms. The simplest is the gradient descent method [25], which iteratively moves towards the global minimum of the loss function using the gradient to guide the direction of steepest descent. The gradient provides the direction and magnitude of the steepest increase in the function's value at any point. Other renowned optimization algorithms include Stochastic Gradient Descent (SGD) [26], Momentum [27], AdaGrad [28], RMSProp [29]

<figure><img src="image_16.png" /><figcaption>FIGURE 14. Diagram illustrating the process of calculating MFCCs from a time series.</figcaption></figure>

and Adam [30].

### E. INTRODUCING THE KEYWORD SPOTTING MODEL

Having covered the fundamentals, we can now examine a practical application: the audio keyword spotting model (KWS20). This model processes a continuous audio stream with a CNN to detect predefined keywords, such as “up”, “down”, “left”, “right”, “stop”, “go”, “yes”, “no”, “on”, “off”, and the numerals “one” through “nine”, as well as “zero” and an “unknown” category.

Keyword spotting operates on one-dimensional time sequences, unlike image-based models. The input is sampled at 16,384 samples per second, with one second of data collected for each inference. The challenge lies in how the network handles time-sequenced data efficiently to ensure accurate classification within the time it takes to process the incoming audio.

A traditional approach to this involves transforming the audio's frequency spectrum into Mel-Frequency Cepstral Coefficients (MFCCs) [31]. MFCCs are features extracted from
<figure><img src="image_17.png" /><figcaption>**FIGURE 15.** Diagram illustrating an alternative approach to traditional signal processing.</figcaption></figure>

audio signals that represent their frequency content on a perceptual scale, and are easier to classify than raw time series data. The MFCC extraction process involves pre-emphasis to balance the spectrum, dividing the audio into overlapping frames with windowing, and applying the Fourier transform to obtain the magnitude spectrum. The frequencies are then mapped to the Mel scale using a filter bank, followed by applying a logarithmic scale. Finally, the Discrete Cosine Transform (DCT) compresses this information into fixed coefficients, the MFCCs (Fig. 14).

Alternatively, modern approaches may bypass traditional signal processing by training the neural network to directly approximate MFCCs from the raw audio signal (see Fig. 15). This approach involves feeding the audio samples into the network, which then uses one-dimensional convolutions to compress and reshape the data into a format that resembles MFCCs. The final layers of the network then perform the classification, identifying the keyword from among the possible options. Each output neuron in the final layer corresponds to a specific keyword, with an additional neuron for detecting “unknown” categories.

# IV. UNDERSTANDING THE MAX78000 ARCHITECTURE

Understanding machine learning fundamentals is crucial, but implementing them on hardware requires a deep dive into the underlying architecture. The next section explains the MAX78000's capabilities and how it supports AI applications.

Traditionally, AI models require powerful computers to manage the heavy computational load. However, having these models run on central servers requires continuous data streaming, leading to latency, higher energy consumption, and potential privacy issues. Embedded AI addresses these challenges by allowing models to be trained on powerful machines but deployed on small, energy-efficient devices. Several SoC platforms, like Google Edge TPU, Nvidia Jetson Nano, and Intel Neural Computing Stick, integrate accelerators for this purpose. Among these, the MAX78000 stands out for its ultra-low power design. The following sections will explore the MAX78000 in detail, demonstrating how it can effectively deploy real-time AI models at the edge.

## A. KEY FEATURES OF THE MAX78000

To understand the capabilities of the MAX78000 CNN accelerator, we need to examine its architecture and resources, depicted in Fig. 16. The accelerator comprises 64 processors,

<figure><img src="image_18.png" /><figcaption>**FIGURE 16.** Overview of the RAM and processors in the MAX78000 CNN accelerator.</figcaption></figure>

meaning that it can handle 64 processing tasks concurrently. It is also equipped with substantial memory resources, which include:

1) 512 kB of data memory. This memory is used to store intermediate data during processing.

2) 432 kB of RAM for weights: Dedicated to storing the weights of the neural network, this memory allows for efficient access and manipulation during inference.

3) 2 kB of bias memory: This memory holds the bias values associated with each neuron.

4) 384 kB of Tornado RAM: This working storage facilitates the operations of the CNN accelerator.

In total, these components provide over a megabyte of RAM dedicated to the CNN accelerator's functions. This extensive memory allocation is crucial for handling the computations required for ML tasks.

Importantly, these memory instances are accessible to the ARM Cortex-M4 supervisory processor. The RAM instances are mapped into the ARM Cortex-M4's APB (Advanced Peripheral Bus) space, enabling the ARM core to initialize the memory before delegating operations to the CNN accelerator. Although the ARM Cortex-M4 can interact with these memory spaces, the CNN accelerator operates as an independent peripheral block, distinct from the microcontroller's cores. This separation ensures that the CNN accelerator can function autonomously, allowing for efficient and parallel processing of neural network tasks.

The 64 processors within the CNN accelerator are organized into four processor quadrants, each containing 16 processors. These quadrants are further subdivided into four groups of four processors each. The manner in which these processors collaborate within their respective groups and how these groups interact within their quadrants form a key aspect of the system's architecture. This organization contributes significantly to the efficiency and effectiveness of the CNN accelerator in handling complex computational tasks.
<figure><img src="image_19.png" /><figcaption>FIGURE 17. Overview of a processor element in the MAX78000.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>FIGURE 18. Overview of a processor group in the MAX78000.</figcaption></figure>

## B. HOW THE CNN ACCELERATOR OPERATES: PROCESSOR-LEVEL INSIGHTS

The CNN accelerator's individual processors, represented in Fig. 17, initiate their operation by reading data from the associated data RAM. During data retrieval, the processor performs pooling if it is enabled for the current stage. The processor executes this pooling operation on the fly, without requiring a separate processing step.

The pooled data is then stored in a compute cache. This cache acts as a buffer, facilitating the convolution engine's simultaneous processing of multiple data pieces by ensuring all necessary elements are readily available. The convolution engine then convolves the input data with the kernel stored in the weight memory.

Each processor has its own segment of weight memory, structured as 768x72 bits. This specific configuration is designed to support common convolution operations. For instance, a 3x3 convolution requires nine weights, each up to eight bits, resulting in a total of 72 bits. Consequently, each processor can access up to 768 3x3 kernels. Importantly, the precision of these weights can be adjusted based on the sensitivity of the neuron's activated output to a given input. Depending on the layer and the requirements, the weights can be configured to use eight, four, two, or even just a single bit. This flexibility allows for more efficient use of memory, as lower precision weights can be packed more densely.

The convolution engine's output is a sum of products, where the product of two eight-bit values yields a 16-bit value. Given that up to nine values may be added together, additional bits are necessary to prevent overflow. Consequently, the convolution engine's output is set to a 26-bit value. An optional bias can be added to this result, which is then placed on a bus to be summed with results from other processors.

## C. PROCESSOR GROUP STRUCTURE

Four processors form a processor group, and each group has a dedicated 32 kB block of data RAM. During any given cycle, each processor can access eight bits of a 32-bit word, enabling simultaneous operations across all four processors.

A key feature of this architecture is the ability for any processor within a group to write to any address in RAM. This flexibility allows processors to share results freely: any

processor can retrieve and operate on the data produced by any other processor in the group. This inter-processor communication capability enhances the collaborative processing power of the group, facilitating efficient data handling and computation.

## D. PROCESSOR QUADRANT STRUCTURE

The MAX78000's CNN accelerator is built around a hierarchical processing structure, where four processor groups form a processor quadrant, as seen in Fig. 19. Each quadrant includes 512 bytes of bias memory that can be dynamically allocated to individual processors depending on the workload.

In the context of a CNN, each neuron in a given layer computes a weighted sum of activations from neurons in the previous layer, followed by an activation function. These activations are typically divided into multiple channels, with each processor in the quadrant managing the computations for a single input channel. For example, if the previous layer consists of 4 channels, the 4 processors of one processor group handle the convolutions independently, with each processor working on one input channel. The results are then summed locally, activated and stored in local memory.

When the previous layer has more than 4 channels, the results must be combined with those from other processor blocks in the quadrant or even from different quadrants. Each processor quadrant includes a sum of products block that aggregates convolution results from all processor groups within the quadrant and shares partial sums with other quadrants. This allows the accelerator to handle up to 64 input channels in a single pass by distributing the workload across quadrants and summing results hierarchically.

The multipass mode extends this capability further, enabling the processor to manage layers with more than 64 channels by storing intermediate results and performing additional passes. This mechanism allows the accelerator to process up to 1024 virtual channels, significantly broadening its applicability to complex neural network architectures.

To illustrate, consider a convolutional layer with 16 input channels. In this case, all computations would be handled
<figure><img src="image_21.png" /><figcaption>FIGURE 19. Overview of a processor quadrant in the MAX78000.</figcaption></figure>

by the 4 processor groups within a single quadrant. Each processor group would handle 4 channels, with the convolution results summed locally within the quadrant. The sum-of-products block in the quadrant would aggregate these results, apply scaling, quantization, and activation, and store the final results.

For larger layers, such as those with 64 input channels, the workload would be distributed across all quadrants. Each quadrant would handle 16 channels, and the results would be aggregated across quadrants to form the final output. In even larger cases, such as 128 channels, the multipass mode would divide the workload into manageable chunks, processing 64 channels per pass and combining intermediate results. The multipass mode enables the MAX78000 to handle layers with up to 1024 virtual channels. This flexibility makes the CNN accelerator capable of handling a wide variety of neural network and ML applications.

## E. LAYER MANAGEMENT IN THE CNN ACCELERATOR

Neurons in a network are structured into layers, with the output of one layer serving as the input to the next. The CNN accelerator in the MAX78000 has the capability to manage up to 64 layers. It computes the result for one layer, then uses the neuron outputs from that layer to form the input for the next layer. The accelerator adjusts RAM pointers as needed to ensure it has the appropriate inputs, weights, and biases for the layer it is currently processing.

## V. DEVELOPING AI MODELS FOR THE MAX78000

With a clear understanding of the MAX78000's architecture, we can now delve into the methodology for developing ML algorithms for this platform. To simplify and accelerate the

development, training, and deployment of AI models on the MAX78000, Analog Devices provides an extensive suite of resources, including pre-trained models, datasets, and detailed documentation.

The *ai8x-training* repository<fnref n="2" /> serves as a robust framework for building and training neural networks optimized for Analog Devices' microcontrollers. This repository includes pre-trained models for immediate deployment or fine-tuning, datasets for training and validation and training scripts. After training a model, the *ai8x-synthesis* repository<fnref n="3" /> provides tools to convert these models into deployable C code, enabling seamless deployment on the MAX78000.

For a comprehensive guide to getting started, the MaximAI Documentation repository<fnref n="4" /> offers step-by-step instructions on setting up your development environment, along with links to additional documentation and resources. Together, these tools and references form an integrated workflow, allowing developers to efficiently design, train, and deploy AI models on the MAX78000 platform.

## A. BUILDING MODELS CONSIDERING THE PRACTICAL LIMITATIONS OF THE MAX78000

We now understand the basis of ML models and how to train them with numerous examples to adjust the weights and biases of each neuron effectively. If we were deploying the ML solution on a training platform with high-performance, power-intensive GPUs, the process would be complete. However, practical deployment often requires solutions that are

<footnote n="2">²https://github.com/analogdevicesinc/ai8x-training</footnote>

<footnote n="3">³https://github.com/analogdevicesinc/ai8x-synthesis</footnote>

<footnote n="4">⁴https://github.com/analogdevicesinc/MaximAI_Documentation</footnote>
<figure><img src="image_22.png" /><figcaption>**FIGURE 20.** Comparison of the neural network development environment and the deployment environment.</figcaption></figure>

compact, lightweight, and highly power-efficient. This calls for specialized and optimized devices. In this discussion, we will explore the MAX78000's capabilities as an efficient platform for deploying ML at the edge.

Let's first examine the neural network development environment. During training, neural network models use continuous values for weights and neuron activations, typically ranging between -1 and 1. The values transferred between neurons often fall between zero and one, but can exceed these bounds before being clamped to fit within the activation function's range. This requires a substantial dynamic range in the training environment.

However, when deploying on the MAX78000, we transition from using floating-point values to eight-bit signed integers, ranging from -128 to +127, a process known as quantization. This shift to a more limited numeric representation presents a key challenge, as weights and biases must be scaled to fit within these constraints. This comparison between the development and deployment environments is shown in Fig. 20.

Development environments typically employ powerful GPUs, which consist of numerous parallel processing cores optimized for handling large datasets and performing repetitive operations efficiently. GPUs are well-suited for tasks like graphics rendering and, by extension, certain aspects of ML due to their parallel processing capabilities.

In contrast, deployed systems often have limited processing power and memory [32]. The MAX78000 is designed to address these constraints by integrating specialized hardware optimized for efficient ML inference.

The second significant challenge involves adapting the model to fit within the memory and processor constraints of the MAX78000. The MAX78000 CNN engine provides 512 kB of data memory and up to 442 kB of weight capacity. While sufficient for small-scale models like those used in keyword spotting or low-resolution image recognition, these limits pose challenges for deploying larger, more complex models.

In addition to these technical constraints, embedded systems must also contend with real-world factors such as time and power efficiency. Inference time, the duration required for

<figure><img src="image_23.png" /><figcaption>**FIGURE 21.** MAX78000 development flow.</figcaption></figure>

the model to process inputs and produce results, must be minimized to meet application requirements. While development systems can afford considerable energy consumption, edge devices often operate on battery power, with expectations of long operational lifespans, sometimes spanning months or even years [33]. Consequently, it is crucial to optimize the quantization for the eight-bit precision of the MAX78000, ensure the model fits within the chip's memory and processing capabilities, and fine-tune the network to balance inference speed with energy efficiency.

*So, how to program the CNN accelerator to execute tasks given the described constraints?* Fortunately, there is no need to learn a new programming language to instruct the 64 processors. The processors in the CNN accelerator function more like state machines that are configured and then set to operate autonomously, rather than stored-program computers. Surrounding logic manages memory pointers and counters, all controlled by a set of registers that are preconfigured before the CNN accelerator starts.

It is not necessary to directly access the CNN accelerator's registers, except for academic curiosity. After building and training the model, the trained weights, biases, and model architecture are processed through a synthesizer, referred to as an *izer*. This process involves passing a trained checkpoint file and a YAML model description through the *izer*, which is a Python script. The output includes a C-language file that can be compiled and loaded onto the MAX78000 platform, containing all the necessary code to configure and initiate the CNN accelerator.

The main takeaway is that while direct interaction with the CNN accelerator's configuration registers is not required, it is essential to provide the synthesizer with the model's architecture and instructions on how to fit the model into the MAX78000's constrained environment. This can be seen as part of the MAX78000 development flow shown in Fig. 21.

To transition from this understanding to practical application, we need to grasp three key concepts:
```python
def forward(self, x):
    """Forward prop"""
    # Run CNN
    x = self voice conv1(x)
    x = self voice conv2(x)
    x = self.drop(x)
    x = self McCormick conv3(x)
    x = self McCormick conv4(x)
    x = self.drop(x)
    x = self.kws conv1(x)
    x = self.kws conv2(x)
    x = self.drop(x)
    x = self.kws conv3(x)
    x = self.kws conv4(x)
    x = x.view(x.size(0), -1)
    y = self.fc(x)

    return y
```

Listing 1: KWS20 forward method extracted from `ai8x-training/models/ai85net-kws20-v3.py`.

* Building a Network with PyTorch<fnref n="5" />: Using the Python tools and the neural network library, PyTorch, to construct a network.

* Describing the Network with YAML: Detailing the network structure using the YAML markup language.

* Invoking the izer: Learning how to execute the izer, including specifying the input files, output files, and necessary options.

These concepts form the foundation for effectively programming and deploying neural networks on the MAX78000 platform.

In this section, we revisit the KWS20 network, which was introduced in Section III-E. This neural network was designed to detect spoken keywords from a set of 20 active keywords. The initial layers of the network create a spectral estimate of the waveform using a series of 1-D convolutions. This estimate is then reorganized into a 2-D tensor before the classification layers are applied. Fig. 22 shows an updated, more efficient version of the model where 1-D convolutional steps are used throughout the entire process.

## B. CREATING A NETWORK WITH PYTORCH

Having outlined the key considerations for developing AI models for the MAX78000, the next step involves the practical process of constructing such a model. PyTorch, a versatile deep learning framework, offers an efficient environment for building and training neural networks. In this subsection, we delve into how PyTorch can be used to create AI models designed specifically for deployment on the MAX78000.

Consider the snippet shown in Listing 1 from the model's defined PyTorch class. The `forward()` method accepts a tensor `x` containing 16,384 audio samples, representing one second of recorded audio. It performs extensive processing and ultimately returns the modified tensor `y`, now a vector of

21 values. These values correspond to the probabilities that the audio contains one of the 20 keywords, or that it contains no recognizable keywords.

The methods called within `forward()` are defined in the class's initialization, shown in Listing 2. To understand their function, let's examine them one by one. The input dimensions of the voice clip are 128 channels of 128 samples, arranged in rows. We call a method from the `ai8x` library named `FusedConv1dReLU()`. This method performs a 1-D convolution followed by a ReLU activation, fusing these operations into a single step. It is a one-element convolution: each output element is the weighted sum of the corresponding element across all input channels. The output consists of 100 channels, each containing 128 elements.

The second step also uses the `FusedConv1dReLU()` method, but with a three-element convolution kernel. This involves more extensive weighted summing. However, due to the kernel's size, we lose one element from each end of the data array, resulting in rows with only 126 elements each. Additionally, the number of output channels is reduced to 96.

Next, we perform another convolution, beginning with a max pooling operation. This method, `FusedMaxPoolConv1dReLU()`, simultaneously executes the max pooling, convolution, and activation operations. In this instance, the max pooling operation organizes the elements of each row into pairs and outputs a vector containing only the largest element of each pair, effectively reducing the row size to 63 elements. This reduced set of rows is then passed to the convolution engine. Normally, a convolution with a kernel size of three would reduce the row size by two, resulting in 61 elements. However, we apply padding, adding a zero to the front and back of the row to maintain the original 63 elements. This step produces 64 output channels.

The fourth convolution uses a three-element kernel without padding. As a result, the row output size is reduced by two, resulting in 61 elements. This step produces 48 output channels.

In the next step, we start with a max pool operation that discards the odd elements from the end of the row, reducing the size to 30 elements. These elements are then convolved using a three-element kernel with padding, which maintains the row size at 30 elements. The number of output channels is increased to 64.

In step six, a three-element convolution kernel is used without padding, which reduces the row size to 28. The number of output channels is increased to 96.

In step seven, average pooling is applied to reduce the row size from 28 to 14. This is followed by a three-element convolution kernel with padding, which maintains the row size at 14. The channel count is increased from 96 to 100.

In the final convolution step, starting with 14-element rows, a max pooling operation reduces the row size to 7 elements. Padding is applied, effectively increasing the row size to 9 elements. A six-element convolution kernel is then used.

To determine the number of output elements per row, consider the following: with 9 input elements (numbered

<footnote n="5">⁵https://pytorch.org/</footnote>
<figure><img src="image_24.png" /><figcaption>FIGURE 22. More efficient version of the KWS20 network using only 1-D convolutions.</figcaption></figure>

```python
class AI85KWS20Netv3 nn.Module):
    """
    Compound KWS20 v3 Audio net, all with Conv1Ds
    """
    # num_classes = n keywords + 1 unknown
    def __init__(
        self,
        num_classes=21,
        num bidirectional channels=128,
        dimensions=(128, 1), # p(y|X|D|W) = p(D|W) = p(W|D|Y) = p(Y|W|D)
        bias=False,
        **kwargs
    ):
        super().__init__()
        self.drop = nn，请 dropout(p=0.2)
        # Time: 128 Feature :128
        self.voice conv1 = ai8x.FusedConv1dReLU(num bidirectional channels, 100, 1, stride=1, padding=0,
                                                bias=bias, **kwargs)
        # T: 128 F: 100
        self.voice conv2 = ai8x.FusedConv1dReLU(100, 96, 3, stride=1, padding=0,
                                                bias=bias, **kwargs)
        # T: 126 F: 96
        self.voice conv3 = ai8x.FusedMaxPoolConv1dReLU(96, 64, 3, stride=1, padding=1,
                                                bias=bias, **kwargs)
        # T: 63 F: 64
        self.voice conv4 = ai8x.FusedConv1dReLU(64, 48, 3, stride=1, padding=0,
                                                bias=bias, **kwargs)
        # T: 61 F: 48
        self.kws conv1 = ai8x.FusedMaxPoolConv1dReLU(48, 64, 3, stride=1, padding=1,
                                                bias=bias, **kwargs)
        # T: 30 F: 64
        self.kws conv2 = ai8x.FusedConv1dReLU(64, 96, 3, stride=1, padding=0,
                                                bias=bias, **kwargs)
        # T: 28 F: 96
        self.kws conv3 = ai8x.FusedAvgPoolConv1dReLU(96, 100, 3, stride=1, padding=1,
                                                bias=bias, **kwargs)
        # T: 14 F: 100
        self.kws conv4 = ai8x.FusedMaxPoolConv1dReLU(100, 64, 6, stride=1, padding=1,
                                                bias=bias, **kwargs)
        # T: 4 F: 64
        self.fc = ai8x Linear(256, num_classes, bias=bias, wide=True, **kwargs)
```

Listing 2: KWS20 model initialization. Snippet extracted from `ai8x-training/models/ai85net-kws20-v3.py`
0 through 8) and a six-element kernel, the kernel slides across the input vector, covering positions as follows: 0–5, 1–6, 2–7, and 3–8. This results in 4 output elements. In this step, 64 output channels are computed.

Before concluding with the final operation, it is important to revisit the `forward()` method. After each pair of convolutional operations, the `drop()` method is called. This method performs a seemingly counterintuitive task: it randomly sets some values in the vector of intermediate results to zero based on a specified probability. The rationale behind this approach lies in mitigating overfitting, a common challenge in small neural networks. Overfitting, as presented in section III-D1, occurs when a model becomes too specialized to the training data, resulting in poor generalization to new, unseen data. By randomly dropping some features during training, the network learns to identify more general patterns and features within the dataset, rather than memorizing specific examples. Additionally, the `view()` method is invoked to rearrange the elements of a tensor. This method transforms a 2-D tensor, for example, with 64 rows and 4 elements per row, into a 1-D tensor, or potentially into another desired dimensionality. In this case, the `view()` method converts the tensor into a linear array of 256 elements.

In the final step, the 256 elements serve as inputs to a fully connected layer, which is a type of layer in which every neuron in the layer is connected to every neuron in the preceding layer, as well as to every neuron in the subsequent layer. The output of this layer comprises 21 classifications, corresponding to the keywords.

This concludes the model definition. An overview of the layers described is shown in Fig. 23. The specific numbers – such as the number of channels per layer and the dimensions of the tensors as they progress through the network – are somewhat arbitrary. The primary objective is to transform the incoming audio into a meaningful feature map through the initial layers, then to classify these feature maps into distinct commands using the final layers. The reason for our detailed exploration of the model architecture is to better understand how to create an accurate and comprehensive blueprint for the model, which is then used by the synthesizer to generate the necessary code for the MAX78000.

### C. DESCRIBING THE NETWORK WITH YAML

After defining the model, the next step is to train it using training data, a process that we describe in a later section. The training process produces a checkpoint file that includes all the weights computed during training and a record of the network architecture. This checkpoint file does not provide information on how to fit the model into a device like the MAX78000. Therefore, one needs a network description to instruct the synthesizer, found on the `ai8x-synthesis` repository, on resource allocation for each layer of the model. This is achieved by manually creating a YAML model description file. YAML was chosen over other file types for its superior readability and support for comments, making it easier to understand, maintain, and well-suited for configuration

```yaml
arch: ai85kws20netv3
dataset: KWS_20
layers:
  # Conv ID - 4 layers
  - pad: 0
    activate: ReLU
    out_offset: 0x2000
    processors: 0xfffffffffffffff
    data_format: HWC
    operation: Conv1d
    kernel_size: 1
  - pad: 0
    activate: ReLU
    out_offset: 0x0000
    processors: 0x0000FFFFFFFFFFFF
    operation: Conv1d
    kernel_size: 3
  - max_pool: 2
    pool_stride: 2
    pad: 1
    activate: ReLU
    out_offset: 0x2000
    processors: 0x0000FFFFFFFFFFFF
    operation: Conv1d
    kernel_size: 3
  - pad: 0
    activate: ReLU
    out_offset: 0x0000
    processors: 0xffffffffFFFFFFFF
    operation: Conv1d
    kernel_size: 3
  - max_pool: 2
    pool_stride: 2
    pad: 1
    activate: ReLU
    out_offset: 0x2000
    processors: 0x0000FFFFFFFFFFFF
    operation: Conv1d
    kernel_size: 3
  - pad: 0
    activate: ReLU
    out_offset: 0x0000
    processors: 0xffffffffFFFFFFFF
    operation: Conv1d
    kernel_size: 3
  - avg_pool: 2
    pool_stride: 2
    pad: 1
    activate: ReLU
    out_offset: 0x2000
    processors: 0x0000FFFFFFFFFFFF
    operation: Conv1d
    kernel_size: 3
  - max_pool: 2
    pool_stride: 2
    pad: 1
    activate: ReLU
    out_offset: 0x0000
    processors: 0x0000FFFFFFFFFFFF
    operation: Conv1d
    kernel_size: 6
  - flatten: true
    out_offset: 0x2000
    processors: 0xffffffffFFFFFFFF
    operation: MLP
    output_width: 32
    activate: None
```

<caption>Listing 3: KWS20 YAML description file extracted from `ai8x-synthesis/networks/kws20-v3-hwc.yaml`.</caption>
<figure><img src="image_25.png" /><figcaption>FIGURE 23. Overview of the KWS20 network layers.</figcaption></figure>

files. The YAML description file for the KWS20 model is given in Listing 3.

To understand the components of the YAML description file, we first need to identify the architecture (ai85kws20netv3) and the dataset (KWS_20) used to create the model. The architecture is defined in the Python class we reviewed, and the training dataset is established during training. These fields simply need to point to the relevant PyTorch components set up in the training tool.

Next, we delve into the layer definitions. It is useful to have the network definition side by side with the description for clarity. In the network description, the order of the items is not crucial as long as all necessary elements are included. Each layer in the YAML file starts with a dash, indicating the beginning of a new layer definition.

We'll begin with layer one. In the model definition, we call the method FusedConv1dReLU() and specify a kernel size of one. This layer has 128 input channels and generates 100 output channels. In the YAML description, we inform the izer that there is no padding and that ReLU is used for activation. The next line, out_offset, sets the relative address inside the data memory where the processor will write the layer's output. We set this to 0x2000. To avoid accidentally overwriting a layer's input data, we alternate the output address between 0 and 0x2000 throughout the model.

The next line instructs the CNN accelerator which of the 64 processors to use for the operation. Each bit in this field enables one of the 64 processors. Typically, a single processor computes a single input channel, making the setup straightforward for layers with 64 or fewer input channels. For example, in a layer with 16 input channels, setting processors: 0x0000000000000000 would enable all 16 processors within the first quadrant. Alternatively, setting processors: 0x000f000f000f000f would enable the first four processors in each quadrant, distributing the workload across quadrants, but making it less energy efficient.

In this network, however, the input layer consists of 128 input channels, which is double the number of processors available. To handle more input channels than processors, the

MAX78000 allows for multiple passes, storing intermediate results in RAM between passes. The number of processors to enable is calculated by dividing the number of inputs by the number of passes and rounding up to the next multiple of four if necessary, as shown in (1). As for which processors to enable, it's generally standard to enable the lower-order processors unless there's a specific reason to do otherwise.

$$ 
\begin{aligned}
n_passes &= \lfloor n\_inputs/64 \rfloor \\
n\_proc &= \lfloor n\_inputs/n\_passes/4 \rfloor * 4
\end{aligned}
\qquad (1) $$

In this particular model, we have 128 inputs. Dividing 128 by 64 gives us exactly two passes. Therefore, we will make two passes through the CNN accelerator, enabling all 64 processors for the operation.

The next line specifies the data format. The data is organized by number of rows (height), followed by number of columns (width), and finally by number of channels. The alternative arrangement is by channel number first, but that requires a different processor setup.

Following this, we specify the operation, which is a 1-D convolution, and the kernel size, which is one. With these details, you have provided the izer with all the necessary information about the input layer.

Let's examine layer two. This layer employs the same function as before, but with a few variations. The input and output channel counts differ, and this convolution uses a three-element kernel.

In the description file, these changes are reflected as follows. Firstly, the kernel size is modified from one to three. Secondly, a different number of processors is enabled to match the new channel requirements. For the number of processors in layer two, following our rules, we see that the first integer greater than 100 divided by 64 is two. The smallest multiple of four greater than 100 divided by two is 52. Therefore, we need to enable 52 processors, which is indicated in this line.

For the next convolution, it begins with a max pool operation, followed by a 1-D convolution with a kernel size of three and a one-element padding on each end. In the description, we see the two-element max pool, and we instruct the izer
to apply a two-element stride. The only other difference here is the number of processors, 48, which matches half of the 96 input channels.

The fourth convolution involves a kernel size of three with no padding. All processors are active because there are 64 input channels, allowing all computations to be completed in one pass.

The fifth convolution is similar, involving a max pool operation with a kernel size of three and 48 processors active to match the number of input channels.

The sixth convolution is straightforward, but in the seventh, we employ average pooling. The only notable detail is that we're back to two passes through the processor array, activating 48 processors to handle 96 input channels.

Finally, the eighth convolution is also straightforward, except for the larger kernel size.

Now we arrive at the fully connected layer. Recall that in the Python model definition, we used the `view()` method to restructure the tensor as a single array of 256 elements. In the MAX78000, this step is simplified. We simply instruct the device to flatten the data, removing any dimensionality and treating the entire dataset as a single string. This operation is defined as MLP, which stands for “Multi-Layer Perceptron”, meaning every input connects to every classification neuron in the output layer. Notably, we define the output width as 32, making this the only part of the model where the precision extends beyond 8 bits, up to 32 bits.

At this stage, we have quantized trained weights in a checkpoint file and a YAML file that provides the *izer* with all necessary details to generate the C code. The next step is to instruct the *izer* to proceed with its task.

## D. INVOKING THE *IZER*

The *izer* is executed through a Python script named `ai8xize.py`. To run the *izer*, you need to specify a directory for the generated files (`demos/`) and provide a meaningful prefix for naming the directory (`kws20`). You also need to indicate the location of the quantized checkpoint file<fnref n="6" /> (`trained/kws20_qat_best_q8.pth.tar`), the YAML network description file (`networks/kws20-v3-hwc.yaml`), and specify the target device (`MAX78000`). With this information, the `ai8xize.py` script will process the inputs and generate a C file ready for compilation and deployment on the MAX78000. This process typically completes within a few seconds. Calling the *izer* to generate the files needed for the KWS20 model would look like the command shown in Listing 4.

A few additional command-line flags can enhance the functionality of the `ai8xize` script. Setting the `-verbose` flag provides more detailed information about the synthesis process. The `-compact-data` flag directs the synthesizer to include a call to the C library function `memcpy` for loading sample data, optimizing both time and space in the generated

C file. Similarly, the `-mexpress` flag applies the same optimization for the weights. Finally, the `-softmax` flag activates a software function that converts the 32-bit output from the CNN accelerator to a 16-bit value, facilitating quicker determination of classification confidence.

Upon executing the script with these options, the result might seem understated: the script confirms the device, network description file, and trained weights file being used, and then returns to the command prompt. However, upon examining the directory created by the *izer*, you find all necessary files: the main C file for initializing the chip and invoking CNN functions, the `cnn.c` and `cnn.h` files containing the CNN functions, a Makefile and launch file for the Eclipse<fnref n="7" /> environment and GCC build system, as well as configuration files for Microsoft Visual Studio Code<fnref n="8" /> (VSCode).

Having explored the process of converting a trained checkpoint file into functional C code, the natural progression would be to load this code onto a MAX78000 and assess its operational efficacy. However, it is instructive at this point to revisit the preceding steps and examine the methodology behind obtaining the trained quantized checkpoint file.

## VI. TRAINING A NETWORK FOR THE MAX78000

With the essential components in place, including PyTorch's neural network classes and Analog Devices' extensions for the MAX78000, we have established a framework for constructing and synthesizing neural networks<fnref n="9" />. We have examined the procedures for generating a network description file from Python-based model definitions and running the *izer* to produce the C code necessary for deployment on the MAX78000. The remaining crucial step is training the network.

Training is a pivotal and often complex phase in the ML pipeline. During this process, gradient descent algorithms iteratively refine the network's parameters, which requires executing forward inference steps millions of times. This intensive computational task requires powerful hardware to ensure timely completion.

To manage computational demands and ensure efficient processing, the training data is divided into smaller subsets called batches. A batch is a fixed number of training examples that are processed together during a single forward and backward pass. The size of a batch, referred to as the batch size, is an important hyperparameter of the training process.

Another key hyperparameter is the learning rate, which determines the step size taken during each parameter update. A higher learning rate accelerates convergence but may cause instability or overshooting the optimal solution, while a lower learning rate ensures stability but requires more iterations to converge.

The training process itself presents several challenges, including determining the optimal convergence rate, deciding

<footnote n="6">⁶How to obtain this checkpoint file is explained in Section VI</footnote>

<footnote n="7">⁷https://eclipseide.org/</footnote>

<footnote n="8">⁸https://code.visualstudio.com/</footnote>

<footnote n="9">⁹Refer to https://github.com/analogdevicesinc/MaximAI_Documentation</footnote>
python ai8xize.py --verbose --test-dir demos --prefix kws20 --checkpoint-file trained/kws20_qat_best_q8.thp.tar --config-file networks/kws20-v3-hwc.yaml --device MAX78000 --compact-data --mexpress --softmax

<caption>Listing 4: Command to run the izer for a trained KWS20 model.</caption>

whether to use a static or adaptive learning rate, and selecting
appropriate batch sizes.

Moreover, the quality and quantity of the dataset play a
significant role in training efficacy [34]. Questions regarding
the necessary amount of data, the balance between training
and validation sets, and the sources for acquiring data must
be addressed. While not all of these aspects will be explored
in depth in this discussion, we will provide an overview of
how to initiate network training and offer some best practices
to guide the process.

To train a network like the KWS20 model used previously,
you need to assemble several key components. This model,
which we previously converted into C source files and a
network description, serves as our example for illustrating the
training process. To train a custom model, you will require the
following:

1) **Model Definition:** Use PyTorch's `nn.Module` to define your network architecture. For models intended to run on the MAX78000, incorporate Analog Devices' `ai8x` extensions to tailor the network for deployment on small embedded devices. The model definition for the KWS20 example is presented in Section V-B.

2) **Training Data:** This dataset is crucial for teaching the network how to recognize patterns and make predictions. Fortunately, you are not required to gather this data manually. Numerous publicly available datasets are suitable for various tasks, including image recognition, speech processing, and more. Depending on your specific use case, you can select an appropriate dataset or leverage existing ones to avoid the complexities of data collection.

3) **Test Data:** This dataset evaluates the network's performance and generalization capability. It is essential for validating the accuracy and effectiveness of the trained model. As with the training data, test datasets are often available from public sources or can be part of the same dataset used for training.

The development flow for a network intended for the
MAX78000 involves integrating these elements into the train-
ing process, refining the network through iterative training,
and preparing the model for deployment.

A. COLLECTING TRAINING AND TEST DATA

Researchers at Google have already collected a compre-
hensive dataset for training<fnref n="10" />, which includes recordings
of 35 keywords—20 relevant to specific applications and
15 intended for the “other” category. Fortunately, you do

not need to manually download this dataset; the training
script train.py available in the ai8x-training repos-
itory will handle the download and setup automatically via
a custom-designed data loader. Once downloaded and ex-
tracted, the dataset will be organized into folders containing
WAV files used for network training.

In total, the dataset comprises 99,720 files. These are 16-bit monaural linear PCM audio recordings sampled at 16 kHz. However, for the training process, these samples need to be converted into 8-bit format and organized into a 128 by 128 Python array. The training script facilitates this conversion by employing a data loader, which is essential for managing and processing the dataset. Analog Devices has developed specific data loader classes for different models, including the KWS20 model, and these can be found in the `ai8x-training/datasets` directory.

B. EXECUTING THE TRAINING SCRIPT

At this stage, we have a model, a dataset, and a data loader in
place, which means we are prepared to execute the training
script, with the command given in Listing 5. The command
line for initiating the training process may appear complex,
but we will explain each component for clarity.

Firstly, consider the `-epochs` flag. This parameter is used
to specify the number of training epochs, set here to 200. It is
important to recall that a training epoch refers to a single pass
through the entire dataset, which is partitioned into numerous
batches.

The subsequent parameter, `-optimizer`, specifies the
choice of optimizer, in this case, the Adam optimizer. The
term “optimizer” refers to the method employed to enhance
the gradient descent process. While stochastic gradient de-
scent is the default algorithm, Adam, uses a more sophisti-
cated second-order function to better estimate the necessary
adjustments after each mini-batch. This approach improves
upon the standard gradient descent by adapting the learning
rate based on estimates of first and second moments of the
gradients.

The next parameter, `-lr`, represents the learning rate and is set to 0.001 in this case. The learning rate dictates the magnitude of weight adjustments during training. For instance, if the gradient for a weight is 0.5, the weight will be adjusted by 0.5x0.001, resulting in an increase of 0.0005. Typically, the learning rate is not static throughout training; it often decreases over time to refine the model as it approaches the minimum error.

The `-compress` keyword refers to a learning rate sched-
ule file, named `schedule_kws20.yaml` in this context.
This file outlines how the learning rate should be adjusted
over the course of training. Initially, a higher learning rate

<footnote n="10">¹⁰https://www.tensorflow.org/datasets/community catalogue/huggingface/speech_commands</footnote>
```bash
python train.py --epochs 200 --optimizer Adam --lr 0.001 --deterministic --compress
policies/schedule_kws20.yaml --model ai85kws20netv3 --dataset KWS_20 --enable-tensorboard
--confusion --device MAX78000
```

<caption>Listing 5: Command to train the KWS20 model.</caption>

```yaml
lr_scheduler:
  trainingzielr:
    class: MultiStepLR
    milestones: [100, 140, 180, 190, 195]
    gamma: 0.5

policies:
  - lr_scheduler:
      instance_name: traininglr
      starting_epoch: 0
      ending_epoch: 200
      frequency: 1
```

<caption>Listing 6: KWS20 model compression schedule.</caption>

facilitates quicker progress by making larger weight adjustments. As training advances and approaches convergence, reducing the learning rate allows for finer adjustments, helping the model to settle more precisely into a minimum error region.

Here's a detailed review of the YAML compression schedule shown in Listing 6:

* **Definition:** The initial line specifies that we are defining a learning rate scheduler. This YAML file could potentially include other training cycle adjustments, such as pruning or quantization methods, which are not covered in this example.

* **Name:** The second line names the learning rate schedule `trainingzielr`. This name identifies the specific schedule being defined and allows it to be referenced later in the training script.

* **Scheduler Type:** The third line specifies that we are defining a multistep learning rate scheduler. While this example uses a multistep approach, other scheduler types such as linear, multiplicative, or exponential are also available. For a comprehensive list of schedulers, see the Neural Network Distiller documentation<fnref n="11" />.

* **Milestones:** The milestones parameter lists the epochs at which the learning rate will be adjusted: at epochs 100, 140, 180, 190, and finally at epoch 195. These milestones define the specific points in the training process where changes to the learning rate will occur.

* **Gamma:** The gamma parameter indicates that the learning rate will be multiplied by 0.5 at each milestone. This means that the learning rate will be halved at each of the specified milestones, allowing for finer adjustments as training progresses.

* **Execution:** Finally, the policies section specifies when to apply the defined scheduler. It states that the

`trainingzielr` scheduler should be activated from epoch 0 to epoch 200, and that it should be executed at every training epoch.

The remaining parameters in the command line configuration specify the name of the model being trained and the training dataset. Additionally, the `-enable-tensorboard` parameter instructs the training script to log information in a format that can be used by TensorBoard<fnref n="12" />, providing a means to visualize and monitor various training metrics. We will explore the output from TensorBoard shortly.

The configuration includes the `-confusion` keyword, which directs the system to generate a confusion matrix. This matrix provides a detailed visualization of the network's classification performance. For instance, in the case of the KWS20 network, the matrix displays labels along the vertical axis corresponding to the spoken word, and labels along the horizontal axis representing the network's classification.

The final parameter, `-device`, specifies that the training is intended for the MAX78000. With all configurations in place, the training process can begin. Our setup for this training includes a 2023 MacBook Air, equipped with an Apple M2 processor operating at a maximum of 3.5GHz, with an 8 core CPU, 10 core GPU and a 16 core neural engine. In this case, the training benefits from MPS acceleration on Apple Silicon<fnref n="13" />.

Examining the log file provides a clearer view of the training process. The log file begins with details about the Adam optimizer, revealing the arguments used in its configuration, as seen in Listing 7. This includes the learning rates, which are expected, but also the betas, which control how quickly the optimizer adapts by forgetting past states. Additionally, the eps parameter is included, which is a small constant added to avoid division by zero. These values are typical for the Adam optimizer, though they can be adjusted if needed.

The next section of the log file provides details about the datasets (Listing 8). After processing the dataset which includes augmenting it [35] and removing the silent audio files, the script reports a final training set of 316,395 files, a validation set of 29,943 files and a test set of 11,005 files. Remember, only the training set influences the network's weights, with the validation set being used periodically to assess the training progress. Finally, the test set is reserved for evaluating the trained model on entirely new samples after the completion of all training epochs.

Once the network training begins, the training log provides detailed records (Listing 8), with each entry reflecting the performance over approximately 10 mini-batches of 256 sam-

<footnote n="11">¹¹https://intellabs.github.io/distiller/schedule.html</footnote>

<footnote n="12">¹²https://www.tensorflow.org/tensorboard</footnote>

<footnote n="13">¹³https://developer.apple.com/metal/pytorch/</footnote>
```
2024-07-24 11:01:47,541 - Log file for this run:
  /ai8x-training/logs/2024.07.24-110147/2024.07.24-110147.log
2024-07-24 11:01:47,552 - Configuring device: MAX78000, simulate=False.
2024-07-24 11:01:47,621 -
2024-07-24 11:01:47,621 - ----------------------------------------------------------------
2024-07-24 11:01:47,621 - Logging to TensorBoard - remember to execute the server:
2024-07-24 11:01:47,621 - > tensorboard --logdir='./logs'
2024-07-24 11:01:47,621 -
2024-07-24 11:01:47,993 - Optimizer Type: <class 'torch.optim.adam.Adam'>
2024-07-24 11:01:47,993 - Optimizer Arg: {'lr': 0.001, 'betas': (0.9, 0.999), 'eps': 1e-08,
'weight_decay': 0.0001, 'amsgrad': False, 'maximize': False,
'foreach': None, 'capturable': False, 'differentiable': False,
'fused': None}
...
```

Listing 7: Extract from the KWS20 model training log found in `ai8x-training/logs`. This snippet corresponds to the beginning of the log.

```
...
2024-07-24 11:02:15,909 - Reading compression schedule from: policies/schedule_kws20.yaml
2024-07-24 11:02:16,030 - torch.compile() not available, using "eager" mode
2024-07-24 11:02:16,030 - Dataset sizes:
  training=316395
  validation=29943
  test=11005
2024-07-24 11:02:16,030 -
2024-07-24 11:02:16,040 - Training epoch: 316395 samples (256 per mini-batch, world size: 1)
2024-07-24 11:02:31,524 - Epoch:[0][10/1236] Overall Loss 3.044518 Objective Loss 3.044518
  LR 0.001000 Time 1.548320
2024-07-24 11:02:31,872 - Epoch:[0][20/1236] Overall Loss 3.044515 Objective Loss 3.044515
  LR 0.001000 Time 0.791543
2024-07-24 11:02:32,224 - Epoch:[0][30/1236] Overall Loss 3.044507 Objective Loss 3.044507
  LR 0.001000 Time 0.539415
2024-07-24 11:02:32,574 - Epoch:[0][40/1236] Overall Loss 3.044015 Objective Loss 3.044015
  LR 0.001000 Time 0.413321
2024-07-24 11:02:32,929 - Epoch:[0][50/1236] Overall Loss 3.043645 Objective Loss 3.043645
  LR 0.001000 Time 0.337754
...
```

Listing 8: Extract from the KWS20 model training log found in `ai8x-training/logs`. This snippet corresponds to the middle of the log.

ples each. During this period, the training script executes 10 gradient descent steps. The processing time for each set of 10 mini-batches ranges between 0.3 and 1.6 seconds on this computer.

### C. ANALYZING THE TRAINING PROCESS WITH TENSORBOARD

Additionally, since we enabled TensorBoard in the command line, we can now use it to analyze the training process. TensorBoard is a powerful utility that interprets the log output from the training session and presents it through an HTTP server accessible via any web browser. This tool transforms raw log data into interactive graphs and other visualizations, offering a more insightful view of the training dynamics than plain text output. The main TensorBoard page for this training run presents a comprehensive overview of the training metrics. It includes details on the learning rate, objective and overall loss, training time per step, and the Top-1 and Top-5 performance for the training set.

Top-1 refers to the percentage of test samples for which the model's highest probability prediction (the top prediction) is correct. In other words, it measures how often the model's most confident guess is the right one. Top-5 extends this idea by checking if the correct label is within the model's top five predictions for each test sample. However, the key plots of interest are the loss plot for the validation set, the Top-1 accuracy plot for the validation set, and the Top-5 accuracy plot for the validation set, all shown in Fig 24.

Let us examine the network's performance by focusing on the Top-1 accuracy plot. Initially, the plot reveals that early epochs yield low accuracy, but this can be addressed with further analysis. For instance, at the end of the first epoch, the network achieved roughly 35% accuracy, which improves to approximately 72% after 10 epochs. By the time the network reaches epoch 100, it has been training for about two hours and a half, and we observe that the accuracy stabilizes around
<figure><img src="image_26.png" /><figcaption>FIGURE 24. TensorBoard key plots of interest.</figcaption></figure>

80%.

The notable change occurs at epoch 100, when the learning rate is halved. This adjustment prompts a renewed upward trajectory in the network's performance. By epoch 140, performance continues to rise, albeit more gradually, reaching just over 82% accuracy. Another halving of the learning rate at epoch 180 pushes the accuracy closer to 84%. Subsequent reductions in the learning rate at epochs 190 and 195 yield further small improvements, ending in an overall accuracy of approximately 85%.

As observed, the rate of improvement diminishes as the network approaches higher accuracy levels, resulting in a smoother trend line. The entire training process extends over a period of about five hours.

Now, let us turn our attention to accuracy. TensorBoard provides a colorful graphical representation of the confusion matrix, which illustrates the model’s performance across different classes after each epoch. A few of these matrices can be seen in Fig. 25. Initially, after epoch one, the confusion matrix reveals a considerable distance from optimal performance, with lighter colors indicating greater inaccuracies. By epoch 102, just after reducing the learning rate, the confusion matrix exhibits more favorable colors, corresponding to an accuracy approaching 88%. Ultimately, after full training, the confusion matrix for the validation set shows that most word accuracies have improved, though some classes have seen minimal changes, and a few have even declined slightly.

Analyzing performance, the word “six” performs best with only 1.3% incorrect classifications. In contrast, the words “off” and “up” exhibit one of the highest error rates, around 11.9% and 7.3% incorrect, and are frequently confused with each other. Another point of interest is the network’s tendency to mistakenly classify some “other” words as one of the target keywords, occurring 27.6% of the time. This misclassification could be a potential issue depending on the application context. For instance, if keywords like “tree” are frequently

misinterpreted as “three”, it may suggest a need for more training data or model adjustments to enhance specificity. However, the model’s current performance might be acceptable for your particular use case, and any adjustments would depend on your specific requirements.

### D. QUANTIZATION

Once training is complete and the network parameters have been refined to minimize the loss function, the next step is preparing the model for deployment on the MAX78000. This involves quantization, a critical process for adapting the trained floating-point model to the hardware’s constraints.

The quantization process for the MAX78000 involves a two-stage training approach that combines conventional Floating-Point (FP) training with Quantization Aware Training (QAT). This method ensures that the model parameters are optimized for quantization, enhancing performance on the hardware.

* **Floating-Point Training:** The model first undergoes conventional FP training, storing the parameters as `checkpoint.pth.tar` and `best.pth.tar`, representing the last epoch and the best-performing model on the validation set, respectively. FP training continues until the model reaches a steady state, which is crucial for establishing a solid baseline.

* **Quantization Aware Training (QAT):** After a predefined number of epochs, QAT is initiated. QAT employs fake quantization, where model parameters remain stored as FP32, but the forward pass is executed after quantizing these parameters. This ensures that the overall loss includes the quantization effect, allowing the model to optimize for quantization during training. Parameters from this stage are stored as `qat_checkpoint.pth.tar` and `qat_best.pth.tar`.

QAT is enabled by default and controlled by a policy file,
<figure><img src="image_27.png" /><figcaption>FIGURE 25. Confusion matrices at different stages of training.</figcaption></figure>

```python
---
start_epoch: 30
weight_bits: 8
```

Listing 9: Default `qat_policy.yaml` file.

specified by the `-qat-policy` flag passed during training (Listing 5). A configuration file (`qat_policy.yaml`) determines the epoch at which QAT starts. Until this epoch, the FP32 models (`checkpoint.pth.tar` and `best.pth.tar`) are updated, and after this epoch, the QAT models (`qat_checkpoint.pth.tar` and `qat_best.pth.tar`) are updated. For final applications, `qat_best.pth.tar` is typically used due to its optimized performance. If no specific QAT policy file is provided during training (like in our case), the default policy stored in `policies/qat_policy.yaml` is used, where weights are quantized to 8-bits after 30 epochs.

The default policy file is shown in Listing 9. Key aspects of QAT include:

* **Start Epoch:** The epoch at which QAT begins is specified by `start_epoch`. After this epoch, training learns an additional parameter corresponding to a shift of the final sum of products. Proper selection of `start_epoch` is crucial. As a rule of thumb, set `start_epoch` to a high value (e.g., 1000) and adjust based on when the model stops learning during FP training.

* **Weight Bits:** The `weight_bits` parameter describes the number of bits available for weights.

All models need to be quantized using the `quantize.py` script from the `ai8x-synthesis` repository. QAT models generally yield better results than FP32 models, as quantization losses are minimized during training. The command to quantize the KWS20 model by running the `quantize.py` script, assuming it was trained with QAT using the default policy, is given in Listing 10. In this command, you first pass the PyTorch checkpoint file you want to quantize (`trained/qat_best.pth.tar`<fnref n="14" />), followed by the name you wish to give to the output checkpoint file for the quantized model (`trained/kws20_qat_best_q8.pth.tar`) and then specify the device.

QAT can be disabled by specifying `-qat-policy None` when training, which results in using post-training quantization. Although simpler, it typically results in lower performance compared to QAT. Post-training quantization involves clipping weights using a fixed scale factor. The optimal scale factor for simple clipping is highly model- and data-dependent. For post-training quantization, the input checkpoint to quantize.py is typically `best.pth.tar` or `checkpoint.pth.tar`.

## E. TRAINING CONSIDERATIONS

On a relatively recent Apple M2 processor, as mentioned in section VI-B, training this network took approximately 5 hours. In contrast, the same network with the same training set completed in just over an hour and a half on a properly configured system equipped with a supported GPU. From this comparison, three key conclusions can be drawn:

First, even without a GPU, a sufficiently fast computer can still accomplish significant work. While training times will be longer, it is entirely feasible to train and modify simpler models effectively. This approach allows for valuable experimentation and evaluation without an immediate investment in a dedicated training system. Just be prepared for extended training times.

Second, the presence of a GPU does not render results instantaneous. Unlike some microcontroller projects where compile times are mere seconds, neural network training involves substantial computation. However, this should not be a major concern if your model is reasonably close to being functional for your application; you can determine relatively quickly whether training will converge.

<footnote n="14">¹⁴This file was generated during training and manually copied over to the `ai8x-synthesis/trained` directory.</footnote>
```
python quantize.py trained/qat_best.pth.tar trained/kws20_qat_best_q8.pth.tar --device MAX78000
```

<caption>Listing 10: Command to quantize the KWS20 model.</caption>

Third, AI applications vary widely in complexity, from relatively simple tasks to highly intricate models. Some applications will require minimal development and fit comfortably within the chip's capabilities, leaving ample room for additional tasks. Others will demand extensive design, training, and testing iterations. For such complex cases, a faster machine becomes crucial to expedite the development process.

## F. DISTINGUISHING TRAINING FROM DEPLOYMENT

When working with AI models on hardware accelerators, it's crucial to separate the development phase, particularly the model training, from the deployment phase.

Training an AI model is a one-time or infrequent task that happens in a controlled environment using high-performance computing resources. The duration of training, which could take days or even weeks, is often less important compared to the overall quality and accuracy of the model that is ultimately produced.

Once the model is trained, it can be deployed to potentially millions of devices in the field. This deployment process is designed to be highly efficient, focusing on scalability and ease of distribution. The time and resources spent during training are justified by the ability to quickly and effectively download the trained model to numerous devices in production environments.

Embedded AI chips are optimized for this deployment phase, where it is able to run the pre-trained models with minimal energy consumption and high efficiency. The primary concerns during deployment are not about the training time, but about ensuring that the model runs reliably and quickly on the device, enabling real-time AI processing across a vast network of deployed devices.

By clearly distinguishing these phases, it becomes evident that the investment in training, no matter how resource-intensive, is a one-time effort. The true value is realized in the deployment phase, where the benefits of the trained model are scaled across many devices, enabling sophisticated AI-driven applications in the field.

## VII. DEPLOYING MODELS ON THE MAX78000

In the previous sections, we discussed training a model and applying QAT to optimize it for deployment. After training, we used the quantize.py script to transform the PyTorch checkpoint into a quantized model file. The next step involved synthesizing this quantized model into C source files using the izer, which generates the following files in the demos/kws20 in the ai8x-synthesis directory, as specified in Listing 4.

* Makefile*

<figure><img src="image_28.png" /><figcaption>FIGURE 26. MAX78000 Evaluation Kit.</figcaption></figure>

* cnn.c

* cnn.h

* weights.h

* log.txt

* main.c

* sampledata.h

* sampleoutput.h

* softmax.c

* modeluraunch

* .cproject

* .project

* .settings/*

* .vscode/*

These C files are essential for compiling and running a known-answer test on the MAX78000 using either VSCode or Eclipse. The application firmware is created by integrating the files generated into the C/C++ project via #include directives. Analog Devices provides a software development kit (Analog Devices MSDK<fnref n="15" />), containing all necessary components and configurations. This SDK includes peripheral drivers, examples, and instructions to facilitate application development.

Once the Analog Devices SDK is installed and the VSCode environment is set up, the sample project generated by the izer is ready to be compiled and loaded into the MAX78000¹⁶.

Analog Devices has developed two hardware platforms for this purpose: the MAX78000EVKIT<fnref n="17" /> and the

<footnote n="15">¹⁵https://github.com/analogdevicesinc/msdk</footnote>

<footnote n="16">¹⁶The process of installing the SDK as well as compiling and loading a project is detailed in https://analogdevicesinc.github.io/msdk/USERGUIDE/</footnote>

<footnote n="17">¹⁷https://www.analog.com/en/resources/evaluation-hardware-and-software/evaluation-boards-kits/max78000evkit.html</footnote>
<figure><img src="image_29.png" /><figcaption>FIGURE 27. MAX78000 Feather Board.</figcaption></figure>

*** CNN Inference Test kws20 ***

*** PASS ***

Classification results:

<table><tr><td>[ 23575 ]</td><td>-></td><td>Class 0: 5.6%</td></tr><tr><td>[ -34901 ]</td><td>-></td><td>Class 1: 0.3%</td></tr><tr><td>[ 7683 ]</td><td>-></td><td>Class 2: 2.8%</td></tr><tr><td>[ -34457 ]</td><td>-></td><td>Class 3: 0.3%</td></tr><tr><td>[ -12737 ]</td><td>-></td><td>Class 4: 1.4%</td></tr><tr><td>[ 32013 ]</td><td>-></td><td>Class 5: 5.6%</td></tr><tr><td>[ -25567 ]</td><td>-></td><td>Class 6: 0.7%</td></tr><tr><td>[ 4170 ]</td><td>-></td><td>Class 7: 2.8%</td></tr><tr><td>[ 42180 ]</td><td>-></td><td>Class 8: 11.1%</td></tr><tr><td>[ 51054 ]</td><td>-></td><td>Class 9: 11.1%</td></tr><tr><td>[ 10580 ]</td><td>-></td><td>Class 10: 2.8%</td></tr><tr><td>[ 3272 ]</td><td>-></td><td>Class 11: 2.8%</td></tr><tr><td>[ -29755 ]</td><td>-></td><td>Class 12: 0.7%</td></tr><tr><td>[ 76153 ]</td><td>-></td><td>Class 13: 44.6%</td></tr><tr><td>[ -25741 ]</td><td>-></td><td>Class 14: 0.7%</td></tr><tr><td>[ -86133 ]</td><td>-></td><td>Class 15: 0.0%</td></tr><tr><td>[ -25033 ]</td><td>-></td><td>Class 16: 0.7%</td></tr><tr><td>[ -53456 ]</td><td>-></td><td>Class 17: 0.2%</td></tr><tr><td>[ -47702 ]</td><td>-></td><td>Class 18: 0.2%</td></tr><tr><td>[ 8564 ]</td><td>-></td><td>Class 19: 2.8%</td></tr><tr><td>[ 14578 ]</td><td>-></td><td>Class 20: 2.8%</td></tr></table>

Listing 11: KWS20 known-answer test on a MAX78000 Evaluation Kit.

MAX78000FTHR<fnref n="18" />. The MAX78000EVKIT is shown in Fig. 26, while the MAX78000FTHR, a smaller feather form board, is depicted in Fig. 27. Both boards come equipped with a VGA camera and a microphone.

We load the known-answer test generated by default into a MAX78000 Evaluation Kit. By opening a serial terminal one can see the test results, obtaining what is shown in Listing 11. Furthermore, inference energy and speed measurements were taken using the EvKit's Power Monitor. The Power Monitor (PMON) consists of an MCU (MAX32625), a power measurement AFE, sense resistors, an LCD, and a dedicated USB port. The AFE measures the voltage and current at 1024 Hz of four supplies that power the MAX78000. The PMON firmware collects measurement data and displays it on the small LCD.

The PMON measures the following values for a CNN inference of the KWS20 model: energy transformed during the window (E), active measurement period (T), idle period power measurement (I), and active period power measurement (A). The measured results are given in Table 1.

<table><caption>TABLE 1. Measurements from the PMON.</caption><thead><tr><td>Measurement</td><td>Value</td></tr></thead><tbody><tr><td>E (uJ/inf)</td><td>113.00</td></tr><tr><td>T (ms/inf)</td><td>2.50</td></tr><tr><td>I (mW/inf)</td><td>8.70</td></tr><tr><td>A (mW/inf)</td><td>53.50</td></tr></tbody></table>

<table><caption>TABLE 2. Comparative Energy Efficiency and Latency Measurements.</caption><thead><tr><td>Device</td><td>Latency (ms/inf)</td><td>Energy (uJ/inf)</td></tr></thead><tbody><tr><td>MAX78000</td><td>3.5</td><td>251</td></tr><tr><td>ARM Cortex M4F</td><td>905</td><td>11,200</td></tr></tbody></table>

As seen in the results, the inference process on the MAX78000 is particularly efficient. After recording a 1-second audio sample, the model completes the inference process in approximately 2.5 ms. This quick inference time aligns with the original goal of real-time processing in embedded systems. By achieving such a fast inference, the MAX78000 enables immediate response to audio inputs.

## A. COMPARATIVE ENERGY EFFICIENCY ANALYSIS FROM PUBLISHED RESEARCH

For reference, we present the findings by Ulkar et al. [36] that compares the energy efficiency and operational latency between the MAX78000 and an ultra-low power ARM Cortex-M4F microprocessor using the same KWS20 model. The analysis highlights the substantial energy savings provided by the MAX78000. The findings from the study are summarized in Table 2.

The MAX78000 significantly outperforms the ARM Cortex-M4F in terms of both energy consumption and latency. The KWS model consumes over 40 times more energy on the ARM Cortex M4F than on the MAX78000, demonstrating the latter's energy efficiency, which is crucial for battery-operated devices. This difference underscores the advantage of using specialized hardware for deploying AI models at the edge.

## B. SCALABILITY CONSIDERATIONS

When scaling an AI model to recognize more than just 20 audio commands, for example, several important trade-offs and considerations must be kept in mind.

Currently, the model designed for 21 audio classes—including the “unknown” category—contains a total of 169,526 weights. If you increase the number of classes, even by just expanding the final layers from 21 to 41 classes, the total number of weights increases to 174,646. This demonstrates how even modest increases in the number of classes can lead to significant changes in the model’s size and complexity.

As you add more classes, the model needs more weights and biases in the hidden layers as well to maintain accuracy, which in turn requires more memory and could result in longer inference times. For example, doubling the number of

<footnote n="18">¹⁸https://www.analog.com/en/resources/evaluation-hardware-and-software/evaluation-boards-kits/max78000fthr.html</footnote>
classes might require an increase in the number of weights by approximately 1.5 to 2 times, depending on the additional classes' complexity and the overall architecture.

This increase in model size not only impacts memory but also affects processing time. While the MAX78000 is optimized for low-power AI tasks, expanding the model could lead to higher energy consumption and longer inference times. Therefore, it is important to consider the available memory, processing capabilities, and power constraints of your deployment environment.

When planning to scale, it's crucial to consider the potential need for additional layers in the network to capture the necessary features for accurate classification. This could further increase the model's size and computational demands. A good practice is to conduct incremental tests as you add more classes, ensuring that the model scales effectively without exceeding the hardware's limitations.

### C. TROUBLESHOOTING THE DEPLOYMENT PROCESS

Deploying AI models on the MAX78000 can involve challenges at various stages. This guide outlines common issues and practical troubleshooting strategies to streamline the process.

#### 1) Ensure the Network Fits within the MAX78000's Constraints

Start by verifying that your network architecture fits within the hardware limitations of the MAX78000. This includes ensuring the model's total number of weights, memory footprint, and layer depth are within the device's capabilities. The memory footprint refers to the total amount of memory required to store the model's parameters (weights and biases) and intermediate data generated during computations. If the model's memory footprint exceeds the device's capacity, it may fail to run or perform inefficiently. To address this, you can analyze the model's resource usage, reduce the number of layers, shrink dense layers, or apply pruning techniques. Testing smaller sections of the model independently can also help identify problematic layers or operations.

#### 2) Test the Model on the Hardware

Once the model is confirmed to fit within the device's constraints, testing it directly on the MAX78000 hardware is essential. Load the model onto the MAX78000 and run a series of inferences to see how it performs. Monitor key metrics such as inference time, memory usage, and energy consumption to ensure they align with your expectations.

#### 3) Unexpected Classification Results

If the model's accuracy drops or results are inconsistent, it may be due to quantization errors introduced during deployment. To address this, consider fine-tuning the model using QAT to reduce quantization errors, and adjusting quantization parameters to further balance precision with resource constraints. Ensuring that input data during deployment is

preprocessed identically to the training data, including scaling, normalization, and augmentation steps, is crucial for consistency.

#### 4) Iterative Testing and Refinement

Troubleshooting is often an iterative process. Beginning with a basic model that functions correctly and gradually increasing complexity helps isolate issues and ensures each model component is optimized for the MAX78000's architecture. When problems arise, reverting to a previously working configuration can be instrumental in identifying the source of the issue. Logging and profiling tools provide additional clarity by tracing errors and measuring performance bottlenecks.

### VIII. PRACTICAL EXAMPLES

Aside from the KWS model, this section lists several other IoT applications that leverage the capabilities of the MAX78000, from environmental monitoring to interactive systems.

#### A. BED: OBJECT DETECTION SYSTEM FOR EDGE DEVICES (BED)

BED [37] is a real-time object detection system specifically designed for edge devices using the MAX78000 deep neural network (DNN) accelerator. This system integrates a camera and an LCD display to perform and exhibit image detections directly on the device. BED is tailored for environments where traditional computing resources are limited, making it ideal for applications that require immediate processing and response, such as surveillance, autonomous navigation, or interactive systems where latency and power efficiency are critical. The system not only captures and processes images, but also displays the detection results in real-time, demonstrating the capabilities of edge computing in handling complex tasks like object detection with minimal latency and energy consumption.

#### B. INTERMITTENT INTELLIGENT CAMERA WITH LEO SENSOR-TO-SATELLITE CONNECTIVITY

Nardello et al. [38] leverage a novel dual-stage energy storage strategy, integrating intermittent power sources like energy harvesters with edge computing for environmental monitoring. Their setup involves a smart camera that processes data locally using the MAX78000, optimizing for low energy consumption and intermittent functionality. The camera is designed to activate a larger energy storage unit only during data transmission phases, which is essential for supporting its sub-GHz Low Earth Orbit (LEO) transmission capabilities. This approach not only enhances the resilience of the system to power fluctuations but also increases the throughput of processed images under ultra-low light conditions. Such a system is particularly beneficial for deployment in remote or inaccessible areas where continuous environmental monitoring is critical, yet traditional power and communication infrastructures are lacking.
### C. ULTRA-FAST EDGE COMPUTING APPROACH FOR HAND GESTURE CLASSIFICATION BASED ON EIT MEASUREMENTS

Mnif et al. [39] introduce an innovative solution for real-time robotic control through hand gesture recognition. It utilizes Electrical Impedance Tomography (EIT) to classify hand gestures in American Sign Language, a method chosen for its precision but challenged by its complexity. By deploying a lightweight CNN on an edge device with Tiny Machine Learning (TinyML) capabilities, the system achieves rapid inference and high energy efficiency, both crucial for real-time applications in resource-constrained environments. This edge computing approach allows for real-time interaction between humans and robots in various applications, including industrial automation and assistive technologies. The model's effectiveness is demonstrated by its ability to train in just 10.2 s and perform inference in 66 µs, achieving an accuracy rate of 89.37% across 10 sign language classes.

### D. OTHER IOT APPLICATIONS OF COMPLEX PATTERN AND SIGNAL PROCESSING AT THE EDGE

Complex pattern and signal processing expands the potential of IoT applications, particularly in fields that require initial data interpretation through sophisticated recognition systems. This is evident across both one-dimensional and two-dimensional data structures.

In one-dimensional scenarios, applications range from audio signal processing for speech recognition [40] to health-care monitoring through Electrocardiograms (ECG) and Photoplethysmograms (PPG) [41]. For instance, in speech recognition, AI models deployed on edge devices have shown impressive performance. A study by He et al. [42] demonstrated that an embedded English speech recognition system can achieve a recognition rate exceeding 90% when the vocabulary size is less than 100 words. In healthcare, edge processing of ECG signals enables real-time anomaly detection. Seitani-dis et al. [43] proposed a novel classifier based on a CNN, optimized for storage and computational complexity, to enable deployment on edge devices suitable for hospital emergency departments. Using the MIT-BIH arrhythmia database, the proposed CNN achieved an overall accuracy of 95.3%.

In two-dimensional fields, applications include image processing for medical diagnostics [44] and thermal imaging for industrial maintenance [45]. For medical diagnostics, edge processing of medical images offers the potential for faster, more accessible, and potentially more affordable analysis. A study focusing on melanoma detection [46] exemplifies this. This research developed a portable medical device based on a Raspberry Pi, implementing a deep learning model trained on the HAM10000 dataset for skin lesion image classification. The device achieved a 76% accuracy on the testing dataset, demonstrating its potential for melanoma detection in a real-time setting. In industrial maintenance, Oliveira et al. [47] developed an IoT system for detecting thermal anomalies in industrial machines using deep neural networks (DNNs)

with edge computing, achieving over 90% accuracy while significantly reducing data transmission to external servers.

Transitioning such complex recognition tasks to traditional microcontroller code is a considerable challenge due to the extensive rules required to cover all possible variations and exceptions. Leveraging neural networks on advanced hardware simplifies this process. These networks can learn to identify and classify patterns effectively, without the need for extensive manual programming, making them ideal for implementing robust and flexible IoT solutions in complex environments.

## IX. FUTURE PROSPECTS

The practical examples highlight the current capabilities, but the future holds even greater potential. In this section, we will explore the future prospects for embedded AI.

The landscape of IoT and embedded AI is evolving rapidly, and the need for devices that can support these advancements is becoming more critical. Microcontrollers with built-in AI capabilities are well-equipped to take advantage of these trends. As IoT applications continue to grow in complexity, several key trends and advancements will play a significant role in shaping their future.

One of the most transformative trends is the integration of federated learning into IoT systems. Federated learning allows distributed devices to collaboratively train machine learning models without sharing raw data, addressing privacy concerns and reducing the communication overhead associated with centralized training. By enabling on-device learning and periodic model updates, federated learning opens up possibilities for more adaptive and personalized IoT solutions. For instance, in healthcare, federated learning can be used to enhance diagnostic models by aggregating insights from multiple devices while maintaining patient privacy [48].

Another key trend is the development of embedded AI devices capable of supporting more complex neural networks. This capability goes hand in hand with the need for more memory to support the additional weights and biases required by these sophisticated models. As these devices become capable of handling advanced applications such as real-time object detection and natural language processing, we will see more sophisticated and responsive AI-driven solutions across various fields, including security, healthcare, and consumer electronics.

Energy efficiency will also continue to be a crucial focus. Innovations in energy harvesting and power management will be essential for extending the operational life of IoT devices, particularly those deployed in remote or difficult-to-access locations. Improved energy efficiency ensures that these devices can function reliably for longer periods without the need for frequent maintenance or battery replacements.

Finally, interoperability and standardization will be critical challenges to overcome. Ensuring that embedded AI devices can seamlessly integrate with legacy systems and that there are standardized methods for managing edge computing and
<table><caption>TABLE 3. Comparison of Features between MAX78000 and MAX78002.</caption><thead><tr><th></th><th>MAX78000</th><th>MAX78002</th></tr></thead><tbody><tr><td>Arm Cortex-M4 Speed</td><td>100 MHz</td><td>120 MHz</td></tr><tr><td>Flash</td><td>512 kB</td><td>2.5 MB</td></tr><tr><td>ROM</td><td>-</td><td>64 kB</td></tr><tr><td>SRAM</td><td>128 kB</td><td>384 kB</td></tr><tr><td>GPIO Pins</td><td>52</td><td>60</td></tr><tr><td>8-Bit Weight Capacity</td><td>442 kB</td><td>2 MB</td></tr><tr><td>CNN Data Memory</td><td>512 kB</td><td>1.3 MB</td></tr><tr><td>Max Input Image Size</td><td>1024x1024 pixels</td><td>2048x2048 pixels</td></tr><tr><td>Max Network Depth</td><td>64 layers</td><td>128 layers</td></tr></tbody></table>

data collection will be vital for the widespread adoption of these technologies.

## A. IMPROVEMENTS IN THE MAX78002

The MAX78002¹⁹ represents a significant upgrade over the MAX78000, incorporating a range of enhancements that address key limitations and expand the capabilities of edge AI applications. These advancements make the MAX78002 a more powerful and efficient solution for complex AI tasks.

One of the most notable improvements in the MAX78002 is the significant increase in memory capacity. The device now features 2M 8-bit weight capacity, up from 442k in the MAX78000, and 1.3 MB of dedicated CNN data memory. Furthermore, the maximum network depth that the MAX78002 can handle is 128 layers, up from 64 layers in the MAX78000. This allows the MAX78002 to support larger, more complex models with more weights and biases, enabling it to handle sophisticated AI tasks.

The MAX78002 also features a faster Arm Cortex-M4 processor, running at up to 120 MHz compared to the 100 MHz in the MAX78000. Additionally, the MAX78002's CNN accelerator can handle input image sizes up to 2048x2048 pixels. These enhancements allow for quicker inference times, making the device better suited for real-time applications that require rapid processing and decision-making. The enhancements seen in the MAX78002 are summarized in Table 3.

Given the advancements seen in the MAX78002, future iterations of MAX78000-like devices should focus on further increasing memory capacity. More memory allows for even larger and more complex AI models to be deployed at the edge, improving accuracy and enabling new applications that are currently beyond reach.

## X. CONCLUSION

This tutorial has demonstrated the development and deployment of AI models on the MAX78000, a microcontroller designed for embedded AI in resource-constrained IoT environments. By combining practical guidance with theoretical insights, this work bridges the gap between understanding AI concepts and implementing them effectively on ultra-low power devices.

The outcomes of this tutorial are significant. Readers gain a detailed understanding of the MAX78000's architecture,

including its CNN accelerator and energy-efficient design. Furthermore, through hands-on guidance, the tutorial equips readers with the practical skills needed to create, train, quantize, and deploy AI models tailored for the MAX78000, ensuring solutions are optimized for resource efficiency and real-time operation.

Moreover, the tutorial emphasizes tool proficiency by introducing the *ai8x-training* and *ai8x-synthesis* repositories, which streamline model optimization and deployment processes. This practical focus reduces development complexity, making advanced AI techniques accessible for embedded systems developers.

The real-world application examples further underscore the versatility of embedded AI, demonstrating its transformative potential across domains such as healthcare, industrial maintenance, and consumer electronics. By illustrating the practical impact of integrating AI into IoT systems, the tutorial highlights how these technologies can enhance decision-making, efficiency, and responsiveness in edge devices.

By making the MAX78000 accessible to both the machine learning and embedded systems communities, this tutorial contributes to the broader adoption of intelligent edge solutions. As IoT applications grow increasingly complex, the insights provided here will play a vital role in advancing embedded AI and making the intelligent edge a reality.

## XI. ACKNOWLEDGEMENT

This document is issued within the frame and for the purpose of the OpenSwarm project. This project has received funding from the European Union's Horizon Europe Framework Programme under Grant Agreement No. 101093046. Views and opinions expressed are however those of the author(s) only and the European Commission is not responsible for any use that may be made of the information it contains.

## REFERENCES

[1] M. Merenda, C. Porcaro, and D. Iero, "Edge machine learning for ai-enabled iot devices: A review," *Sensors*, vol. 20, no. 9, p. 2533, 2020.

[2] L. Kong, J. Tan, J. Huang, G. Chen, S. Wang, X. Jin, P. Zeng, M. Khan, and S. K. Das, "Edge-computing-driven internet of things: A survey," *ACM Computing Surveys*, vol. 55, no. 8, pp. 1-41, 2022.

[3] A. Reuther, P. Michaleas, M. Jones, V. Gadepally, S. Samsi, and J. Kepner, "Ai accelerator survey and trends," in *2021 IEEE High Performance Extreme Computing Conference (HPEC)*, pp. 1-9, IEEE, 2021.

[4] M. Capra, B. Bussolino, A. Marchisio, G. Masera, M. Martina, and M. Shafique, "Hardware and software optimizations for accelerating deep neural networks: Survey of current trends, challenges, and the road ahead," *IEEE Access*, vol. 8, pp. 225134-225180, 2020.

[5] M. Capra, B. Bussolino, A. Marchisio, M. Shafique, G. Masera, and M. Martina, "An updated survey of efficient hardware architectures for accelerating deep convolutional neural networks," *Future Internet*, vol. 12, no. 7, p. 113, 2020.

[6] F. A. Azevedo, L. R. Carvalho, L. T. Grinberg, J. M. Farfel, R. E. Ferretti, R. E. Leite, W. J. Filho, R. Lent, and S. Herculano-Houzel, "Equal numbers of neuronal and nonneuronal cells make the human brain an isometrically scaled-up primate brain," *Journal of Comparative Neurology*, vol. 513, no. 5, pp. 532-541, 2009.

[7] A. Abraham, "Artificial neural networks," *Handbook of measuring system design*, 2005.

[8] J. Stokes, *Inside the machine: an illustrated introduction to microprocessors and computer architecture*. No starch press, 2006.

<footnote n="19">¹⁹https://www.analog.com/en/products/max78002.html</footnote>
[9] J. P. Van Santen, R. Sproat, J. Olive, and J. Hirschberg, *Progress in speech synthesis*. Springer Science & Business Media, 2013.

[10] J. S. Suri, "Computer vision, pattern recognition and image processing in left ventricle segmentation: The last 50 years," *Pattern Analysis & Applications*, vol. 3, no. 3, pp. 209-242, 2000.

[11] O. A. Montesinos López, A. Montesinos López, and J. Crossa, "Fundamentals of artificial neural networks and deep learning," in *Multivariate statistical machine learning methods for genomic prediction*, pp. 379-425, Springer, 2022.

[12] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436-444, 2015.

[13] Z. Zhang and J. Li, "A review of artificial intelligence in embedded systems," Micromachines, vol. 14, no. 5, p. 897, 2023.

[14] W. Lin, A. Adetomi, and T. Arslan, "Low-power ultra-small edge ai accelerators for image recognition with convolution neural networks: Analysis and future directions," Electronics, vol. 10, no. 17, p. 2048, 2021.

[15] B. Bouyssounouse and J. Sifakis, *Embedded systems design: the ARTIST roadmap for research and development*, vol. 3436. Springer, 2005.

[16] S. R. Dubey, S. K. Singh, and B. B. Chaudhuri, "Activation functions in deep learning: A comprehensive survey and benchmark," Neurocomputing, vol. 503, pp. 92-108, 2022.

[17] A. Vabalas, E. Gowen, E. Poliakoff, and A. J. Casson, "Machine learning algorithm validation with a limited sample size," PloS one, vol. 14, no. 11, p. e0224365, 2019.

[18] D. Steinkraus, I. Buck, and P. Y. Simard, "Using gpus for machine learning algorithms," in *Eighth International Conference on Document Analysis and Recognition (ICDAR'05)*, pp. 1115-1120, IEEE, 2005.

[19] L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamaría, M. A. Fadhel, M. Al-Amidie, and L. Farhan, "Review of deep learning: concepts, cnn architectures, challenges, applications, future directions," Journal of big Data, vol. 8, pp. 1-74, 2021.

[20] S. Hijazi, R. Kumar, C. Rowen, et al., "Using convolutional neural networks for image recognition," Cadence Design Systems Inc.: San Jose, CA, USA, vol. 9, no. 1, 2015.

[21] N. Akhtar and U. Ragavendran, "Interpretation of intelligence in cnn-pooling processes: a methodological survey," Neural computing and applications, vol. 32, no. 3, pp. 879-898, 2020.

[22] C. Nwankpa, W. Ijomah, A. Gachagan, and S. Marshall, "Activation functions: Comparison of trends in practice and research for deep learning," arXiv preprint arXiv:1811.03378, 2018.

[23] M. G. Abdolrasol, S. S. Hussain, T. S. Ustun, M. R. Sarker, M. A. Hannan, R. Mohamed, J. A. Ali, S. Mekhilef, and A. Milad, "Artificial neural networks based optimization techniques: A review," Electronics, vol. 10, no. 21, p. 2689, 2021.

[24] X. Ying, "An overview of overfitting and its solutions," in *Journal of physics: Conference series*, vol. 1168, p. 022022, IOP Publishing, 2019.

[25] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas, "Learning to learn by gradient descent by gradient descent," Advances in neural information processing systems, vol. 29, 2016.

[26] P. Netrapalli, "Stochastic gradient descent and its variants in machine learning," Journal of the Indian Institute of Science, vol. 99, no. 2, pp. 201-213, 2019.

[27] G. Nakerst, J. Brennan, and M. Haque, "Gradient descent with momentum—to accelerate or to super-accelerate?," arXiv preprint arXiv:2001.06472, 2020.

[28] A. Lydia and S. Francis, "Adagrad—an optimizer for stochastic gradient descent," Int. J. Inf. Comput. Sci, vol. 6, no. 5, pp. 566-568, 2019.

[29] T. Tieleman, "Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude," COURSERA: Neural networks for machine learning, vol. 4, no. 2, p. 26, 2012.

[30] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.

[31] S. Gupta, J. Jaafar, W. W. Ahmad, and A. Bansal, "Feature extraction using mfcc," Signal & Image Processing: An International Journal, vol. 4, no. 4, pp. 101-108, 2013.

[32] F. Svoboda, J. Fernandez-Marques, E. Liberis, and N. D. Lane, "Deep learning on microcontrollers: A study on deployment costs and challenges," in *Proceedings of the 2nd European Workshop on Machine Learning and Systems*, pp. 54-63, 2022.

[33] X. Vilajosana, Q. Wang, F. Chraim, T. Watteyne, T. Chang, and K. S. Pister, "A Realistic Energy Consumption Model for TSCH Networks," IEEE Sensors Journal, vol. 14, no. 2, pp. 482-489, 2013.

[34] A. Jain, H. Patel, L. Nagalapatti, N. Gupta, S. Mehta, S. Guttula, S. Mujumdar, S. Afzal, R. Sharma Mittal, and V. Munigala, "Overview and importance of data quality for machine learning tasks," in *Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining*, pp. 3561-3562, 2020.

[35] T. DeVries and G. W. Taylor, "Dataset augmentation in feature space," arXiv preprint arXiv:1702.05538, 2017.

[36] M. G. Ulkar and O. E. Okman, "Ultra-low power keyword spotting at the edge," arXiv preprint arXiv:2111.04988, 2021.

[37] G. Wang, Z. P. Bhat, Z. Jiang, Y.-W. Chen, D. Zha, A. C. Reyes, A. Niktash, G. Ulkar, E. Okman, X. Cai, et al., "Bed: A real-time object detection system for edge devices," in *Proceedings of the 31st ACM International Conference on Information & Knowledge Management*, pp. 4994-4998, 2022.

[38] M. Nardello, L. Caronti, and D. Brunelli, "Intermittent intelligent camera with leo sensor-to-satellite connectivity," in *Proceedings of the 11th International Workshop on Energy Harvesting & Energy-Neutral Sensing Systems*, pp. 79-85, 2023.

[39] M. Mnif, S. Sahnoun, M. Kaaniche, B. B. Atitallah, A. Fakhfakh, and O. Kanoun, "Ultra-fast edge computing approach for hand gesture classification based on eit measurements," in *2024 IEEE International Symposium on Robotic and Sensors Environments (ROSE)*, pp. 1-7, IEEE, 2024.

[40] H. Purwins, B. Li, T. Virtanen, J. Schlüter, S.-Y. Chang, and T. Sainath, "Deep learning for audio signal processing," *IEEE Journal of Selected Topics in Signal Processing*, vol. 13, no. 2, pp. 206-219, 2019.

[41] M. H. Chowdhury, M. N. I. Shuzan, M. E. Chowdhury, Z. B. Mahbub, M. M. Uddin, A. Khandakar, and M. B. I. Reaz, "Estimating blood pressure from the photoplethysmogram signal and demographic features using machine learning techniques," Sensors, vol. 20, no. 11, p. 3127, 2020.

[42] L. He, G. Jin, and S.-B. Tsai, "Design and implementation of embedded real-time english speech recognition system based on big data analysis," *Mathematical Problems in Engineering*, vol. 2021, no. 1, p. 6561730, 2021.

[43] P. Seitanidis, J. Gialelis, G. Papaconstantinou, and A. Moschovas, "Identification of heart arrhythmias by utilizing a deep learning approach of the ecg signals on edge devices," Computers, vol. 11, no. 12, p. 176, 2022.

[44] B. J. Erickson, P. Korfiatis, Z. Akkus, and T. L. Kline, "Machine learning for medical imaging," *radiographics*, vol. 37, no. 2, pp. 505-515, 2017.

[45] O. Janssens, R. Van de Walle, M. Loccfuier, and S. Van Hoecke, "Deep learning for infrared thermal image based machine health monitoring," *IEEE/ASME Transactions on Mechatronics*, vol. 23, no. 1, pp. 151-159, 2017.

[46] A. A. Ali, R. E. Taha, R. Kaur, and S. M. Afifi, "Multi-class classification of melanoma on an edge device," in *2023 International Conference on Microelectronics (ICM)*, pp. 46-51, IEEE, 2023.

[47] V. M. Oliveira and A. H. Moreira, "Edge ai system using a thermal camera for industrial anomaly detection," in *International Summit Smart City 360°*, pp. 172-187, Springer, 2021.

[48] H. G. Abreha, M. Hayajneh, and M. A. Serhani, "Federated learning in edge computing: a systematic survey," Sensors, vol. 22, no. 2, p. 450, 2022.

**MARTINA BALBI** is a research engineer at Analog Devices and a member of the AIO research team at Inria Paris, specializing in the design and implementation of innovative networking solutions. Holding a Master's degree in Robotics and AI from IMT Atlantique in France (2022) and an earlier degree in Electrical Engineering from Universidad de la Republica in Uruguay (2020), she is currently pursuing a PhD focused on Embedded AI for Low-Power Wireless Networks. This academic journey reflects her commitment to bridging theoretical knowledge with practical applications, aiming to contribute significantly to the evolution of technology, particularly within the domain of efficient low-power wireless networks.
**LANCE DOHERTY** is a System Architect at Analog Devices, specializing in reliable wireless communication protocol development. Beginning his career at the Dust Networks startup, Lance helped define the transition to wireless in the Industrial space through the WirelessHART standard and the SmartMesh product line. He holds over 25 US patents focused on improving the efficiency and robustness of low-power communication and is now adapting these concepts to electric vehicles, including the world's first Wireless Battery Management System at Analog Devices. Lance received his PhD in EECS from the University of California, Berkeley (2004) and his B. Eng. in Engineering Physics from the University of Alberta, Canada (1998).

**THOMAS WATTEYNE** is an insatiable enthusiast of low-power wireless technologies. He holds a Senior Research Director position at Inria in Paris, where he leads the AIO research team that designs, models and builds networking solutions based on a variety of Internet-of-Things (IoT) standards. He is a Wireless System Architect at Analog Devices, the undisputed leader in supplying low power wireless mesh networking solutions for critical applications for industrial and beyond. Since 2023, Thomas is the scientific coordinator of the Horizon Europe OpenSwarm project. In 2019, he co-founded Wattson Elements, the company that develops the award-winning Falco marina management solution. Between 2013 and 2022, Thomas co-chaired the IETF 6TiSCH working group to standardize how to use IEEE 802.15.4e TSCH in IPv6-enabled mesh networks. He was a postdoctoral research lead in Prof. Kristofer Pister's team at the University of California, Berkeley. Between 2005 and 2008, he was a research engineer at France Telecom, Orange Labs. He holds a PhD in Computer Science (2008), an MSc in Networking (2005) and an MEng in Telecommunications (2005) from INSA Lyon, France.

...