## 3 Experiments and Results
### State-of-the-Art Models  
In order to measure the performance of state-of-the-art NN architectures and show their limitations on hyper-constrained devices, we start our experiments considering well-known NN architectures. More in details, we consider the following highly-popular NNs:  
* _MobileNet-V2_--as one of the most light-weight NN architecture available, it was picked as the simplest baseline
* _ResNet-50_--being one of the most representative NN architecture, it was selected to show possible run-time issues
* _VGG-16_--since it represents one of the heaviest NN architecture, it was chosen so as to stress issues with embedded devices  
The selected models were trained via standard Stochastic Gradient Descent over the train split of the dataset collected in Section 2.2. Training was completed on a general laptop machine. Computational limitations of the case-study board makes it infeasible to run the training procedure directly on the microcontroller. Trained NNs were then converted to _TensorFlowLite_ ([[https://www.tensorflow.org/lite/microcontrollers](https://www.tensorflow.org/lite/microcontrollers)]([https://www.tensorflow.org/lite/microcontrollers](https://www.tensorflow.org/lite/microcontrollers)), accessed on 14 November 2021) architectures and deployed on the ARM-based board. The conversion process is required, since the board runs the _MicroPython_ operating system ([[http://micropython.org/](http://micropython.org/)]([http://micropython.org/](http://micropython.org/)), accessed on 14 November 2021) which itself supports _TensorFlowLite_ architectures only. Inference over the testing dataset was run on the OpenMV microcontroller and performance--both from accuracy and speed perspective--are shown in Table 4.  
Interestingly enough, the ResNet-50 and VGG-16 models show the common issues arising when dealing with hyper-constrained devices. Inference for those models cannot terminate on the board due to memory issues. The reason behind the behaviour is to be found on their architectures, which rely on fairly big convolutional filters whose footprint exceeds the 32 MBs available on the microcontroller. Results in Table 4 also shows how state-of-the-art NNs overfit the learning task. This behaviour was expected and is highlighted by the poor performances that those models achieve over the testing set (testing set accuracy for ResNet-50 and VGG-16 was obtained on the training machine). Overfitting issues are due to the complexity of state-of-the-art models and the simplicity of the dataset at hand. Finally, results show that state-of-the-art models are not applicable to embedded hyper-constrained devices as they are characterised by unbearable inference times. Indeed, the only successfully running model is the MobileNet-V2 model which runs a single inference step in 12.35 s. Such measure is far from the theoretical value obtained in Section 2.1 and unsatisfactory for any applicative scenarios.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & **Memory Footprint** & **Accuracy** & **Inference Time** \\ \hline MobileNet-V2 & 8.6 MB & 29.40\% & 12.35 s \\ ResNet-50 & 37.1 MB & 30.66\% & ✗ \\ VGG-16 & 115.3 MB & 28.55\% & ✗ \\ \hline \hline \end{tabular}
\end{table}
Table 4: SOTA models performance when deployed on the OpenMV microcontroller.