## 6 Discussion and Conclusions  
This work proposed an approach to building an ELM CNN FE based on BO and NAS. Two search spaces were defined and tested, as well as two datasets and two different surrogate models. The output of each search iteration was an FE topology which was fed with image data, and an extracted set of feature vectors as the output. The datasets used were CIFAR-10 and MNIST. The output was then compared to a reference feature set, obtained by a trained FE based on the MLCommons Tiny benchmark ResNet model. The best result in terms of similarity to the reference feature set achieved an \(MSE\) of 0.64 for the CIFAR-10 dataset, and 1.468 for the MNIST dataset. This outcome was still not enough to achieve a high classification accuracy using the FE as part of an ODL model; a TinyRCE classifier fed by the obtained features scored a 37.4% test set accuracy, 41.1% less than what the same classifier scored when fed with the reference features. Despite the unsatisfactory results collected so far, the experimental approach itself proposes an innovative integration of the NAS and BO techniques for the purpose of automatically designing a well-performing ELM FE, as well as a simple yet effective evaluation procedure that allows us to assess the quality of an FE without the necessity of training a learning agent connected to it. An automatically designed ELM FE becomes a fundamental building block for an NN to be deployed in ODL applications. Such an application would save all the memory required to store the activations and dataset needed to support backpropagation-based learning, which can eventually be reduced even further by weight selection techniques. This would lower the costs of inference workloads enough to satisfy what is required by ODL in tiny edge devices. Thus, the present study lays a solid groundwork for future improvements to the proposed approach. In conclusion, this work serves as an initializer and catalyst for future endeavors, inviting researchers to explore and build upon the theoretical framework \begin{table}
\begin{tabular}{c c c} \hline \hline
**Scope** & **Variable** & **Values** \\ \hline Whole network & Number of blocks & 3 \\ \hline Convolutional blocks & Padding type & ’same’, ’same’, ’valid’. \\ \hline Convolutional blocks & Kernel size & 3, 7, 3. \\ \hline Convolutional blocks & Number of filters & 16, 16, 64. \\ \hline Convolutional blocks & L2 Regularization & False, False, True. \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Variable settings for best found topology (MNIST dataset). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **Search Space Topology** & **Surrogate Model** & **Optimal _MSE_ Found** \\ \hline CIFAR-10 & Type 1 & GP & 216.128 \\ \hline CIFAR-10 & Type 1 & RF & **0.640** \\ \hline CIFAR-10 & Type 2 & RF & 8196.919 \\ \hline MNIST & Type 1 & GP & 2.537 \\ \hline MNIST & Type 1 & RF & 3.778 \\ \hline MNIST & Type 2 & RF & **1.468** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Summary of obtained results. Best _MSE_ results for each dataset are highlighted in bold. Lower is better. presented herein. Future experimentation will focus on improving the search outcomes; in particular, the target \(MSE\) to be achieved shall be lowered by at least one order of magnitude (\(10^{-2}\)). The correlation between the achieved \(MSE\) and the quality of the extracted features has not been studied enough; future investigations are required in this regard. Additionally, different metrics of feature quality could be experimented with. To improve the results of the search, new atomic block structures (e.g., based on attention layers, recurrent neural layers and many more) should be devised and tested, as well as different surrogate models and acquisition functions. Furthermore, the experimental approach should be tested on a greater variety of datasets, to better assess its generalizability. The datasets will include image recognition ones--e.g., CIFAR-100, MSCOCO14--and audio recognition ones such as Speech Commands. Conceptualization, A.P.