## 3 Related Works  
### Bayesian Optimization  
Also known as Sequential Model Based (Bayesian) Optimization, BO is a sample efficient, accurate global optimization strategy for black box, multi-extremal functions, such as the NN training and validation process [14; 15; 16; 17]. It is the de-facto standard in AutoML [18] and, more recently, AutoTinyML [19]. It is also well adopted at the industrial level (e.g., Matlab ([[https://it.mathworks.com/help/stats/bayesian-optimization-algorithm.html](https://it.mathworks.com/help/stats/bayesian-optimization-algorithm.html)]([https://it.mathworks.com/help/stats/bayesian-optimization-algorithm.html](https://it.mathworks.com/help/stats/bayesian-optimization-algorithm.html)) (accessed on 30 November 2023))). The BO approach to parametric optimization originates from the work of [PERSON], who described the technique in a series of publications throughout the 1970s and 1980s [20; 21]. [PERSON] [14] describes BO as a philosophical approach to optimization problems, from which a family of different algorithms have stemmed over the years. An optimization \begin{table}
\begin{tabular}{c l} \hline \hline
**Codename** & **Requirement** \\ \hline R1 & The proposed solution shall search and design an ELM CNN FE type topology. \\ \hline R2 & The devised ELM CNN FE shall generate features that can be compared with those of a reference CNN which had been previously trained with backpropagation on a given dataset. \\ \hline R3 & The mean square error \((MSE)\) between the features mentioned in R2 shall be less than or equal to one. \\ \hline R4 & The proposed solution shall employ BO. \\ \hline R5 & The proposed solution shall test at least two surrogate functions. \\ \hline R6 & The proposed solution shall be tested on at least two network topology spaces. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: List of requirements to be fulfilled by the solution proposed in this work. problem is defined as the systematic search within a quantitative domain for a point \(x^{*}\) which globally maximizes an objective function \(f^{*}\), as shown in Equation (2):  
\[x^{*}\in\text{argmax}_{x\in X}f(x). \tag{2}\]  
In such contexts, BO approaches rely on probability and Bayesian inference [22] to reason about uncertain quantities within the optimization problem, including the objective function itself. Thus, the objective function is seen as a stochastic process. What is known and assumed about the objective function is encoded within a probabilistic prior which applies restrictions on the process itself. This prior will be iteratively shaped into a surrogate model, i.e., an approximation of the unknown objective function, and it is often (but not always) a Gaussian Process (GP). In the proposed work, both GP and Random Forest (RF) have been tested as surrogate functions. A GP is a stochastic process which extends the concept of multivariate Gaussian distribution to infinite dimensions. RF is an ensemble model defined as a set of decision trees [23], and has been proven to be more computationally efficient than GP [24], especially in contexts where some decision variables are discrete or conditional, and when the number of total decision variables is high (greater than 20). Examples of decision variables for NNs are: the learning rate (continuous); the number of hidden layers, and the number of neurons per each hidden layer (discrete); if any of the hidden layers are instantiated (conditional), etc. In such an example, the search space would be tree structured. At each iteration, an evaluation of the objective function is computed by sampling the prior distribution at a given point \(x\), which is chosen following the strategy that is defined by the acquisition function. The acquisition function defines the 'utility' of evaluating each point of the dominion, and each iteration's \(x\) will correspond to the acquisition function's global maximum, i.e., the most useful point to evaluate next [16].