## 3 Background and Methods
### Differential Evolution  
Differential Evolution (DE) is one of the most popular population-based meta-heuristic algorithms, introduced by [PERSON] and [PERSON] in 1997 [39], and is considered as one of the most appropriate algorithms for continuous optimization. Besides the general popularity of DE algorithm for solving optimization tasks, it was also successfully applied to various ML problems such as HPO problem [40], problem of designing neural network architecture [41] or feature selection problem [42].  
The DE algorithm is composed of _Np_ real-coded vectors and three operators: mutation, crossover, and selection. The _Np_ real-coded vectors are representing the candidate solutions (individuals) which can be formally defined as presented in Equation (2), where each element of the solution is in the interval \(x_{i,1}^{(t)}\in[x_{i}^{(L)},x_{i}^{(l)}]\), while \(x_{i}^{(L)}\) and \(x_{i}^{(U)}\) denote the lower and upper bounds of the \(i\)th variable, respectively.  
\[\mathbf{x}_{i}^{(t)}=(x_{i,1}^{(t)},\ldots,x_{i,n}^{(t)}),\quad\text{for }i=1, \ldots,\textit{Np}, \tag{2}\]DE's basic strategy consists of mutation, crossover, and selection operations. The mutation operation can be formally expressed as follows:  
\[\mathbf{u}_{i}^{(t)}=\mathbf{x}_{r_{1}}^{(t)}+F\cdot(\mathbf{x}_{r_{2}}^{(t)}- \mathbf{x}_{r_{3}}^{(t)}),\ \ \ \text{for}\ i=1,\ldots,Np, \tag{3}\]  
where \(F\) represents the scaling factor as a positive real number that scales the rate of modification while \(r_{1}\), \(r_{2}\), and \(r_{3}\) are randomly selected values in the interval \(1\ldots Np\).  
To increase the diversity of the parameter vectors, the crossover operator is introduced as presented in Equation (4), where \(CR\in[0.0,1.0]\) controls the fraction of parameters that are copied to the trial solution.  
\[w_{i,j}^{(t+1)}=\begin{cases}u_{ij}^{(t)}&\text{rand}_{j}(0,1)\leq CR\lor j=j_{ mnd},\\ x_{i,j}^{(t)}&\text{otherwise},\end{cases} \tag{4}\]  
Finally, the selection operator is utilized to decide whether or not the produced vector should become a member of generation utilizing the greedy criterion. The selection could be formally expressed as follows:  
\[\mathbf{x}_{i}^{(t+1)}=\begin{cases}\mathbf{w}_{i}^{(t)}&\text{if}\ f(\mathbf{ w}_{i}^{(t)})\leq f(\mathbf{x}_{i}^{(t)}),\\ \mathbf{x}_{i}^{(t)}&\text{otherwise}\,.\end{cases} \tag{5}\]  
where \(f(\mathbf{x}_{i}^{(t)})\) denotes the fitness function defined for solving specific optimization problem. The DE algorithm works in an iterative manner, where each produced solution is evaluated using the given fitness function. Based on the selection operator, presented in Equation (5), the DE algorithm's search mechanism seeks for solutions of ever better fitness scores. Naturally, the DE algorithm is optimized for solving the minimization optimization problems and therefore the fitness function should be tailored to give the lower score to the preferred solutions.  
In order for a DE algorithm to yield good performance, the DE parameters, namely \(Np\), \(F\), and \(CR\), need to be set appropriately. The \(Np\) parameter, as presented, denotes the population size of real-coded vectors (individuals) on top of which the mutation, crossover, and selection operators are applied. If the population size is too small, it may lead to premature convergence and stagnation, while a too big population slows down the search by wasting time on too many unpromising moves [43]. The \(F\) parameter denotes the differential weight that determines how far from point \(x_{i}\) the offspring should be generated [44]. The \(CR\) parameter is called crossover probability and controls how many parameters in expectation are changed in a population member [45].