# Strategies of Automated Machine Learning for Energy Sustainability in Green Artificial Intelligence
## 3 Hyperparameter Optimization Problem
### Bayesian Optimization HyperBand  
The Bayesian optimization HyperBand algorithm, which combines the strengths of Bayesian optimization and the HyperBand algorithm, is designed for HPO [12].  
BOHB's efficiency is derived from its integration of HyperBand's early stopping mechanism and Bayesian optimization. The latter allows for a more informed selection of hyperparameter configurations, thereby enhancing the exploration of the search space. This combination results in a robust and efficient algorithm. However, the performance of BOHB may be influenced by the quality of the initial surrogate model used in Bayesian optimization [12]. Furthermore, the computational overhead associated with Bayesian optimization can pose a limitation. The formalization of the algorithm is presented in Algorithm 2.  
```
1:Input: \(R\) (maximum resources per configuration), \(\eta\) (resource reduction factor)
2:Output: the best hyperparameter configuration
3:procedureHyperBand(\(R,\eta\))
4:\(s_{\max}\leftarrow\lfloor\log_{\eta}R\rfloor\)
5:\(B\leftarrow(s_{\max}+1)\cdot R\)
6:for\((s=s_{\max})\)down to \(0\)do
7:\(n\leftarrow\lceil\frac{B}{R}\cdot\eta^{s}/(s+1)\rceil\)
8:\(r\gets R\cdot\eta^{-s}\)
9:\(T\leftarrow\text{Sample}(n)\)
10:for\(i=0,\ldots,s\)do
11:\(n_{i}\leftarrow\lfloor n_{i}\cdot\eta^{-i}\rfloor\)
12:\(r_{i}\gets r\cdot\eta^{i}\)
13:\(L\leftarrow\text{Run}(T,r_{i})\)
14:\(T\leftarrow\text{Top}(\frac{\eta}{\eta},L)\)
15:endfor
16:endfor
17:return best configuration of \(T\)
18:endprocedure
```  
**Algorithm 2** HyperBand.  
In this context, \(R\) denotes the maximum number of resources that can be allocated to a single configuration, and \(\eta\) is a resource reduction factor. The objective function \(f\) measures the model's performance for a given hyperparameter configuration, while \(D\) is the density model used for sampling these configurations. The maximum number of iterations is represented by \(s_{\max}\), and \(B\) is the total resource budget. The number of configurations to test is given by \(n\), and \(r\) is the number of resources allocated to each configuration. The set of hyperparameter configurations is denoted by \(T\). The function \(\text{Sample}(n,D)\) selects \(n\) hyperparameter configurations from the distribution \(D\), and \(\text{Run}(T,r_{i},f)\) runs the model with the configurations in \(T\) and \(r_{i}\) resources, returning the results of the objective function \(f\). The function \(\text{Top}(k,L)\) selects the top \(k\) configurations from \(L\) based on a specific performance metric, and \(\text{Update}(D,L)\) updates the density model \(D\) with the results of the objective function \(L\).  
```
1:Input: \(R\) (maximum resources per configuration), \(\eta\) (resource reduction factor), \(f\) (objective function), \(D\) (density model)
2:Output: the best hyperparameter configuration
3:procedureBOHB(\(R,\eta,f,D\))
4:\(s_{\text{max}}\leftarrow\lfloor\log_{\eta}R\rfloor\)
5:\(B\leftarrow(s_{\text{max}}+1)\cdot R\)
6:for\((s=s_{\text{max}})\) down to 0 do
7:\(n\leftarrow\lceil\frac{B}{R}\cdot\eta^{s}/(s+1)\rceil\)
8:\(r\gets R\cdot\eta^{-s}\)
9:\(T\leftarrow\text{Sample}(n,D)\)
10:for\(i=0,\ldots,s\)do
11:\(n_{i}\leftarrow\lfloor n\cdot\eta^{-i}\rfloor\)
12:\(r_{i}\gets r\cdot\eta^{i}\)
13:\(L\leftarrow\text{Run}(T,r_{i},f)\)
14:\(T\leftarrow\text{Top}(\frac{n_{i}}{\eta},L)\)
15:\(D\leftarrow\text{Update}(D,L)\)
16:endfor
17:endfor
18:return best configuration of \(T\)
19:endprocedure
```  
**Algorithm 2** Bayesian optimization HyperBand.