## 4 Experiments
### Hyperparameter Tuning  
Tuning hyperparameters involves the procedure that determines the network configurations' values, contributing to providing more accurate classification accuracy [81]. It highly influences machine learning algorithms. Tuning hyperparameters are typically done manually (trial and error). Nevertheless, this approach takes time to test all the possible hyperparameters that may result in an acceptable performance. Other approaches, such as grid searches [82; 83], genetic algorithm [84], and particle swarm optimization [85; 86], are widely used as hyperparameters tuning methods. Grid and random search divide the hyperparameter values into several folds to build a certain space range of grid search space. Then, it crosses over all grid points to find the best hyperparameters values. It requires that many hyperparameters be tested routinely by automatically re-training the model for each hyperparameter value [82], which makes it useful by mapping the problem space and giving more optimization possibilities. However, these methods and metaheuristic algorithms are restricted to a large-scale network. They also take time because experimental methods require several experiments to estimate the value. Many researchers are attracted to BO as an efficient method for tuning networks hyperparameters [87]. Using a small number of samples can obtain the values of the optimum hyperparameters for optimization, unlike traditional optimization methods [88]. It does not need the functions of explicit expression [89]. BO is a very effective in solving this kind of optimization problem [87; 90]. Nonetheless, for hyperparameter tuning in deep neural networks, the time required to evaluate the validation error for even a few hyperparameter settings remains a bottleneck. BO is slightly more sophisticated compared to manual tuning since it combines prior information about the unknown function with sample information to obtain posterior information of the function distribution. Then, based on this posterior information, we can deduce where the function obtains the optimal value [91]. In other words, It can be useful to model-based approaches for automatically configuring hyperparameters to generate a surrogate model of some unknown function that would otherwise be too expensive. In BO, rather than considering the objective function as a black-box about which we can only obtain point-wise information, regularity assumptions are made. These are used to actively learn a model of the objective function. The resulting algorithms are practical and provably find the global optimum of the objective function while evaluating the function at only few parameters [92]. These hyperparameters include the number of filters, kernels, optimizer, momentum, learning rate, batch size, and dropout rate. In experiments, the initial configurations of hyperparameters are randomly generated with shuffled training and validation data. Bayesian optimization searches for the best hyperparameters and fitting a new model of the best \(k\)-cv selection. For each point in the search space, Bayesian optimization cross-validation provides cross-validation estimates of performance statistics. Different data split between folds may generate different optimal tuning parameters. The tuning hyperparameters were then selected with the lowest average cross-validation error. We refer to it as the optimal cross-validation choice for tuning hyperparameters. A detailed list of hyperparameters and search bounds are shown in Table 2. For the neural network training, we trained it through a variant of adaptive optimizers like Adam, RMSprop, AdaBound, and EAG [93] with initial learning rate of \(10^{-4}\), \(\beta_{1}=0.90\) and \(\beta_{2}=0.99\), \(filter=128\), and \(kernal=2\). We used an initial value of 0.25 as dropout probability in the ninth layer to prevent network overfitting at the training stage. The batch size is set to a value in the range of (4 and 128), while all weight parameters are subjected to L2 regularization.