## 4 Results and Discussion
### Bayesian Optimization Experiments Results
In fact, an augmentation of the epoch number tended to generate some overfitting phenomena observed with huge validation losses for experiments B3 and B5, as shown in Table 4. In addition, the best segmentation performances seem to be reached with an overall learning rate coefficient alpha of \(1\times 10^{-4}\). The RMSprop optimizer configuration represents 39% of the BO trials. It counts around 24% of hyperparameter configurations with a validation IoU above 0.5. However, it records 24% of discarded experiments due to the largest exploration of the hyperparameter range. We clearly observed a drop in segmentation performance with batch size in this group. The batch size and epoch number seem to follow a linear trend with the runtime and confirm the same observed effect on the two other groups. Thus, the RMSprop optimizer  
\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**ID**} & **Total** & **Total** & **Total** & **Total** & **Total** & **F1-score** & **Optimizer** & **Epochs** & **Batch Size** & **\(\alpha\)** & **Runtime** & **Early Stops** & **Val IoU** & **Val F1** & **Loss** & **Val Loss** \\ \hline A1 & & & & & & & & & & & & & & \\ A2 & & & & & & & & & & & & & \\ A3 & 5 & 1 & 20\% & & & & & & & & & & \\ A4 & & & & & & & &  
tends to be more stable than Adam in regard to overfitting when the training time increases. Finally, the RMSprop optimizer achieved the overall best performance, with an IoU over 0.52 and an F1-score of 0.68 with a quick training time (experiment C9, Table 4). Thanks to this qualitative study, we could formulate the following rules about the observed effects of hyperparameter range and optimizer choice for the semantic segmentation of DGW images:  
* The batch size and epoch number have a linear effect on the runtime. For the three studied optimizers, increasing the batch size and epoch number by 50% led to an augmentation of 50% of the training time. * Increasing the batch size over a value of 4 with a high epoch number led to a significative reduction in segmentation performance. * The Adam optimizer tends to be more sensitive to overfitting with large epoch numbers. * Decreasing the learning rate coefficient alpha in conjunction with the epoch number exhibits the best overall performances for Adamax and Adam optimizers. * With RMSprop optimizer, no evident patterns are seen for fine-tuning the learning rate alpha. * Adam and RMSprop optimizers exhibit better segmentation performances with minimal training time compared to Adamax. * The overall best configuration is reached by RMSprop optimizer with a reduced training time of 25% compared to Adam for equivalent segmentation performances. Finally, the best hypermeter configurations found by our BO optimization strategy are summarized in Table 5.  
Finally, it is of importance to highlight some limitations of using pre-trained ResNet34 backbone weights on the encoder part of the network. In fact, although transfer learning techniques allow us to train U-Net deep learning architecture on a small dataset, it prevents us from fully using our AutoML framework. Indeed, input images must be 256 \(\times\) 256 \(\times\) 3 due to transfer learning usage, which leads to a lack of flexibility in regard to input images size. In addition, we are not able to fully leverage the power of Bayesian optimization algorithms for neural architecture search (number of CNN layers, size of convolution kernel and so on) due to the inherent constraints of using the ResNet34 weights on the encoder part.