## 5 Model Architectures and Training
### Model Training and Optimization
#### 5.2.2 Structural Optimization  
An extensive ablation study of the proposed architectures is performed with the intention to adapt the architectures to the problem setting and therefore optimizing performance and model size with respect to the task. For this, the Bayesian-based Tree-structured Parzen Estimator (TPE) [60] is utilized, in order to search for each model's optimal configuration in a corresponding hyperparameter space (see Tables A1 and A2 in Appendix A). The configuration of the original model is listed next to the partly discrete and continuous search ranges and acts as upward estimation in terms of complexity, since the the input data are of lower dimension and the prediction task is assumed to be less complex compared to benchmark applications. The experimental setup aims to conduct approximately 100 trials per model, irrespective of the corresponding size of its hyperparameter space. Opposed to the final training, the models are trained for 30 epochs per trial and evaluated on the validation data. This configuration strikes a balance between computational resources and the number of trials, minimizing memory usage while providing a sufficient learning window to assess model performance effectively.  
The circa 100 resulting model configurations per architecture are evaluated with respect to their classification and regression accuracy versus their model size (see Figure 12). Since model size and accuracy are typically conflicting optimization goals, a Pareto analysis is conducted. In order to do so, a scatter plot is generated, in which each point represents one resulting model configuration, marking the accuracy and MSE over the according number of model parameters. For the sake of clarity, instead of all trials, only the resulting Pareto front in the form of the convex hull over the best model configurations is visualized for each architecture. The closer the classification accuracy curves are to the upper left corner, the better the model performance is with respect to size and accuracy. In case of regression performance (in terms of the MSE metric), the same applies to the lower left corner. To emphasize optimizations and adaptations with respect to the problem setting, the original names of the model architectures are extended with the suffix \"Precipitation Rate Estimation and Classification\" (PREC).  
Looking at the classification performance, one can clearly see that the ViT models are outperforming the convolution-based ResNet as well as the Conv-LSTM model. Especially, the ViT-S architecture is showing excellent performance, with up to over 89 % classification accuracy on the validation dataset with a fairly small model size of fewer than 1 M parameters. Opposed to that, the Conv-LSTM was only able to increase performance with an unacceptable size of greater than 5 M parameters.  
A similar result can be observed for the regression performance. Here, the Conv-LSTM-PREC and ResNet3D-PREC model, however, showed massive problems in learning a meaningful correlation at all. Due to this, neither of the models are illustrated and were not further considered for the regression task.  
Figure 12: Pareto study on the classification and regression accuracy with respect to the model size for each investigated DL architecture during hyperparameter and network structure optimization, based on the validation data. The ViT models especially stand out by reaching a comparatively high accuracy with a fairly small model size of fewer than 500 k parameters.  
Based on this study, two parameterizations per model are selected for a final training with a maximum of 200 epochs and early stopping as described above: the best performing model within a size of fewer than 5 M parameters and a small model version with significantly smaller number of parameters but only slight performance loss compared to the best in class model. The small model version is marked with a \"small\" (s) suffix in the following.