# Approaches for Dealing with Seasonality in Clinical Prediction Models for Infections
## 4 Experiments and Results
### Experimental Settings  
We experimented with five different approaches used to deal with seasonality in data:  
* _None_: we built a single model that ignores the season; the results of this experiment were used as a gold standard for comparison. * _Season as a feature_: we built a single model that includes the season as an additional feature for prediction. * _Model per season_: we built isolated models, one for each season; for a given observation to be predicted, the model corresponding with the relevant season is used. * _Monthly/seasonal window (3,5,7,9)_: as explained earlier, a sliding window was adapted to account for seasonality; we experimented using windows with lengths of 3, 5, 7, and 9 months and with a window containing both the season of the observation to be predicted and the adjacent ones (i.e., a seasonal window). * _Monthly/seasonal ensemble_: we used an ensemble model that aggregates the output of different models for each prediction, as explained previously; we experimented with an ensemble of twelve models (i.e., one model per month) and four models (i.e., one model per season). Figure 5 provides an overview of the workflow employed in the experimental settings. We adopted a training-validation-testing strategy [40]. First, the dataset was split into training/validation (80% of the data) and testing datasets (20% of the data). We followed a random sampling strategy only within each class group in order to preserve the overall class distribution of the data. We then generated 100 datasets from the original training/validation subset using a sampling with replacement strategy. Each particular combination of techniques was then applied to each resampling of the training/validation dataset. The first step consisted of applying the technique related to seasonality. Several of these approaches might imply the creation of various models over different subsets of data (e.g., a model based on data for winter only). After the training/validation resample had been partitioned (if necessary), we then applied the  
Figure 5: Workflow followed when performing the experiments. balancing approach and the feature selection algorithm, and the model was eventually built using either logistic regression or the C5.0 algorithm for tree models. In addition to the tested approaches for feature selection, we applied the LASSO technique to the logistic regression models and winnowing to the decision trees in all our experiments. These techniques are common approaches which are used together with the aforementioned modeling techniques in the presence of high dimensionality, and as such are suitable for inclusion in this scenario. We consequently trained 100 models per combination of approaches and dataset (three options for feature selection, five options for class balancing, ten options for seasonality, and two techniques for interpretable models), that is, a total of 30,000 models per dataset. Each model we created was then used to predict the test datasets, and the resulting area under the receiver operating characteristic curve (AUC) was stored. This made it possible to obtain 100 AUC results per combination of techniques. A \(t\)-test was then performed in order to calculate the mean and 95% confidence intervals per combination used in the remainder of the analysis. The number of predictors included in these models was studied as an approximation with which to evaluate the differences in complexity between models. All the experiments were performed using the R platform version 4.0.2 and RStudio version 1.3.1093. The LASSO models were fitted using the _glmnet_ R package [41; 42]. The \(\alpha\) parameter, which controls the elastic net behavior, was set to \(\alpha=1.0-10^{-5}\) in order to obtain a LASSO effect and ensure numerical stability [42].