## 4 Validation and Implementation
The AUC is the measure of the ability of a classifier to distinguish between classes and is used as a summary of the ROC curve. The higher the AUC, the better the performance of the model at distinguishing between the positive and negative classes. Log-loss is indicative of how close the prediction probability is to the corresponding actual/true value (0 or 1 in case of binary classification). The more the predicted probability diverges from the actual value, the higher is the log-loss value. These metrics selected since the vast majority of researchers that perform binary classification use them. Regarding the second dataset, the DIGIX, 1 M were randomly selected and the sample contained nearly 78% 'No' and 22% 'Yes' regarding the dependent variable (so as to be equivalent with the first sample). Once again, 80% used for training and 20% for testing. Additionally, the same tuning was followed. Table 1 and Figures 1 and 2 below display the results in these two datasets. It can be noticed that state-of-the-art deep learning models clearly outperform traditional models, providing very satisfactory results. There are not any significant differences among them (DCNv2, FLEN, and FiBiNet have slightly better results). It is also worth mentioning that the ensembler improved the results. Additionally, more experiments have been conducted in these two datasets to explore further the performance of the models for the frozen start problem. Due to the fact that these two datasets can be considered as slightly imbalanced (regarding the distribution of the target variable-\(0^{\prime}\) is considered as the majority class since it covers almost 80% of the rows in both samples), undersampling technique has been used to conduct experiments on totally balanced datasets as well (1:1 regarding the distribution of the target variable in our cases) [42]. The undersampling technique was used in both datasets. In the context of a binary (two-class) classification problem where class 0 is the majority class and class 1 is \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**Avazu**} & \multicolumn{2}{c}{**DIGIX**} \\ \hline
**Model** & **AUC** & **Log-Loss** & **AUC** & **Log-Loss** \\ \hline Gradient Boost & 0.51 & 6.1 & 0.50 & 7.89 \\ CatBoost & 0.536 & 5.45 & 0.521 & 7.81 \\ LightGBM & 0.54 & 5.876 & 0.517 & 7.867 \\ Logistic Regression & 0.54 & 5.46 & 0.528 & 7.83 \\ ONN & 0.746 & 0.397 & 0.657 & 0.654 \\ xDeepFM & 0.753 & 0.383 & 0.655 & 0.655 \\ IFM & 0.754 & 0.382 & 0.654 & 0.656 \\ DCN v2 & 0.755 & 0.38 & 0.654 & 0.655 \\ FiBiNET & 0.755 & 0.381 & 0.655 & 0.656 \\ FLEN & 0.755 & 0.381 & 0.658 & 0.654 \\ Ensember & 0.76 & 0.371 & 0.661 & 0.496 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Model performances for the Avazu and Digix dataset. Figure 1: AUC scores for models regarding Avazu and Digix dataset. Figure 2: Log-loss scores for models regarding Avazu and Digix dataset. the minority class, such as Avazu and DIGIX, undersampling techniques remove examples from the training dataset that belong to the majority class in order to better balance the class distribution, such as reducing the skew from a 1:100 to 1:2, or even a 1:1 class distribution. This technique was helpful to check the performance of the models when less records are available [37]. Once again:  
* Data were shuffled balanced and 80% used for training and 20% for testing. * The same tuning was followed for the models. * AUC and log-loss were calculated. Table 2 and Figures 3 and 4 below display the results for these two balanced undersampled datasets. The results show that deep learning techniques outperform traditional/baseline methods in any case. Furthermore, in the Avazu dataset the result of the ensembler is really close to the score that the algorithms achieve with all the attributes available. Following the experiments that have been conducted by other researchers, the algorithms achieve scores between 0.75-0.77 when all attributes are available [28].