## 3 Results
### Analysis of the Results
the shift was 0.025, while with a 40-day shift, this difference decreased to 0.0143. This supports Hypothesis 1, showing that mapping evaluations to the testing time windows reduces the impact of shift drift. Furthermore, this shift partly supports Hypothesis 2, as the smaller difference between the curves indicates that partial churn labels can be effectively used to predict full churn if the prediction threshold is properly adjusted. The differences in AUC and F1 scores, measured using the L2 norm (as described in Section 2.2.2), were visualized through the heatmaps in Figures 10 and 11, and reflect the variations observed in Figures 4 and 5. The AUC showed minimal differences (range: 0.0041-0.0111), indicating that it remains largely unaffected by scenario variations. In contrast, the F1 score exhibited more significant variation (range: 0.025-0.0923), particularly between the full and partial churn scenarios, due to a sharp drop in the F1 scores for full churn predictions. These minimal AUC differences between the full and partial churn scenarios support Hypothesis 2, suggesting that partial churn labels can effectively approximate full churn predictions without significant performance loss, particularly when the AUC is the preferred metric. Meanwhile, the higher variability in F1 scores across scenarios indicates that the F1 is more sensitive to changes in classification results, as it combines both precision and recall. Heatmaps for other metrics, including accuracy (range: 0.0159-0.0375), recall (range: 0.0237-0.0549), and precision (range: 0.0314-0.1424), are presented in Appendix A Figures A1-A3. Precision exhibited the highest variation, suggesting that the balance between true and false positive predictions is highly dependent on scenario and its specific factors. The additional metrics in Appendix A further highlight these trends, with accuracy and recall showing moderate robustness, while the heightened sensitivity of precision emphasizes the importance of careful threshold calibration. In summary, the results underscore the critical role of metric selection in analyzing model performance under drift, with the AUC and F1 score providing the most significant evidence to validate the hypotheses. Figure 11: AUC differences using the L2 norm between scenarios. Figure 10: F1 score differences using the L2 norm between scenarios. Next, we further investigate the possibility of exploiting the specificity of the difference between full and partial churn scenarios, where the difference between these scenarios is small when using the decision threshold-independent metric of AUC and large when using other metrics, especially the F1 score. To investigate whether we can improve the F1 score, we deployed a new experiments; setup with the following properties:  
1. We modified the original Catboost model by selecting a custom decision threshold, varying it from 0.5 to 0.7 with a step of 0.01. 2. For clarity, we analyzed two scenarios that obtained the richest results from a practical point of view--both full train and true test scenarios, with different labels of partial and full churn, respectively. 3. At each step, we evaluated the performance by averaging the F1 score metric over all time windows. 4. Finally, we selected the decision threshold that obtained the best results and investigated it more closely. The outcome of the first three points that were mentioned is provided in Figure 12. We can see that the full churn scenario performed best when the threshold value was around 0.6. The partial churn scenario started to perform worse--this is an expected result because the model was trained using partial churn labels. In short, there are two potential reasons that the full churn scenario might perform worse than the partial churn scenario:  
* As previously mentioned, the model is trained to solve different tasks than it was tested on; although the difference is small, these differences lead to the unique effect of an indirect connection between tasks being observed.