# Customer Churn Prediction in B2B Non-Contractual Business Settings Using Invoice Data
## 4 Results  
To obtain experimental results, we adopt the setup illustrated in Figure 3. Starting with raw, invoice-level data, for each combination of churn definition (Label window _v_) and length of _DF_ periods (_DFnum_), we derive training and test sets by leveraging the multi-slicing approach described in Section 3.2. Within every iteration, datasets comprise delta and cumulative features, as well as appropriate churn labels. We then proceed to train and tune the selected algorithms by using the hyperparameter values presented in Table 2, and evaluate the resulting models using out-of-period testing. This setup resulted in a total of 45 models, the performance of which we assessed via AUC and TDL measures. The obtained results are shown in Table 3, with the best performance for each measure (for each churn definition) outlined in bold and underlined. In terms of AUC, the RF classifier yields the highest overall results among the three classifiers, over all combinations of churn definitions and _DF_ length. The top AUC score of 0.9050 was obtained for a churn definition of two months when 12 months of historical data were used to construct _DF_, closely followed by AUC's of 0.9020 for a churn definition of three months (also with _DFnum_ period of 12 months) and 0.8970 for the one-month churn definition (_DFnum_ period was 9 months in this case). RF also shows the best performance when TDL is in question, with a lift of 4.9090 obtained for churn definition of two months, 4.8091 for a churn definition of three months (in both cases with length of 12 for _DFnum_) and 4.4604 for a one-month churn definition. In the last case, results were obtained with the model using _DFnum_ of 3. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & & & **LR** & & **RF** & & **SVM** \\ \hline \multirow{2}{*}{**Label Window**\(v\)**} & _DFnum_ & \multirow{2}{*}{**Methods**} & \multirow{2}{*}{**AUC**} & \multirow{2}{*}{**TDL**} & \multirow{2}{*}{**AUC**} & \multirow{2}{*}{**TDL**} & \multirow{2}{*}{**AUC**} & \multirow{2}{*}{**TDL**} \\  & & & & & & & & \\ \hline \multirow{4}{*}{3} & 12 & 0.8803 & 3.6985 & **0.9020** & **4.8091** & 0.8359 & 3.9694 \\  & 9 & 0.8800 & 3.9694 & 0.9015 & 4.5038 & 0.8270 & 3.8167 \\  & 6 & 0.8884 & 4.4253 & 0.8987 & 4.5801 & 0.8563 & 4.3511 \\  & 3 & 0.8908 & 4.5038 & 0.8940 & 4.3511 & 0.8694 & 4.4274 \\  & 1 & 0.8921 & 4.6564 & 0.8966 & 4.6564 & 0.8710 & 4.0458 \\ \hline \multirow{4}{*}{2} & 12 & 0.8957 & 4.4545 & **0.9050** & **4.9090** & 0.8658 & 3.5454 \\  & 9 & 0.8744 & 4.1818 & 0.8875 & 4.5454 & 0.8628 & 3.7272 \\  & 6 & 0.8682 & 4.3636 & 0.8813 & 4.1818 & 0.8509 & 3.6363 \\  & 3 & 0.8688 & 4.2727 & 0.8832 & 4.5454 & 0.8534 & 4.0000 \\  & 1 & 0.8642 & 4.1818 & 0.8728 & 4.5454 & 0.8463 & 3.6363 \\ \hline \multirow{4}{*}{1} & 12 & 0.8803 & 4.2446 & 0.8917 & 3.9568 & 0.8350 & 2.9496 \\  & 9 & 0.8826 & 4.0287 & **0.8970** & 4.3165 & 0.8210 & 3.0935 \\  & 6 & 0.8772 & 4.1007 & 0.8962 & 4.3884 & 0.8258 & 3.5971 \\  & 3 & 0.8754 & 4.1007 & 0.8948 & **4.4604** & 0.8370 & 3.0212 \\  & 1 & 0.8801 & 4.3165 & 0.8847 & 3.8129 & 0.8203 & 3.3093 \\ \hline Average & & 0.8799 & 4.2333 & 0.8925 & 4.4375 & 0.8452 & 3.6751 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Experimental results. Figure 3: Experimental setup. To test whether the observed differences in classifier performances are significant, we conducted a one-way ANOVA for both AUC and TDL measures, with [PERSON]'s HSD as a post-hoc test. The obtained results are summarized in Figure 4. ANOVA for AUC showed that there are significant differences between classifiers at \(p<0.001\) significance level (we use \(\alpha=0.05\) threshold) and [PERSON]'s HSD revealed that differences exist between all three classifiers (\(p<0.001\) for differences between SVM and LR/RF, and \(p=0.024\) for differences between RF and LR). In terms of TDL, ANOVA also showed that there are differences between classifiers (\(p<0.001\)), but [PERSON]'s HSD revealed that significant differences at \(\alpha=0.05\) exist only between SVM and the other classifiers.