# Customer Churn Prediction in B2B Non-Contractual Business Settings Using Invoice Data
## 4 Results
No significant differences were detected between LR and RF (\(p=0.235\)). Then, to identify the features that are most important for making predictions in the top performing models (for each churn definition, that is label window \(v\)), we adopted the permutation-based feature importance approach [43], the results of which are reported in Table 4. In this approach, values of a single feature are randomly shuffled and the change in model performance when using these values is measured repeatedly, the implication being that if the performance drops after permutation, the feature is important (i.e., the model depends on it). Finally, to test whether the multi-slicing approach to dataset creation yields better performing models when compared to the more commonly employed single-slicing approach, by using the overall best-performing method identified previously (RF), we devised predictive models for all \(DF_{num}\) and churn definition combinations, but leveraged only a single slice (i.e., the most recent one) of data. To make for a fair comparison, when devising models using a single-slicing approach, we adhered to the same experimental procedure described earlier. Results are presented in Table 5 (for easier comparison, we also included relevant results from Table 3). We then ran a two-sample \(t\)-test for both AUC and TDL measures, which indicated that significant differences exist between the approaches in both cases (\(p=0.001\) for AUC and \(p<0.001\) for TDL at \(\alpha=0.05\) level). Figure 4: Differences in classifier performance and significance of post-hoc test results (Tukey’s HSD) at \(\alpha=0.05\) for AUC and TDL measures. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Relative Feature**} & \multicolumn{3}{c}{**Churn Definition (in Months)**} \\ \cline{2-4}  & **One** & **Two** & **Three** \\ \hline \multirow{2}{*}{1} & Total number of & Total number of & Total amount \\  & invoices in _CF_ & invoices in _CF_ & invoiced in _m\({}_{t}\)_ \\ \hline \multirow{2}{*}{2} & Total number of & Total number of & Total amount \\  & invoices in _m\({}_{t}\)_ & invoice lines in _m\({}_{t-1}\)_ & invoiced in _m\({}_{t-10}\)_ \\ \hline \multirow{2}{*}{3} & SD of invoiced & Total number of & SD of amount \\  & amount in _m\({}_{t}\)_ & invoice lines in _CF_ & invoiced in _m\({}_{t-12}\)_ \\ \hline \multirow{2}{*}{4} & Total number of & Total amount & Total number of \\  & invoices in _m\({}_{t-8}\)_ & invoiced in _m\({}_{t-9}\)_ & invoices in _CF_ \\ \hline \multirow{2}{*}{5} & Total amount & Total amount & SD of number of \\  & invoiced in _m\({}_{t-9}\)_ & invoiced in _m\({}_{t-10}\)_ & invoice lines in _m\({}_{t}\)_ \\ \hline \multirow{2}{*}{6} & Total amount & Number of distinct \\  & invoiced in _m\({}_{t-1}\)_ & products invoiced in _CF_ & invoices in _m\({}_{t-10}\)_ \\ \hline \multirow{2}{*}{7} & Tenure (_m\({}_{t}\)_) & SD of number of \\  & invoice lines in _m\({}_{t}\)_ & invoice lines in _CF_ \\ \hline \multirow{2}{*}{8} & Days since last & Average invoiced \\  & invoice (_m\({}_{t}\)_) & amount in _m\({}_{t-10}\)_ & total invoiced \\  & invoice (_m\({}_{t}\)_-_m\({}_{t-10}\)_) & amounts (_m\({}_{t-m}\)_-_m\({}_{t-10}\)_) \\ \hline \multirow{2}{*}{9} & Difference in total & Difference in total \\  & number of invoice & number of invoices \\  & lines (_m\({}_{t}\)_-_m\({}_{t-6}\)_) & (_m\({}_{t}\)_ -_m\({}_{t-4}\)_) & _m\({}_{t-11}\)_ \\ \hline \multirow{2}{*}{10} & SD of number of & Number of distinct \\  & invoiced lines in _m\({}_{t}\)_ & products invoiced in \\  & invoiced lines in _m\({}_{t}\)_ & \({}_{t-12}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Top 10 most important features for each churn definition (label window _v_).