# Customer Churn Prediction in B2B Non-Contractual Business Settings Using Invoice Data
## 3 Materials and Methods
### Evaluation Metrics  
In binary classification problems, the performance of a model can be evaluated by using different metrics, the suitability of which will vary depending on the model's ultimate purpose and the potential cost of misclassification. Even though accuracy, precision, recall, and F1 measures are often used with binary classifiers [19; 26; 28], given the purpose for which we devise our models for (i.e., churn prediction), we opt for measures that are well established within the domain and which capture the overall model performance and allow for their assessment from a decision-making standpoint: AUC and TDL [18; 42]. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Classifier** & **Hyperparameter** & **Values** \\ \hline Logistic Regression & Regularization C & \([10^{-5},10^{-4},\ldots,10^{2}]\) \\ \hline Support Vector Machines & Regularization C & \([0.1,0.2,\ldots,1.2]\) \\ \hline \multirow{3}{*}{Random Forests} & Maximum number of features (F) & \([\sqrt{F},2\sqrt{F},3\sqrt{F}]\) \\  & Maximum tree depth & \([3,4,\ldots,15]\) \\ \cline{1-1}  & Minimum samples per leaf & \([2,3,5]\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Ranges used for hyperparameter grid search Probabilistic models, such as those devised within our work, enable the assignment of a particular class label to every instance based on a predefined threshold (e.g., if the probability output by the model is \(\geq\)50%, then assign label \"1\" or \"churn\"). This subsequently enables the construction of a confusion matrix: an overview of correctly and incorrectly classified instances where rows represent the number of ground-truth instances per class, while columns represent model predictions per class. Correctly classified instances are those where the model-assigned labels agree with the ground truth class and can be observed as either True Positives (_TP_), when an actual churner is labeled as a churner by the model, or True Negatives (_TN_) when an actual non-churner is labeled as a non-churner by the model. Incorrectly classified instances, on the other hand, occur when there is a disagreement between predictions and actual labels and can be either False Positives (_FP_), when an actual non-churner is labeled as a churner by the model, or False Negatives (_FN_), when an actual churner is labeled as a non-churner by the model. Depending on the threshold set, the number of _TP_, _TN_, _FP_, and _FN_ will vary, so in order to assess the overall predictive performance of a model over all possible thresholds, a summary measure is required. This measure is known as _AUC_ and is calculated by approximating:  
\[AUC=\int_{0}^{1}TPRdFPR \tag{1}\]  
where the True Positive Rate (_TPR_) and False Positive Rate (_FPR_) are calculated for different probability thresholds as:  
\[TPR=\frac{TP}{TP+FN} \tag{2}\]  
\[FPR=\frac{FP}{FP+TN} \tag{3}\]  
_AUC_ ranges between 0 and 1, where a random classifier has a score of 0.5, while a perfect classifier has a score of 1. While _AUC_ gives a good overall assessment of the classifier performance, _TDL_ is often used to estimate how useful the model would be if applied in a real-world setting, such as for retention activities. Given a list of churn probabilities obtained via a model and sorted in a descending order, one would expect a baseline classifier (i.e., random guessing) to capture a proportional percentage of churners within each decile of the list (e.g., the top 10% of the list would contain 10% of actual churners, the top 20% of the list would contain 20% of actual churners, and so on). _TDL_ expresses the improvement a model provides over this baseline for the top decile (i.e., top 10% of the list), since retention activities are usually directed towards customers most likely to churn. It is calculated for the top 10% of the customers predicted by the model as most likely to churn as:  
\[TDL=\frac{\frac{TP}{TP+FP}}{CR} \tag{4}\]  
where _CR_ is the overall churn rate.