# A Deep Learning Approach with Feature Derivation and Selection for Overdue Repayment Forecasting
## 4 Experiments
### Evaluation Indicators  
The statistics of the positive and negative sample proportions described in Section 4.2 indicate that the data distribution is unbalanced with few positive samples. Supposing that every input individual is classified as negative, then we can still obtain an accuracy of 81.7%. Obviously, this accuracy value is quite high but meaningless, because no overdue behavior can be recognized by the model. As the performance of the model cannot be measured appropriately only in terms of accuracy, we supply other evaluation indicators including recall, the area under the curve (AUC) value and KS value.  
The confusion matrix is a fundamental evaluation indicator in binary classification tasks and is essential for the calculation of multiple other indicators as well. The confusion matrix is a \(2\times 2\) table with four combinations of actual values and prediction values:  
* TP: True positive, showing that the actual value and prediction value are both positive.
* TN: True negative, showing that the actual value and prediction value are both negative.
* FP: False positive, showing that the actual value is negative while the prediction value is positive.
* FN: False negative, showing that the actual value is positive while the prediction value is negative.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Parameter Description** & **Value** \\ \hline time\_step & Length of input sequence & 15 \\ lr & Learning rate & 0.03 \\ optimizer & Optimization method & Adam \\ lstm\_unit & Neuron number in single LSTM & 50 \\ DNN\_units & Neuron numbers in hidden & [150, 300] \\  & layers of DNN & \\ epoch & Training rounds & 20 \\ batch\_size & Batch size & 128 \\ dropout & Dropout ratio of LSTM & 0.15 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Parameter setting.  
On the basis of the four combination values above, TPR, TNR, FPR and FNR are the four ratio values that represent the proportion of corresponding combination values. The formulas for TPR and FPR are as follows:  
\[TPR =\frac{TP}{TP+FN} \tag{15}\] \[FPR =\frac{FP}{TN+FP} \tag{16}\]  
All of the evaluation indicators can be calculated by the combination values and ratio values mentioned above. The recall value represents the proportion of the positive samples that are predicted as the correct class, according to the following formula:  
\[recall=\frac{TP}{TP+FN} \tag{17}\]  
where a higher recall value indicates that more positive samples are distinguished by the model, which satisfies the actual demand of our research; i.e., to distinguish overdue repayment behaviors.  
Gradually changing the threshold of classification from 0 to 1 and calculating the corresponding TPR and FPR, we can form a line graph regarding the (TPR, FPR) pairs as points, named the receiver operating characteristic (ROC) curve. The AUC is the area under ROC curve. A large AUC value indicates good performance of the classification model, and a perfect model has an AUC of 1.  
Similarly, taking the threshold as the x-axis and the TPR and FPR as the y-axis, we can obtain a graph with two lines, named the KS curve. The KS value is the maximum distance between the FPR curve and FPR curve in the vertical direction.  
\[KS=\max(|TPR-FPR|) \tag{18}\]  
where a higher KS value indicates that positive and negative samples are distinguished more obviously.