# An Empirical Study on Customer Churn Behaviours Prediction Using Arabic Twitter Mining Approach
## 3 Methodology
### Modelling  
Performance evaluation metrics: There are useful metrics that should be used to assess the performance of the model and compare it with a benchmark. Numerous churning prediction studies used certain performance metrics, such as precision, recall, F1, accuracy, confusion matrix, specificity, sensitivity, area under the curve (AUC) and receiver operating characteristic curve (ROC). A confusion matrix is a tool used with binary classification; it compares the actual positive and negative and the predicted positive and negative.  
It uses the true negative (TN), true positive (TP), false negative (FN) and false positive (FP) ([PERSON], [PERSON] et al. 2009) as follows:  
* FP: indicates that our model predicts the customer is a churner but the customer is a non-churner.
* FN: indicates that our model predicts the customer is a non-churner but the customer is a churner.
* TP: indicates our model correctly predicts the customer is a churner.
* TN: indicates our model correctly predicts the customer is a non-churner.  
There are other metrics used in addition to _TP_, _TN_, _FN_ and _FP_, such as sensitivity, specificity and accuracy. The weakness of an accuracy measure originates from overusing the sensitivity and specificity measures ([PERSON] et al., 2014). Sensitivity is equal to recall. Meanwhile, specificity is the ratio of the negative correctly predicted as shown in the following equation:  
\[Specificity=TN/(TN+FP) \tag{1}\]  
High sensitivity is more preferred than high specificity in telecom providers because the cost of an untrue classification of a non-churner is less than the cost of an untrue classification of a churner [1]. Some churning prediction studies prefer to evaluate model performance by using ROC and AUC because of the ability of these curves to remain the same with imbalanced data, even if the positive and negative instances are changing [1].  
ROC is a two-dimensional curve drawn to show the relation between TP, the churner that is correctly predicted, and FP, the non-churner incorrectly predicted as a churner [93]. The best model performance occurs when the ROC is close to (0,1). A better model performance also has higher AUC.  
Moreover, we used a cross-entropy/logarithmic loss (log loss) as loss function; both calculate the same in the classification problem. The loss function is an error metric to measure uncertainty. It is one of the measures used for evaluating the performance of a binary classifier from the probability estimation between 0 and 1. Log loss penalises both types of errors, especially those predictions where the confidence is inaccurate. If the log loss is closer to zero, then this indicates a good performance of the model.  
Figure 5: Final dataset after preparation.  
Using the log loss provides us with an accurate view of our performance model based on the prediction of probabilities, not only the output.  
\[Hp(q)^{n}=\frac{1}{N}\sum_{i=1}^{n}y_{i}\;\log((P(y_{i})+(1-y_{i})\log(1-P(y_{i})) \tag{2}\]  
where \(N\) is the number of items on the training set; \(1/N\) is the probability of each class; log is the natural logarithm; \(y\) is the binary label, which is either 0 or 1; and \(p(y)\) is the probability predicted of the class.