# Research on the Prediction of Operator Users' Number Portability Based on Community Detection
## 2 Modeling and Experiment Comparison
### Comparison of Prediction Models  
The main purpose of this article was to build a prediction model with higher accuracy based on existing data records and provide telecom operators with a list of users who are prone to number portability. In this paper, we chose the widely used KNN, random forest model, SVM, deep neural network, and the Light GBM model to compare the prediction accuracy of different models, and to find the optimal user number portability prediction model. Firstly, to ensure the fairness and reliability of the comparison between the models, the performance of the four models was evaluated via ten-fold cross-validation, to calculate the accuracy rate of each evaluation. As shown in Figure 6, the predictive accuracy of the KNN and SVM is the lowest at below 85%, followed by the RF model. The DNN and Light GBM models have the highest prediction accuracy, both exceeding 90%. The accuracy of Light GBM is slightly higher than that of the DNN. Therefore, the prediction accuracy of the DNN and Light GBM is significantly higher than that of the other three models, while that of Light GBM is slightly better than that of the DNN. Figure 6: Comparison of model prediction accuracy. \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline  & **Advantage** & **Disadvantage** \\ \hline Filter & The calculation efficiency is the highest (it took 152 s to run the final result in this project), and overfitting can be effectively avoided. In the end, this method retained 60 eligible features & It does not consider the characteristics of the machine learning algorithm and weakens the fitting ability of the learner. \\ \hline Wrapper & It is more targeted and good for model performance. & The algorithm requires a large amount of computation, runs for too long (the took about 9 h to run the final result in this project), and is easy to overfit when the sample is not sufficient. In the end, this method retained 34 eligible features. \\ \hline Embedding & It can make full use of the model’s evaluation of features. & Only feature selection and model training can be performed at the same time, and the subsequent feature selection of the trained model cannot be performed, which takes a long time (40 min in this project). In the end, this method retained 52 eligible features. \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of advantages and disadvantages of the three feature selection methods. Secondly, to study the influence of the number of characteristic variables on the prediction accuracy of each model, the prediction accuracy of each model with different numbers of characteristic variables is calculated. As shown in Figure 7, when the number of characteristic variables is less than 10, the prediction accuracy of each model is significantly improved with the increase in the number of characteristic variables. When the number of characteristic variables is greater than 15, the prediction accuracy of each model is not obviously improved, remaining basically unchanged. As a single indicator cannot reflect the overall performance of the model, in this project, we must combine different indicators to reflect the overall performance of the model. Therefore, here, we adopt accuracy, precision, recall, F1-score, and AUC-value to evaluate the performance of the model. The evaluation results are shown in Table 4. Among the five prediction models, KNN's evaluation indicators are lower than other models. On the contrary, all the evaluation indicators of Light GBM are higher than other models. Figure 8 is a schematic diagram of the ROC curves for each model, and the results are consistent with the data of each evaluation indicator in Table 4. The area under the ROC curve of KNN is the smallest, and the area under the ROC curve of Light GBM is the largest. It can be seen from Table 4 and Figure 8 that, compared with other machine learning algorithms, the performance of each evaluation index of the Light GBM model is better.