# An Approach to Trustworthy Article Ranking by NLP and Multi-Layered Analysis and Optimization
## 5 Expanded Experiments, Results, and Discussion
### Random Forest Model Validation  
We tested the Random Forest model thoroughly to check its stability. The learning curves (Figure 7A) showed some gap between training accuracy (about 0.90) and cross-validation scores (about 0.88). This gap suggests some overfitting, but not enough to harm practical use, as the cross-validation scores remained steady (mean: 0.884, standard deviation: 0.0049). Moreover, the precision-recall curve (Figure 7B) stayed above 0.90 precision until the recall reached about 0.80, with an average precision (AP) of 0.96. This means the model maintains high-quality predictions across different settings. The ROC (Receiver Operating Characteristic) curve (Figure 7C) showed a quick rise and high AUC (0.93), \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Classification Report** & & & & \\ \hline
**Class** & **Precision** & **Recall** & **F1-Score** & **Support** \\ \hline
**0** & 0.89 & 0.71 & 0.79 & 881 \\
**1** & 0.90 & 0.97 & 0.93 & 2330 \\
**Accuracy** & & & 0.89 & 3211 \\
**Macro Avg.** & 0.88 & 0.84 & 0.86 & 3211 \\
**Weighted Avg.** & 0.89 & 0.90 & 0.89 & 3211 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Performance of Random Forest classification. Figure 6: Confusion matrix and feature importance of the Random Forest classification model. The percentage of the model’s correct and incorrect predictions is shown in the confusion matrix. The impact of each feature on the model’s predictions is shown in the feature importance. further proving the model's strong performance. The steep initial part of the ROC curve shows that the model can find many retracted papers with few false alarms. To validate our modeling approach for retraction prediction, we conducted a comprehensive comparison of ten machine learning algorithms (results shown in Table 3): Random Forest, Gradient Boosting, XGBoost, Logistic Regression, Support Vector Machine, Neural Network (MLP), AdaBoost, K-Nearest Neighbors, Decision Tree, and Naive Bayes. All models underwent hyperparameter tuning using 5-fold cross-validation with the F1-score as the evaluation metric to account for the class imbalance in our dataset. Our experimental results show that Gradient Boosting achieved the highest performance with a test F1-score of 0.9336 and AUC of 0.9477, followed closely by XGBoost (F1-score: 0.9333, AUC: 0.9452) and Random Forest (F1-score: 0.9305, AUC: 0.9377). Statistical significance testing using [PERSON]'s test revealed that the differences between these top three models were not statistically significant (Gradient Boosting vs. XGBoost: \(p\) = 1.0000; Gradient Boosting vs. Random Forest: \(p\) = 0.1980), indicating comparable predictive capabilities. Despite Gradient Boosting's marginally superior performance (0.31% improvement), we selected Random Forest as our primary classifier due to its superior interpretability through intu  
Figure 7: Metrics used to validate the performance of the Random Forest model. The learning curve shows the model’s performance with different iterations. The precision–recall curve shows the relationship between precision and recall under different classification thresholds. The ROC curve shows the true positive rate against the false positive rate across different thresholds, the dash line represents the line of no discrimination. itive feature importance rankings, computational efficiency with faster training times and lower hyperparameter tuning requirements, and greater stability with a more consistent cross-validation performance. For scientific retraction detection, the model interpretability and ease of explanation are crucial for gaining trust from the research community, making Random Forest's transparent decision-making process more valuable than the negligible performance difference. The comparison results validate that ensemble methods significantly outperform traditional machine learning approaches, with all three top models achieving F1-scores above 0.93 and demonstrating the effectiveness of tree-based algorithms for this classification task.