# An Approach to Trustworthy Article Ranking by NLP and Multi-Layered Analysis and Optimization
## 5 Expanded Experiments, Results, and Discussion
### Exemplar Result and Discussion  
In our designed three-layer architecture, the ranking mechanism serves as the final layer, following the BERT-based similarity comparison and article elimination stages as mentioned above in Sections 5.1-5.3. Figure 8 shows an exemplar result in our work after all the steps in the three layers are finished. In this example, \"climate change effects\" was the input keyword for the literature search. A total of 1605 similar articles were retrieved \begin{table}
\begin{tabular}{l l c c} \hline Configuration & Features & Test F1 & Performance Drop \\ \hline Full Model & Citations + Altmetric + & **0.9305** & - \\ Best Two-Feature & Citations + Altmetric & **0.9334** & 0.29\% \\ Without Citations & Altmetric + Impact & 0.831 & \(-\)10.70\% \\ Without Altmetric Score & Citations + Impact & 0.888 & \(-\)4.58\% \\ Without Impact Factor & Citations + Altmetric & 0.9334 & 0.31\% \\ Citations Only & Citations & 0.869 & \(-\)6.62\% \\ Altmetric Score Only & Altmetric Score & 0.8246 & \(-\)11.38\% \\ Impact Factor Only & Impact Factor & 0.8348 & \(-\)10.29\% \\ \hline \end{tabular}
\end{table} 
Table 4: Ablation study results for feature contributions. \begin{table}
\begin{tabular}{l l l l l l l} \hline
**Approach** & **F1-Score** & \begin{tabular}{l} **Precision** \\ **(Non-Retracted)** \\ \end{tabular} & \begin{tabular}{l} **Recall** \\ **(Non-Retracted)** \\ \end{tabular} & \begin{tabular}{l} **Precision** \\ **(Retracted)** \\ \end{tabular} & \
\begin{tabular}{l} **Recall** \\ **(Retracted)** \\ \end{tabular} &
\begin{tabular}{l} **Balanced** \\ **Accuracy** \\ \end{tabular} \\ \hline No Weights & \multirow{2}{*}{0.9305} & \multirow{2}{*}{0.8876} & \multirow{2}{*}{0.7083} & \multirow{2}{*}{0.8975} & \multirow{2}{*}{0.9661} & \multirow{2}{*}{0.8372} \\ (Original) & & & & & \\ \hline Balanced Weights & \multirow{2}{*}{0.9243} & \multirow{2}{*}{0.8094} & \multirow{2}{*}{0.7809} & \multirow{2}{*}{0.9183} & \multirow{2}{*}{0.9305} & \multirow{2}{*}{0.8557} \\ (Conservative) & & & & & & \\ \hline Custom Weights & \multirow{2}{*}{0.9203} & \multirow{2}{*}{0.7909} & \multirow{2}{*}{0.7855} & \multirow{2}{*}{0.9191} & \multirow{2}{*}{0.9215} & \multirow{2}{*}{0.8535} \\ (Moderate) & & & & & & \\ \hline \end{tabular}
\end{table} 
Table 5: Model cost-sensitive learning results. from the database based on a similarity comparison in the first layer. In the second layer, the filtering algorithm eliminated 646 retracted articles, leaving 959 valid results. Finally in the third layer, based on the proposed ranking method, the top five most relevant articles were presented to the user. This sequential approach ensures that search results are first filtered for relevance, then screened for trustworthiness, and finally, ranked according to our composite scoring system. Overall, the result demonstrates how multi-factor analysis can improve the search result quality beyond simple relevance matching. After evaluating the experiment results, we were able to summarize the findings and potential limitations of each layer. In Layer 1, from the experiments we conducted in Section 5.1, we found that using BERT embedding with cosine similarity is the best approach for finding the most relevant articles based on the user-input keywords. In Layer 2, while our Random Forest classification model performed well overall, we found several issues that should be noted. First, our dataset had a significant imbalance, with 72.57% retracted papers and only 27.43% non-retracted papers. This uneven split likely affected how the model learned and may explain why it performed differently in each class. Second, the model was very good at finding retracted papers (97% recall) but less effective at correctly identifying non-retracted papers (71.0% recall). This means that 29.0% of good papers were wrongly flagged as problematic. In real use, this could cause valid research to be unfairly questioned. However, this tendency is suitable for our main goal of filtering out questionable articles from search results.