# Mimicking Human Verification Behavior for News Media Credibility Evaluation
## 5 Results and Discussions
### Performance Comparison (RQ1)  
In the previous sections, we described the proposed MiBeMC model, a public dataset for media credibility evaluation, and a performance evaluation metric. To answer RQ1, we conducted a comprehensive experiment on the CLEF-2023 task 4 dataset (a publicly available benchmark dataset). Table 3 shows the performance of each of the models. The experimental results show that the MiBeMC model has lower MAE values than the others. In this comparison experiment, the MiBeMC settings are as follows: using Glove to encode the text input, the hyperparameter \(k\) of the R-AT strategy is 0.3, and the content truncation strategy is to retain 50 characters at the beginning and the end of the article content, as well as the middle 100 characters. The other settings are the same as the default configurations.  
Our proposed model has improved performance by 0.606, 0.539, 0.339, and 0.198 compared to the current baseline models Random, Middle class, Majority class, and Ngram baseline, and it has improved performance by 0.022, 0.194, and 0.268 compared to the state-of-the-art models CUCPLUS, Accenture, and UBCS.  
Compared to the modal-based models (CUCPLUS and Accenture), the MiBeMC model achieves better performance. This may be because these modal-based methods can lead to sub-optimal outcomes since they overlook human behavior in the reading and verification process. From the experiment results, it can be concluded that modeling the media credibility evaluation process from a component perspective combined with the human's reading, understanding, and verification processes is effective because it can achieve an excellent performance compared to the modal-based methods.  
In addition, the MiBeMC provides a complete solution for evaluating the credibility of the news outlet. Even if the number of history articles published by media organizations is small, or even if it is only one, the proposed model can obtain additional information based on the title and the content of the current article. The retrieval information based on the title and the history information based on the cosine similarity provide reliable additional  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **MAE on Dev** & **MAE on Test** \\ \hline Ngram baseline & 0.392 & 0.943 \\ Majority class (i.e., high) & 0.533 & - \\ Middle class (i.e., center) & 0.733 & - \\ Random & 0.800 & - \\ CUCPLUS & 0.216 & 0.295 \\ Accenture & - & 0.467 \\ UBCS & - & 0.541 \\ MiBeMC & 0.194 & 0.273 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of the performance of the proposed model with the compare models on the CLEF-2023 Task 4 dataset.  
relevant information. At the same time, regularized adversarial training is also used in order to reduce the impact of redundant data and to improve the performance of MiBeMC.  
In the end, we also examined the performance of different pre-trained language models combined with our proposed model. Table 4 shows the algorithm performance after using RoBERTa and DeBERTa instead of Glove.  
The results in Table 4 show that after replacing Glove with a pre-trained language model, the MAE values are all decreased. We suggest that this phenomenon is due to the pre-trained language models having a stronger language processing ability and being more suitable for handling the complex task of media credibility evaluation. Through more careful observation, we find that DeBERTa is better than RoBERTa. This may be because DeBERTa's virtual adversarial training strategy is consistent with the goal of the regularized adversarial training strategy that we used, both to reduce the impact of data redundancy and the mismatch between the training and the prediction. RoBERTa also preferred to focus on the token itself, whereas DeBERTa preferred special character dominance due to its enhanced mask decoder. This makes it effective for us to select special characters, such as [CLS] and [SEP], as feature vectors in data preprocessing as the context representation of the input sequence.