## 4 Results and Discussion
### Ablation Study Results of Transformer Models
#### 4.1.5 Ablation Study Results Summary  
After analyzing the importance of each architectural component of the four standard transformer models for FND, the baseline performance is given in Table 10. Based on the aggregated results of Tables 11 and 12, several key insights emerge regarding model effectiveness and architectural sensitivity in FND tasks. Among the four transformer variants evaluated, DistilBERT stands out as the best overall performer, achieving the highest average accuracy of 0.7023 and an average F1 score of 0.6459, despite having the lowest parameter count, demonstrating its efficiency and suitability for resource-constrained environments. Across all models, the attention mechanism is identified as the most critical architectural component, with its removal resulting in the largest average performance drop of 0.1671, reinforcing the foundational role of self-attention in contextual understanding and information representation. On the other end of the spectrum, the relative positional encoding component shows a slightly negative average impact of \(-\)0.0075, indicating that its removal can, in some cases, slightly benefit performance or is otherwise redundant within these architectures. When examining model-specific vulnerabilities, BERT,  
\begin{table}
\begin{tabular}{l l c c} \hline \hline
**Dataset** & **Configuration** & **Test Accuracy** & **Test F1** \\ \hline \multirow{8}{*}{FakeNewsNet} & baseline & 0.5250 & 0.3615 \\  & no\_positional\_encoding & 0.4000 & 0.3816 \\  & no\_attention & 0.4750 & 0.3059 \\  & no\_feedforward & 0.4000 & 0.3688 \\  & no\_residual\_connections & 0.5250 & 0.3615 \\  & no\_layer\_normalization & 0.3500 & 0.3400 \\  & no\_dropout & 0.4250 & 0.3466 \\  & complex\_classifier & 0.5000 & 0.4423 \\ \hline \multirow{8}{*}{ISOT} & baseline & 0.9990 & 0.9990 \\  & no\_positional\_encoding & 0.9990 & 0.9990 \\  & no\_attention & 0.5267 & 0.3634 \\  & no\_feedforward & 0.9983 & 0.9983 \\  & no\_residual\_connections & 0.9990 & 0.9990 \\  & no\_layer\_normalization & 0.9987 & 0.9987 \\  & no\_dropout & 0.9990 & 0.9990 \\  & complex\_classifier & 0.9990 & 0.9990 \\ \hline \multirow{8}{*}{LIAR} & baseline & 0.5830 & 0.5772 \\  & no\_positional\_encoding & 0.5892 & 0.5461 \\  & no\_attention & 0.5585 & 0.4002 \\  & no\_feedforward & 0.5960 & 0.5739 \\  & no\_residual\_connections & 0.5814 & 0.5074 \\  & no\_layer\_normalization & 0.5010 & 0.4471 \\  & no\_dropout & 0.5939 & 0.5944 \\  & complex\_classifier & 0.5835 & 0.5849 \\ \hline \hline \end{tabular}
\end{table}
Table 8: DistilBERT Ablation Study: Performance Across All Datasets.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Component Removed** & **Avg Impact** & **Std Dev** & **Count** \\ \hline no\_attention & 0.1823 & 0.2054 & 3 \\ no\_layer\_normalization & 0.0858 & 0.0714 & 3 \\ no\_positional\_encoding & 0.0396 & 0.0605 & 3 \\ no\_feedforward & 0.0375 & 0.0621 & 3 \\ no\_dropout & 0.0297 & 0.0499 & 3 \\ complex\_classifier & 0.0082 & 0.0119 & 3 \\ no\_residual\_connections & 0.0005 & 0.0007 & 3 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Component Importance Summary Across All Datasets (DistilBERT).  
RoBERTa, and DistilBERT consistently identify the attention mechanism as their most sensitive component, with average performance drops of 0.1504, 0.1688, and 0.1823, respectively, when ablated. In contrast, DeBERTa shows the greatest degradation when the feedforward network is removed, with an average performance drop of 0.0453, suggesting that its unique disentangled attention and decoder mechanisms may reduce its reliance on traditional attention structures. These insights collectively highlight the dominant influence of attention-based components across transformer models while emphasizing the nuanced architectural dependencies unique to each variant.