# Cross-Domain Fake News Detection Using a Prompt-Based Approach
## 5 Results
### Prompt-Engineering Sensitivity
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**Testing (Target Domain)**} & \multicolumn{5}{c}{**Metrics (Baseline GPT2)**} & \multicolumn{5}{c}{**Metrics (GPT2-D)**} \\  & **A(\%)** & **P (\%)** & **R (\%)** & **F1 (\%)** & **A(\%)** & **P (\%)** & **R (\%)** & **F1 (\%)** \\ \hline
**Bus.** & 92.73 & 93.67 & 92.73 & 92.70 & 98.11 & 98.18 & 98.11 & 98.11 \\
**Edu.** & 96.72 & 96.93 & 96.72 & 96.72 & 100.00 & 100.00 & 100.00 & 100.00 \\
**Tech.** & 97.18 & 97.34 & 97.18 & 97.18 & 100.00 & 100.00 & 100.00 & 100.00 \\
**Entert.** & 100.00 & 100.00 & 100.00 & 100.00 & 100.00 & 100.00 & 100.00 & 100.00 \\
**Politi.** & 100.00 & 100.00 & 100.00 & 100.00 & 100.00 & 100.00 & 100.00 & 100.00 \\
**Sports** & 88.06 & 88.06 & 88.06 & 88.06 & 91.04 & 92.29 & 91.04 & 90.88 \\
**Celeb.** & 86.33 & 86.38 & 86.33 & 86.33 & 87.86 & 87.86 & 87.86 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of cross-domain zero-shot transfer learning using the baseline GPT2 model and the GPT2-D model. Both models were trained on a comprehensive cross-domain dataset excluding the training set of the target domain in order to assess their cross-domain generalization capability. for the target concept (i.e., veracity). The observed difference in performance between the two labelling schemes (i.e., \"fake/real\", Table 2, vs. \"false/true\", Table 4) has a moderate to potentially major impact on the model's accuracy. This sensitivity to the phrasing of target words highlights the importance of careful label selection when applying the GPT2-D model to an FND task. * **Template formatting:*
* We explore two modifications to the prompt template:
* Quote addition: We introduce quotation marks around the prompt text, in contrast to the approach of [PERSON] et al. (2018), where quotes around input text were omitted. Table 5 presents the model's performance after surrounding the prompt with quotation marks. Compared to the original results (shown at the bottom of Table 2), a noticeable decline in accuracy is observed across most categories. Therefore, adding quotation marks seems to negatively impact the model's performance, at least at this level of analysis. * Prompt wording variations: This analysis examines the influence of varying the prompt wording on the performance of the GPT2-D model for FND. As illustrated in Tables 6 and 7, the prompt wording significantly influences the performance of the GPT2-D model. While all variations lead to a decline in accuracy compared to the original setup, the extent of the impact varies. Simpler prompts with less specific terminology seem to perform better. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Training Domain**} & \multicolumn{5}{c}{**Test Domains**} \\  & & **Bus.** & **Edu.** & **Tech.** & **Entert.** & **Politi.** & **Sports** & **Celeb.** \\ \hline \multirow{2}{*}{**GPT2-D**} & Bus. & 92.45 & 45.90 & 50.00 & 93.65 & 51.67 & 74.63 & 58.05 \\  & Edu. & 49.06 & 96.72 & 94.29 & 53.97 & 100.00 & 55.22 & 50.13 \\  & Tech. & 45.28 & 96.72 & 98.57 & 53.97 & 98.33 & 55.22 & 50.13 \\  & Entert. & 81.13 & 49.18 & 78.57 & 93.65 & 83.33 & 89.55 & 53.83 \\  & Politi. & 58.49 & 96.72 & 94.29 & 50.79 & 96.67 & 53.73 & 47.76 \\  & Sports & 100.00 & 49.18 & 48.57 & 96.83 & 51.67 & 100.00 & 53.56 \\  & Celeb. & 58.49 & 59.02 & 64.29 & 53.97 & 65.00 & 58.21 & 87.34 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Performance of the GPT2-D model using the prompt “This article discusses [DOMAIN], the classification of this article is”.