Truth over Tricks: Measuring and Mitigating Shortcut Learning in Misinformation Detection
Authors: Herun Wan
Jiaying Wu
Minnan Luo
Zhi Zeng
Zhixiong Su
Venue: NeurIPS 2025 poster ()
Keywords: Shortcut Learning
Injection Attack
Misinformation Detection
Primary area: applications
TL;DR: This paper measures and mitigates shortcut leaning in the misinformation detection.

Abstract: Misinformation detectors often rely on superficial cues (i.e., shortcuts) that correlate with misinformation in training data but fail to generalize to the diverse and evolving nature of real-world misinformation. This issue is exacerbated by large language models (LLMs), which can easily generate convincing misinformation using simple prompts. We introduce TruthOverTricks, a unified evaluation paradigm for measuring shortcut learning in misinformation detection. TruthOverTricks categorizes shortcut behaviors into intrinsic shortcut induction and extrinsic shortcut injection, and evaluates seven representative detectors across 14 popular benchmarks, along with two new factual misinformation datasets, NQ-Misinfo and Streaming-Misinfo. Empirical results reveal that existing detectors suffer severe performance degradation when exposed to both naturally occurring and adversarially crafted shortcuts. To address this, we propose the Shortcut Mitigation Framework (SMF), an LLM-augmented data augmentation framework that mitigates shortcut reliance through paraphrasing, factual summarization, and sentiment normalization. SMF consistently enhances robustness across 16 benchmarks, forcing models to rely on deeper semantic understanding rather than shortcut cues.

Truth over Tricks: Measuring and Mitigating Shortcut Learning in Misinformation Detection
Herun Wan Jiaying Wu', Minnan Luo Zhi Zengl, Zhixiong Su
'School of Computer Science and Technology, Xi Jiaotong University, China
Ministtr of Education Key Laboratory of Intelligent Networks and Network Security, China
National University of Singapore, Singapore
4 Shaanxi Province Key Laboratory Big Data Knowledge Engineering, China
wanherunCstu ..tpsstuujj.uuu. xjtu. edu. cn minnluo@x edu. cn
Abstract
Misinformation detectors often rely superficial cues (i.e., shortcuts) that cor- relate with misinformation in training data but fail to generalize to the diverse and evolving nature of real-world misinformation. This issue is exacerbated by large language models (LL.ss), which can easily generate convincing misinfor- mation using simple prompts. We introduce TRUTHOVVERTRICKS. evaluation paradigm for measuring shorteut learning in misinformation detection. TRUTHOVERTRIAA TRICKS categorizes shortcut behaviors into intrinsic shortcut induc- tion and extrinsic shortcut injection, and evaluates seven representative detectors across 14 popular benchmarks, along with two new factual misinformation datasets, NQ-Misinfo and Streaming-Mistt
unified
Empirical results reveal that existing detectors suffer severe performance degradation when exposed to both naturally occurring and adversarially crafted shortcuts. To address this, we propose the Shortcut Miti- gation Framework (SMF), an LLL--aummetted data augmentation framework that mitigates shortcut reliance through paraphrasing, factual summarization, and senti- ment normalization. SMF consistently enhances robustness across 16 benchmarks, forcing models to rely on deeper semantic understanding rather than shortcut cues.
Introduction
Real-world misinformation, such as fake news [70] and rumors [36], ap- pears in highly diverse forms. It varies sentiment, writing style, and topi- cal focus, and often adapts quickly to emerging events. The emergence of large language models (LLMs) further amplifies this diversity. With simple prompts such as "Rewrite the news article to make it more convincing". LLMs can generate persuasive misin- formation that closely mimics credi- ble content [5).
Causal Graph
Training Set
postiiie
News
Authenticity
#1
#2
#4
+5 Real
Content
Real
Fake
Real
Fake
Y
Test Set
Shartcul
Spurious Associations
#6
#7
#8
Correct
Label
Real
Fake
Real Fake
Shortcuts
Prodiction
Fake
Real
Wrong
Figure 1: Illustration of shortcut learning in misinformation
detection. As depicted in the causal graph, misinformation detectors often learn spurious associations between truth- fulness and task-irrelevant features (i.e., shortcuts) during training. For instance, features such as sentiment domain may lead detectors to infer that positive-toned or politics- related content more likely to be real. As result. models fail to generalize and produce incorrect predictions when presented with more realistic or diverse test instances.
To ensure factual integrity in this dy- namic online landscape, misinforma-
Cooresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

tion detection systems must generalize across a wide range of expressions. However, existing systems are typically trained on static datasets with limited variation in style and semanties. As a result, they tend to rely on shortcuts 12] superficial cues that are correlated with misinformation labels in the training data but are not causally informative. This reliance leads to brittle predictions that fail to generalize to novel or intentionally disguised misinformation, as illustrated in Figure ] Although recent work has begun to explore shortcut learning in this domain- -analnnnnnnggggga surface features such as specific entity mentions [71] publisher-related stylistic traits [59], these investigations are often narrow in scope, focusing on specific shortcut types or misinformation types. Fully addressing this challenge requires a systematic and comprehensive measurement framework that captures the full spectrum of shortcut behaviors across both naturally occurring and adversarially induced settings.
To this end, we propose TRUTHOVERT BRICKS,, a unified evaluation paradigm for diagnosing shorteut learning in misinformation detection systems. TRUTHÖVERTRIICE is grounded in a novel taxonomy of misinformation shortcuts, which distinguishes between two key sources: (1) intrinsic shortcut induction, which captures shortcuts that arise naturally within existing benchmarks, and (2) extrinsic shortcut injection, which introduces adversarial variations crafted by LLMs to obscure misinformation and challenge model robustness. In the intrinsic setting, we systematically investigate whether four commonly used misinformation indicators sentiment, style, topic, and perplexity may inadvertently induce shortcut reliance. In the extrinsic setting. we curate six types of LLM L-M-eeeertted reframings that simulate realistic adversarial attacks on misinformation detectors.
With TRUTHOVERTRICKS we evaluate seven detectors from three main categories of misinformation detectors (i.e., LL.M-based, LM- LM-based, and debiasing approaches) on 14 popular misinformation detection benchmarks. To assess whether models can detect misinformation that requires factual knowledge (for example, "The capital of Greece is Thessaloniki"), we construct two new benchmarks: NQ-Misinfo and Crreaming-Misinfe These are derived by distorting instances from two existing question answering (QA) benchmarks, Natural Questions (NQ) (26) and StreamingQA (11, to create factual misinformation examples. Empirical results show that current detectors experience significant and consistent performance degradation under both intrinsic shortcut induction and extrinsic shortcut injection, with accuracy dropping to 0% in the most challenging cases. Notably, misinformation detectors trained on inttimnnt-injectee data classifier, although the training objective differs significantly.
obtain reasonable capability as a sentiment
To redirect model attention from superficial shorteuts to deeper semantic signals, we propose the Shortcut Mitigation Framework (SMF), an .-guuueeeeeed framework for mitigating misinformation shortcuts and promoting more generalizable detection behavior. To better capture the diversity of real- world misinformation, our framework enhances data variation through three complementary strategies: paraphrase generation, factual summarization, and sentiment normalization. These augmentation techniques encourage models to rely on robust, content-based reasoning rather than shortcut cues, thereby improving resilience to distributional shifts and adversarial manipulations. We integrate framework with five representative misinformation classifiers and evaluate its effectiveness across all 16 benchmark datasets under extrinsic shortcut injection from the TRUTHOVERTRILE Empirical results show that our framework consistently improves model robustness against a wide range of externally injected shorteuts, demonstrating its effectiveness and general applicability.
suite.
2 TRUTHOVERTRICKS
TRUTHOVERTRICKS evaluates shortcut learning in misinformation detection through two com- plementary perspectives: (1) intrinsic shorteut induction and (2) extrinsic shortcut injection. An overview of the framework is shown in Figure 2
2.1 Intrinsic Shortcut Induction
This component of TRUTHOVVRRRRICKS investigates the impact of shortcuts that naturally emerge within existing misinformation benchmarks. Specifically, we evaluate whether four commonly used misinformation indicators, namely sentiment, style, topic, and perplexity, may inadvertently introduce shortcut biases, despite their frequent use in enhancing model performance.

---

(a) positive
Please rewrite it make more commincing.
(b) negative
Sentiment (c) neutral
TruthOverTricks
Vanilla
Evaluate Shortcut Learning in Misitformation Detection
(a) subjective
Misinformation Benchmarks
Style
neutral
Make the fword choicej of the rewsitten content (simple).
(a) sports
Intrinsic
Extrinsic Shorteut Injection
Explicit
(b) crime
Shorteut
Topic
politics
Induction
OOO
AI
You are (elderly). Please reurrite this Implicit passuge.
(a) high
Perplexity
(b) low
Indicator Classifier
LLM Injector
Figure 2: TRUTHOVERTRICKS evaluates misinformation detectors under two types of shortcuts: (1) Intrinsic Shortcut Induction, which captures spurious correlations that naturally emerge within existing benchmarks; and (2) Extrinsic Shortcut Injection, which introduces adversarially crafted variations from L.MS designed to obscure misinformation and challenge detector robustness.
Sentiment. Prior work has shown that leveraging sentiment features can improve detection performance 669 42.. However, we argue that sentiment may function shortcut, as the presence of negative positive emotion does not causally determine whether content is true or false.
Style. Misinformation is often more subjective in tone, while real news is presumed to be more neutral 23 Yet practice, especially on social platforms, real content can also adopt subjective styles to increase engagement 77,, making stylistic cues potentially misleading signal.
Topic. Many detectors struggle to generalize across diverse topics domains [29 60) suggesting that models may overfit to topical patterns rather than learning content-based signals.
Perplexity. Token distribution patterns, often captured via perplexity, are increasingly used to detect
content [39 (). This may cause detectors to learn surface-level language regularities rather than semantic truths.
potential of each indicator, we first train dedicated classifiers (e.g.. to label sentiment positive negative), as detailed in Appendix D We then create data splits where the joint distributions of the ground-truth misinformation label and the indicator label differ between training-validation and test sets. For example, the valiiaiioo set may contain mostly negative-tonee misinformation and positive-toned real news, whereas the test set reverses this pairing. This setting allows us to test whether detectors rely on shortcut cues that do not generalize. Details of the intrinsic setup and indicator statistics are provided in Appendix and Appendix respectively.
To evaluate the
2.2 Extrinsic Shortcut Injection
The advancement of LLMs introduces risks in the form of LLM-based injection attacks. Mali- cious actors may exploit LLMs to generate misinformation that embeds superficial cues, or shortcuts, to evade existing detectors. To simulate such scenarios, TRUTHOVERT TIICKS employs LLMs to explicitly or implicitly inject predefined shortcut-inducing factors into misinformation content.
We construct zero-shot prompts consisting of an instruction text Ping and input texts Pinput to guide the L.Ms.. The full list of prompts is presented in Table Table (Appendix [E.T) and representative outputs are shown in Table 15 (Appendix E.2).. The evaluation setup mirrors the intrinsic shortcut setting, as detailed Appendix
We design three injection strategies:
paraphrase, rewriting, and open-ended, to rewrite texts in general manner. Unlike the original setup, we rewrite both real and misinformation instances avoid introducing confounding label-specific artifacts. This setting evaluates whether detectors overfit generic LLM rewriting patterns. Here, Piopar includes the original text
Vanilla. We adopt three prompts from 5

---

Pamphrase
Neguive
Foenal
Hider
Original
Comples
Simple
Open-andal
Simple
Informal
Pasiiie
Complex
Young
Famale
BERTScore ROUGE-
0.2
Random .99
Original Avg. 2.94x10#
Complex 2.62x1
Simple Avg.: 10
0.7
0.0
0.5
1.0
2.0
Random
1e9
Figure 3: Token count distributions before and after Word Choice injection. The dis- tinct distribution patterns and differences in average token counts show that LLMs can ef- fectively manipulate word choices.
Figure 4: Token-level and semantic-level sim- ilarities of original-rewritten pairs. "Random" denotes that we shuffle the original-rewrittee pairs the baseline. Results show that rewrit- ten texts preserve similarity to the originals.
Explicit. This strategy instructs the LLM to incorporate predefined stylistic or linguistic attributes into the rewritten text. We select three such factors, with Piopur including both the original text and factor description f: (i) Sentiment: positive and negative; (ii) Tone: formal and informal: and (iii) Word Choice: simple and complex.
Implicit. Inspired by [ this strategy prompts the LLM to simulate content authored by individuals with specific demographic traits. It evaluates whether detectors are sensitive implicit author attributes. We consider two common dimensions, where Pinpur includes the original text 8 and author description f: (i) Age: young and elder; and (ii) Gender: male and and female.
Shorteut Injectors. We leverage the open-source Meta-Llama-3-ee Bae-llmma------nsstteee coming from this link to inject shortcuts. The temperature is set to = 0 to ensure reproducibility.
LLMs can inject predefined factors without altering content authenticity. As LLMs are known to suffer from hallucination [22), we first investigate whether they can reliably inject predefined shorteut-inducing factors. Taking Word Choice
the infini-gram fiiii score [32] to assess token commonality and visualize token count distributions before and after injection in Figure 3 The clear distributional shift indicates that LLMs can effectively modify word usage patterns. Additional analyses for other shortcut types are provided in Appendix F.1 To further validate injection fidelity, we conduct human evaluation with three expert annotators (Appendix F22). For explicit shortcut injections, the average Cohen's Kappa between the annotated ground truth and expert judgments is 0.902, confirming high agreement. In contrast, implicit injections (e.g., simulating author age gender) yield much lower average Kappa of 0.035, indicating that LLMs struggle simulate subtle author attributes. We also observe frequent refusals from the LLM to rewrite misinformation under implicit prompts (e.g.. "I cannot rewrite the passage as it contains misinformation"), likely influenced by reinforcement learning from human feedback (RLHF)
example,
Next, we examine whether the rewritten outputs preserve authenticity. We compute ROUGE-L. and BERTScore assess token-level and semantic-level similarity, respectively, of original-rewritten pairs (Figure 4 Results show high similarity scores, substantially higher than randomized baseline, suggesting that the meaning of the original content is largely preserved. Additionally, detector performance on the rewritten datasets under the standard setting (Appendix [:3 Tables [4 and 23). where only slight performance differences are observed. A complementary human evaluation (Appendix F.4) confirms that authenticity is maintained post-injection, with an average accuracy of 91.2% and Fleiss' Kappa of 0.838. We further employ an LLM-based evaluation approach similar inspirit to FactScore [38] (Appendix F.5), where GPT-4-turbo believes that on average 92.5% of the rewritten instances do not alter the authenticity.
These findings confirm the validity and effectiveness of LLM based injection attacks, establishing them as a practical and credible threat misinformation detectors.
2.3 Experiment Settings
Detectors. We apply TRUTHOVERTRICHS to evaluate seven misinformation detectors spanning three major categories:() LLMd- Detectors: We follow the prompting strategy of F3 [35

---

Table I: Accuracy of DEBERTA under four intrinsic shorteut settings. Shorteut refers evaluation using TRUTHOVERTRIICKS, while Random uses randomly sampled training set of equal size to simulate the standard setting for fair comparison. Difference shows relative performance change; "_" indicates undefined values. We highlight (in blue color) the top three datasets with the largest perfor- mance drops per shortcut. Results show that existing detectors experience significant degradation when exposed to intrinsic shortcuts.
E164
[307
2376
878
0%
Randem
00.0
85.8
9016
50-1
10310
922
100.0
18.4
85.9
16.7
Style
Shortcut
50.0
70.7
62.0
41.2
102.0
100,0
80.7
70.0
92.9
52.6
100.0
100,0
95.4
85.0
0.0
14.1
Difference
175.
1736
11155
58%
0.55
076
875
1935
2096
4776
0%F
10075
0%
0%F
10075
90%
Random
81.8
79.7
15.0
78.3
68.2
95.5
100.0
88.2
50.7
56.7
100.0
84.1
100.0
95.7
52.1
45.9
Togle
Shortcut
63.6
64.1
47.8
52.6
78.9
00.5
94.3
85.3
$5.3
57.7
100.0
8L.8
95.9
96.7
10.2
72.1
Difforence
2395
9095
3035
2395
900
05%
395
53%
5756
495
090
4956
Perplesity
Differance
0%
to prompt MISTRALV3 24 and LLAMA3 14 (ii) LM-based Detectors: We adopt BERT 9
and DEBERTA (00), using frozen embeddings followed by multi-layer perceptrons (MLPs) for classification. (iii) Debiasing Detectors: We include three representative debiasing methods: CMTR
[8]. DIsC M.. and CATCH [50). These detectors collectively cover a wide spectrum of existing approaches. Detailed descriptions of all baselines are provided in Appendix H
Evaluation Benchmarks. To comprehensively evaluate shortcut learning on real-world data, we use 14 popular misinformation benchmarks covering fake news, rumors, and disinformation, numbered
D01 through D14 in order of publication: RumourEval [8] Pheme [4], Twitter15, Twitter16
36). Celebrity, FakeNews [44), Politifact, Gossipcop (51), Tianchi published [here]] in 2022,
Multil.ingua, [25] AntiVax [19], COCO 27). Kagglel published [here] in 2024, and Kaggle2
published [here] in 2024.
To assess whether detectors can identify factually incorrect claims, we introduce two new benchmarks for factual misinformation detection: NQ-Misinfo and Streaming-Misinte labeled as D15 and D16, respectively. Inspired by [43,, both are derived from existing question answering datasets: Natural Questions (NQ) [26) and StreamingQA B1. We construct real claims by converting question-answe pairs into declarative sentences, and generate fake claims by distorting the original answers. Detailed construction steps are described Appendix B
Comprehensive dataset descriptions and statistics are provided in Appendix
3
Observations from TRUTHOVERTRICKS
3.1 Intrinsic Shortcut Induction
Existing datasets contain shortcuts irrelevant to the authenticity. We begin by examining the distributions of four commonly used indica- tors: sentiment, style, topic, and perplexity, be- tween misinformation and non-misinforration Figure 5 shows the distributions for Pheme as an example, while results for all datasets provided in Figure [4 and Table i7 Ap- pendix L.1] The distributions of fake and real news are largely similar across these factors, suggesting that they are not causally linked to authenticity. other words, detectors should not rely such attributes-for for instance, assum- ing that a news item with negative sentiment is fake, or with positive sentiment is real.
Count
300
Pilc
Red
Count
300
Rtal
200
100
Negative Nentra
Postrive
Subjectivve Style
Sentiment
Count
200
Density
100-
500
1000
1500
Topic
Perplexity
Figure 5: Distributions of sentiment, style, topic, and perplexity for fake and real instances in Pheme. The similarity between distributions suggests that these factors are not causally linked to authenticity.
5

---

Table 2: Average accuracy of baseline detectors 16 datasets under various LLM-ggnerated extrinsic shortcuts. "Original" denotes performance under the standard training setting. We report relative accuracy changes compared to the original. Results show that LLMs successfully inject predefined factors that induce shortcuts, leading to a performance drop (in blue color) in trainable detectors. Explicit shortcut injection is significantly more harmful than implicit injection. In contrast, LLM- based detectors remain robust, motivating our mitigation approach.
Shortcuts
L..M-hnsed Detectors
LMM-basd Detectors
Debissing Detectors
MISTRALY3
LLAMAS
BERT
DiBERTA
CMTR
DisC
CATCH
Original
58.8
78.1
70.9
78.2
Vanilla
12.6 (2.8%% 83
64.8 (6.9% J)
17.8
(77.8% J]
11.2 (66
22.9
(70.355 13
14.9
(81.055
16.9
(78.5% 1)
Sentiment
(3.6%) 41
52.6 (10.6% 4)
9.1
[68. 455
(20.055
196
8.9
(88.7%
Explictt
Tone
54.3 (0.756 f)
6664 (4.2%6 .)
(88.8%6
(91.6% ()
(M600%% 4)
900
(88.596 4)
(80.0% 4)
Word Choice
64.3 (0.8% 9)
06.3 (8.455 J)
7.0
(91.0%
(922.056 ()
8.9
(88.49% 43
66.8
(91.595 4)
16.1
(79.5% 4)
Implicit
Gender
64.5 (1.1%%
(1.66)
60.8 (8,0%% ()
(8.995
73.4 (6.0%%
(4.955
74.0 J)
74.8 (0.0% J)
(4.4%
(4.5%
73.7
(5.99%
(4.0%% 1)
(6.395
Existing detectors suffer from intrinsic shortcut learning. We report the performance of DE-
BERTA under both the intrinsic shortcut setting and a random control setting in Table Results for
all detectors are presented in Table 18 ( (Appendix J.I). We observe substantial performance degrada- tion under the shortcut setting, with some detectors producing systematically incorrect predictions. This indicates that models may incorrectly associate shorteut features with truthfulness.
Shortcut effects vary across datasets. In certain datasets, the distributions of specific indicators differ notably between misinformation and non-misinformation, and performance does not always decline- occasionally it even improves. This suggests that in some cases, shorteut features may coincidentally align with authenticity, reinforcing the model's s reliance on them. These indicate that detectors are prone to associating auxiliary features with authenticity, regardless of whether such associations are spurious or valid. This validates the need to further investigate the risks posed by extrinsic shortcut injection.
3.2 Extrinsic Shortcut Injection
We report the average accuracy of all detectors on 16 datasets under the standard training setting and under the TRUTHO evaluation in Table [2] Full per-dataset results provided in
Tables 19 and in Appendix J.2 Detailed experiment settings are provided in Appendix J
LLM-based detectors are robust to shortcuts. While LLMs without fine-tuning are generally limited their ability to detect misinformation, we observe that they are substantially more robust to shortcut- based attacks. This suggests that LLMs are less prone to relying on superficial correlations and can instead capture deeper semantic signals. These findings support our motivation use LLMs in SMF (proposed in $4) to rewrite data in a way that removes shortcut cues while preserving authenticity.
Explicit injection attacks severely degrade detector performance. Compared to the standard setting, performance under explicit shortcut injection drops by average of approximately 80%, with the most extreme case reaching a 92.0% drop. Even under the Vamilla rewriting strategy, which alters prompt formats without explicitly targeting shortcut attributes, detectors exhibit notable performance degradation. While debiasing methods outperform standard L.-baaed detectors, they too experience substantial drops. These results indicate that addressing shorteut reliance purely through model design is insufficient, reinforcing the need for data-centric solutions such as our LLM-based data augmentation framework introduced in $4
Detectors are less affected by implicit injection. In contrast to explicit shortcut attacks, implicit injection leads to minimal performance degradation. This aligns with our earlier findings in $2.2 where we observed that LLMs struggle to reliably simulate author-specific attributes such as age or gender. Consequently, the injected features are less effective in inducing shortcut learning. It's notable that in Table 2 we retained instances even when the L.MSs partially or fully refused to follow the injection prompt. This choice preserved label distributions and ensured consistent datasct size

---

Loss Aucui
Train
Validation Text
Accuracy
7.5
70
Senliment
2.5
Word Choice
Gender
60
50
70
Training Epoch
Percentage (%)
Figure 6: Training loss and accuracy of DE- BERTA on AntiVax under the shorteut set- ting, averaged over five runs. Performance degrades over time, indicating that the detec- tor increasingly overfits to shortcuts rather than learning authenticity.
Figure 7: Performance of DEBERTA on An- tiVax as the ratio of shorteut-o--autheniitty associations increases. Accuracy rises sharply with shortcut prevalence, approaching 100% when the ratio nears 100%, indicating strong reliance on shortcuts.
65
Before
After
SST-2
3
.5
45
65
26
MTEB
45
D01
D02
DIO
D11
DI2
Figure 8: Performance of DEBERTA on two sentiment classification datasets before and after training
under the Sentiment injection attack. Accuracy improvements indicate that detectors mistakenly learn sentiment as proxy for authenticity, confirming the presence of shortcut learning.
across original and shorteut conditions, allowing for direct comparison. To rigorously assess the influence of these refusals, we conducted an additional analysis in Appendix K
Misinformation detectors readily learn shortcuts. Previous results demonstrate that detector performance drops sigmificantly under shortcut settings. To further investigate what detectors actually learn during training in such conditions, analyze their learning dynamics and transfer behavior.
We begin by tracking training loss and test accuracy under the shorteut setting, as shown in Figure Test performance deteriorates as training progresses, indicating that detectors overfit to spurious correlations rather than learning authenticity. To examine what information the models acquire, we evaluate detectors trained under the Sentiment injection setting on two external sentiment classification datasets: SST-2 [53] and MTEB [40). As shown in Figure S accuracy on these sentiment tasks increases by up to 26.7% after shortcut-based training, suggesting that detectors trained on sentiment-injected misinformation acquire nontrivial sentiment classification ability.
We also study how performance varies as the ratio of
associations increases, with results presented in Figure 7 As this ratio rises, detector accuracy improves dramatically, approaching 100% in most settings except for implicit injection, where LLMs struggle to simulate author attributes. These findings confirm that detectors learn to rely on shorteut features, even when such associations are spurious or non-causal.
Overall, these results indicate that detectors do not learn internalize correlations between shortcut features and labels, regardless of whether these associations reflect genuine truthfulness.
authenticity per se. Instead, they

---

Table 3: Average accuracy of baseline detectors 16 datasets under extrinsic shorteut settings, with and without data augmentation from our proposed shortcut mitigation framework (SMF). "w/o aug." denotes original performance under shortcut attacks. We report relative improvements and highlight (in blue color) the best results. SMF effectively mitigates shortcut learning, with different variants excelling under different types of injected shortcuts.
LM-based Detectors
Debiasing Detectors
Shorteut
Augmentation
BERT
DEBERTA
CMTR
DisC
CATCH
wro aug.
17.3
11.2
22.9
14.9
Paraphrase
38.8 (123.7% f)
30.2 (169 0% f)
43.2 (89.0% t)
38. (159 f)
13.0 (153.7% f)
Vanilla
Summary
44.1 (154.4% 11
86.2 (222 89%
47.3 (106.8% 11
44.2
(197 0% 1)
49.0 (189.4% 13
Neutral
29.9 (72.2% 1)
21.3 (89.4% T)
35.7 (56.0% 1)
28.1 (88.9% T)
25.7 (51.9% 1)
w/o aug.
9.1
19:2
10.7
8.9
17.0
Sentiment
Paraphrase
14.7 12..0%11 1)
22.7 (18.0% 11)
18.0 (68.6% t)
14.4 (62.8% +)
23.9 (40.6% 1)
Summury
40.0 (340. ()
45.4 (136.7%) 1)
43.2 (304.196
40.0 0 (350 8%
$1.0 (199.4%
Neutral
34.9 (284.6% T)
44.0
(129.4% 1)
40.0 (274.3% f)
36.1 (307.3% 3% f)
47.6 (179.45 1)
aug.
8.8
6.7
10.8
Tome
Paraphrase
Summary
68 (567 .3% 1)
55.9 (536.7% 7%
62.5 (679 8% 1)
(621 9%%
62 (479. 9/6 1)
58.4 (441.2%
56.3 (616.09 f)
56.4
(528.2%) 2%
63.1 (322.88 8%
62.9 (321 00E f)
Neutral
$7.1 (549.5% f)
50.6
(651.9% 1)
$9.8 (450.5% f)
57.4 (539 4%% 1)
60.5 (304.9% 1)
w/o aug.
6.4
Paraphicase
63 1 (800.500 5% ()
56.9 (783 19% 1)
65.3 3 (633 6% 1)
63.3 9 (838 4%% f)
67.7 (321 6% 13
Word Choice
Summary
(542.0%
33.6
.4%
47.8 (437.6%
45.7 (670.2%
57.5 (258. 1%
Neutral
49.1 (598.1% 1)
41.5
(542.8%% 1)
52.9 (494.3% f)
49.7 (629.3%% 1)
58.9 (286.99% f)
w/o aog.
73.4
74.6
73
73.7
78.1
Paraphrase
71.1 (3.156 4)
71.9 (3.7% L)
71.4 (3.0% 4)
71.7 (2.7% 1)
71.5 (2.2%, L)
Summary
(2.3%
69.5
72.1 (1.5%%
72.1 (2.3%
71.0 (2.9%.
Neutral
71.8 (2.2% L)
72.2 (3.2% L)
72.4 (1.6% 1)
13.3 (0.6% L)
72.6 (0.8% L)
w/o aug.
74.3
74.0
73.6
74.8
74.6
Gender
Paraphrase
Summary
71.6 (3.8%
(5,2%
71.9 (2.8% L)
(5.6%%
71.3 (3.1%
70.6 (4.0%
72 3 (3.2% J)
(5.0%
71.2 (1.5%)
71.8 (3. 7%
Neutral
71.1 (4.456 4)
72.8 (1.6% L)
70.7 (3.9% 1]
72.3 (3.3%
71.9 (3.6% L)
Shortcut learning erodes detector credibil-
Llama3
DeBERTa Ererr: 10.75
DisC Eeroc: 10.20
ity. A robust misinformation detector must not only make accurate predictions but also provide well-calibrated confidence estimates to support reliable content moderation. We employ the ex-
Original
Accumey
8e
Error: 36.48 Le
pected calibration error (ECE) [17 quantify calibration of misinformation detectors. Fig- ure shows the calibration curves of detectors before and after shortcut injection. Calibration degrades significantly for most detectors under attack, except for LLM--bsse detectors, which remain relatively stable. Moreover, observe that detector confidence increases even as pre- diction quality declines, further confirming that detectors are learning shortcuts rather than se- mantic authenticity.
Shortcul
Accurncy
222 22.22 S4
Error: 38.27
Eror: 82.52
Beror: $5.04
0.5 666 0.7 0.8 0.9
05 06 67 88 09 Confidence
65 06 07 08
Confidence
Confidence
Figure 9: Calibration of detectors before and af- ter explicit injection attacks. "Error" indicates the expected calibration error (x 100), where lower values denote better calibration. Results show that explicit injection attacks substantially degrade cal- ibration quality.
Shortcut Mitigation Framework (SMF)
Our observations from TRUTHÖVERTRICKK reveal that misinformation detectors often rely on spurious shortcut features rather than learning authentic signals, which not only harms generalization but also degrades calibration and interpretability. To address this, we introduce the Shortcut Mitigation Framework (SMF), a simple yet effective data-centric approach designed to reduce shortcut reliance and enhance model robustness.
4.1 Methodology
Given that L.Mss can inject shortcut-inducing features, we hypothesize they can also be leveraged to remove factors irrelevant to authenticity. To this end, we propose SMF, an L.MM-based data augmentation method that rewrites texts prior both training and inference. SMF is model-agnostic
8

---

and can be applied to any detector. We design three variants and provide their prompts in Table (Appendix G.I and example outputs in Table (Appendix G.2)
Paraphrase. We reuse the same prompt as the paraphrase variant from the Vanilla setting in the extrinsic shortcut injection.
Summary. We ask LLMs to summarize given text and remove irrelevant factors. After summa- rizing, we expect the text will only contain information about authenticity.
Neutral. This variant targets specific shortcut (e.g., sentiment). Since LLMs can inject a desired factor, we ask them to overwrite existing shorteut by injecting neutral alternative (e.g., neutral sentiment). This re-balances distributional mismatches between training and test splits, potentially mitigating shortcut reliance.
4.2 Results
We first verify that SMF can successfully remove injected shortcuts, with supporting results provided in Appendix [.. Table presents the average accuracy of detectors across 16 datasets with and without SMF augmentation under shortcut injection. Full results for each variant are available in Tables 21 22 and 23 (Appendix [.2p. We exclude LLM-basee detectors from this evaluation in the main text since they are already robust to shortcut attacks, and we provide the results of LLM-based detectors in Appendix
SMF significantly improves detection performance. All three augmentation variants lead to notable performance gains, with improvements reaching up to 838.4 under explicit injection attacks. This demonstrates the effectiveness of data-centric mitigation over model-side adjustments. Debiasing detectors outperform standard LM-based ones, and combining debiasing with SMF further enhances robustness. Notably, performance decreases slightly under implicit injection, likely because LLM rewriting may subtly alter authenticity when no shortcut is present.
Each variant excels against specific shortcut types. The Paraphrase variant performs best against the Word Choice shortcut, while Summary consistently yields the highest overall improve- ment. These results suggest that summarizing texts to retain only authenticity-eelevat content is particularly effective mitigation strategy.
Combining SMF variants further enhances robustness. Since each SMF variant targets different shortcut types, we explore whether combining them yields additive benefits. As shown in Table 4 joint augmentation leads to further performance gains up to 54.7% improve- ment, demonstrating that integrated augmenta- tion more comprehensively remove diverse shortcut cues.
Table 4: Performance of DEBERTA the Anti- Vax dataset using single vs. multiple data augmen- tations. Combining multiple augmentations further mitigates shortcut learning and leads to improved accuracy.
Shoricuus
Neutral
Seutta++Summary
Summary
Summary-NNeut Neuural
Vanilla Tone
33.1
f)
(22.6%
(29.7%
5 Related Work
Shortcut learning refers to the phenomenon where neural models achieve high performance on standard benchmarks by exploiting spurious correlations, rather than learning the underlying task 12 37 In the context of misinformation detection, we argue that detectors often rely superficial features (e.g., sentiment, source, or style) rather than identifying factual authenticity. Prior work has examined various forms of bias in this domain, including entity bias 67], source bias [54] B3), and domain bias 122 66). However, systematic evaluation of shortcut learning in misinformation detection remains largely unexplored. This paper aims to fill that gap by providing comprehensive evaluation paradigm and mitigation framework. Additional related work on misinformation detection discussed in Appendix A to further contextualize contributions.

---

Conclusion
We propose TRUTHOVERTRRIKKS unified evaluation paradigm for diagnosing shorteut learning in misinformation detection systems. It encompasses two key components: intrinsic shortcut induction, which captures spurious correlations naturally present in existing benchmarks, and extrinsic shortcut injection, which introduces adversarially crafted shortcut variations using LLMs. Using TRUTHOVVRRRRICKS we evaluate seven representative detectors across 16 misinformation detection benchmarks. Our findings reveal that current detectors suffer substantial performance degradation when exposed to both shortcut types. To address this, we introduce SMF, a unified LLM-basee data augmentation framework designed to mitigate shortcut learning. SMF significantly improves robustness across diverse shortcut types, demonstrating the effectiveness of data-centric mitigation strategies in enhancing generalization and authenticity awareness in misinformation detection.
Acknowledgements
This work is supported in part by the National Natural Science Foundation of China (No. 62192781, No. 62272374), the Natural Science Foundation of Shaanxi Province (No. 2024JC-ICQN-62 the State Key Laboratory of Communication Content Cognition under Grant No. A202502, the Key Research and Development Project in Shaanxi Province (No. 2023GXLH-024), This research supported by the Ministry of Education, Singapore, under its MOE AcRF TIER 3 Grant (MOE- MOET32022-0001), The China Scholarship Council also supports this research.
References
[1] Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, (33333337-31, 2023.
[2] Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang. and Yue Zhang. Fast-detectgpt: text via conditional probability curvature. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Efficient zero-shot detection of
[3] Yuyan Bu, Qiang Sheng, Juan Cao, Peng Qi, Danding Wang, and Jintao Li. Fakingrecipe: Detecting fake news short video platforms from the perspective of creative process. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1351-1360, 2024.
[4] Cody Buntain and Jennifer Golbeck. Automatically identifying fake news in popular twitter threads. In 2017 IEEE international conference on smart cloud (smartCloud) pages 208-215. IEEE, 2017.
[5] Canyu Chen and Kai Shu. Can IIm- -lm-generated misinformation be detected? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OeenReview.net, 2024.
[6] Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. Complex claim verification with evidence retrieved in the wild. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pages 3569-3587, 2024.
[7] Giovanni Da San Martino, Seunghak Yu, Alberto Cedrno, Rostislav Petrov, and Preslav Nakov. Fine- Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNL.P-ICCILP) pages 5636-5646. 2019.
[8] Leon Derezynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. Semeval-2017 task 8: Rumoureval: Determining rumour veracity and support for rumours. Proceedings of the Ith International Workshop on Semantic Evaluation (SemEvvl-2011), pages 69-76, 2017.
10

---

[9] Jacob Devlin, Ming- wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguisties: human language technologies, volume (long and short papers), pages 4171 4171-4186, 2019.
[10] Yiqi Dong, Dongxiao He, Xiaobao Wang, Youzhu Jin, Meng Ge, Carl Yang, and Di Jin. Unveiling implicit deceptive patterss in multi-modal fake news via seuro-symbolic reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 8354-8362,
2024.
[11] Shaohua Fan, Xiao Wang, Yanhu Mo, Chuan Shi, and Jian Tang. Debiasing graph neural networks via learning disentangled causal substructure. Advances in Neural Information Processing Systems,
2022.
[12] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning deep neural networks. Nature Machine Intelligence, 2(1 (6666)6666666) 66366666,, 2020.
[13] Haisong Gong, Weizhi Xu, Shu Wu, Qiang Liu, and Liang Wang. Heterogeneous graph reasoning for fact checking over texts and tables. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 100-108, 2024.
[14] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint ar arXiv:2407.21783 2024.
[15] Karish Grover, SM Angara, Md Shad Akhtar, and Tanmoy Chakraborty. Public wisdom matters! discourse-aware hyperbolic fourier co-attention for social text classification. Advances in Neural Information P Processing Systems, 35:9417-9431, 2022.
[16] Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, and Hao Peng. Language models hallucinate, but may excel at fact verification. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1090-1111, 2024.
[17] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinherger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
[18] Philipp Hartl and Udo Kruschwitz. Applying automatic text summarization for fake news detection. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. pages 2702-2713, 2022.
[19] Kadhim Hayawi, Sakib Shahriar, Mohamed Adel Serhani, Ikbal Taleb, and Sujith Samuel Mathew. Anti-vax: novel twitter dataset for covid-19 vaccine misinformation detection. Public health, 203:23-30, 2022.
[20] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: decoding-enhancee bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[21] Beizhe Hu, Qiang Sheng, Juan Cao, Yuhui Shi, Yang Li, Danding Wang, and Peng Qi. Bad actor, good advisor: Exploring the role of large language models in fake news detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22105-22113, 2024.
[22] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55,, 2025.
[23] Caio Libanio Melo Jeronimo, Leandro Balby Marinho, Claudio EC Campelo, Adriano Veloso, and Allan Sales da Costa Melo. Fake news classification based subjective language. In Proceedings of the st International Conference on Information Integration and Web-based Applications & Services, pages 15-24, 2019.

---

[24] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile 2023.
Saulnier, et al. Mistral 7b, arXiv preprint
[25] Ashkan Kazemi, Zehua Li, Verónica Rosas, Scott A. Hale, and Rada Mihalcea. Matching tweets with applicable fact-checks across languages. In Amitava Das, Tanmay Chakraborty, Asif Ekbal, and Amit P. Sheth, editors, Proceedings of the Workshop on Multi-Modal Fake News and Hate-Speech Detection (DE-FACTI) 2022) co-located with the Thirty-Sixth AAAI Conference on Artificial Intelligence AAAI 2022), Virtual Event, Vancouver, Canada, February 27, 2022, volume 3199 of CEUR Workshop Proceedings. CEUR-WS.org. 2022.
[26] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019.
[27] Johannes Langguth, Daniel Thilo Schroeder, Petra Filkuková, Stefan Brenner, Jesper Phillips, and Konstantin Pogorelov. Coco: an annotated twitter dataset of covid-19 conspiracy theories. Journal of Computational Social Science, 6(2):443-444, 2023.
[28] Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto. Autobencher: Towards declarative benchmark construction. In The Thirteenth International Conference on Learning Representations, 2025.
[29] Yupeng Li, Haorui He, Jin Bai, and Dacheng Wen. Mefend: A multi-source benchmark dataset for chinese fake news detection. In Proceedings of the ACM Web Conference 2024, pages 4018 4027, 2024.
[30] Ying-Jia Lin, Chun- Lin, Chia-Jen Yeh, Yi-Ting Li, Yun-Yu Yu Hu, Chih-Hao Hsu, Mei-Feng Lee, and Yu Kao. Cfever: A chinese fact extraction and verification dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18626-18634, 2024.
[31] Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien De Masson D' Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, al. Stream- ingqa: A benchmark for adaptation to new knowledge over time in question answering models. In International Conference on Machine Learning, pages 13604-13622. PMLR, 2022.
[32] Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini- gram: Scaling unbounded n- g-gram language models to a trillion tokens. In First Conference on Language Modeling, 2024.
[33] Xuannan Liu, Zekun Li, Pei Pei Li, Huaibo Huang, Shuhan Xia, Xing Cui, Linzhi Huang, Weihong Deng, and Zhaofeng He. Mmfakebench: A mixed-source multimodal misinformation detection benchmark for Ivims. In The Thirteenth International Conference on Learning Representations, 2025.
[34] Zhiwei Liu, Kailai Yang, Qianqian Xie, Christine de Kock, Sophia Ananiadou, and Eduard Hovy. Raemollm: Retrieval augmented Ilms for cross-domain misinformation detection using in-context learning based emotional information. arXiv preprint arXiv:2406.1 1093, 2024.
[35] Jason Lucas, Adaku Uchendu, Michiharu Yamashita, Jooyoung Lee, Shaurya Rohatgi, and Dongwon Lee. Fighting fire with fire: The dual role of lims in crafting and detecting clusive disinformation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14279-14305, 2023.
[36] Jing Ma. Wei Gao, and Kam-Fai Wong. Rumor detection on twitter with tree-structured recursive neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1980-1989, 2018.
[37] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina I Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1-35, 2021.
12

---

[38] Sewon Min, Kalpesh Krishna, Xinxi Lyu. Mike Lewis, Wen-tau Yih, Pang Koh, Mohit lyyer. Luke Zettlemoyer, and Hannanch Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076-12100, 2023.
[39] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot prcchine-generatee text detection using probability curvature. In Interna- sional Conference on Machine Learning, pages 24950-24962. PMLR, 2023.
[40] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014 2037, 2023.
[41] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 11:27730-277455 2022.
[42] Tsung-Hsure Pan, Chung-Chi Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. Enhancing society- undermining disinformation detection through fine-grained sentiment analysis rreffintuning. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1371-1377,
2024.
[43] Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min- Kan, and William Wang. On the risk of misinformation pollution with large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1389-1403, 2023.
[44] Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. Automatic detection of fake news. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3391-3401, 2018.
[45] Zubair Qazi, William Shiao, and Evangelos E Papalexakis. Gpt-generated text detection: Benchmark dataset and tensor-based detection method. In Companion Proceedings of the ACM on Web Conference 2024, pages 842-846, 2024.
[46] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019.
[47] Vipul Raheja, Dhruv Kumar, Ryan Koo, and Dongycop Kang. Coedit: Text editing by task- specific instruction tuning. In Findings of the Association for Computational Linguisties: EMNLP 2023, pages
2023.
[48] Lanyu Shang, Yang Zhang. Zhenrui Yue, YconJung Choi, Huimin Zeng, and Dong Wang. A domain adaptive graph learning framework to early detection of emergent healtheare misin- formation social media. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, pages
1421, 2024.
[49] Qiang Sheng, Juan Cao, Xueyao Zhang, Rundong Li, Danding Wang. and Yongchun Zhu. Zoom out and observe: News environment perception for fake news detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4543 4556, 2022.
[50] Paras Sheth, Raha Moraffah, Tharindu S Kumarage, Aman Chadha, and Huan Liu. Causality guided disentanglement for cross-pla ploss-platfomm hate speech detection. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 626-635, 2024.
[51] Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big data, 8(3):711-88, 2020.
[52] Amila Silva, Ling Luo, Shanika Karunasekera, and Christopher Leckie. Embracing domain differences fake news: Cross-domain fake news detection using multi-modal data. In Proceedings of the AAAI conference artificial intelligence, volume 35, pages 557-565, 2021.
13

---

[53] Richard Socher, Alex Perelygin. Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pages 1631-1642, 2013.
[54] Jinyan Su, Terry Yue Zhuo, Jonibek Mansurov, Di Wang, and Preslav Nakov. Fake news detectors are biased against texts generated by large language models. arXiv preprint NDt::2309008674 2023.
[55] Ling Sun, Yuan Rao, Yuqian Lan, Bingean Xia, and Yangyang Li. Hg-sl: Jointly learning of global and local user spreading behavior for fake news early detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5248-5256, 2023.
[56] Lingfeng Tan, Yunhong Wang, Junfu Wang, Liang Yang, Xunxun Chen, and Yuanfang Guo. Deepfake video detection via facial action dependencies estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5276-5284, 2023.
[57] Herun Wan, Minnan Luo, Zhixiong Su, Guang Dai, and Xiang Zhao. On the risk of evidence pollution for malicious social text detection in the era of LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 9731-9761, Vienna, Austria, July 2025. Association for Computational Linguistics.
[58] Bo Wang, Jing Ma, Hongzhan Lin, Zhiwei Yang. Ruichao Yang, Yuan Tian, and Yi Chang. Explainable fake news detection with large language model via defense among competing wisdom. In Proceedings of the ACM on Web Conference 2024, pages 2452-2463, 2024.
[59] Jiaying Wu, Jiafeng Guo, and Bryan Hooi. Fake news in sheep's s clothing: Robust fake news detection against IIm-empowered style attacks. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 3367-3378, 2024.
[60] Jiaying Wu and Bryan Hooi. Probing spurious correlations in popular event-based rumor detection benchmarks. In Machine Learning and Knowledge Discovery in Databases- European Conference, ECML PKDD 2022, Grenoble, France, September 19-23, 2022, Proceedings, Part II. pages 274-290. Springer, 2022.
[61] Liang Xiao, Qi Zhang, Chongyang Shi, Shoujin Wang, Usman Naseem, and Liang Hu. Msynfd: Multi-hop syntax aware fake news detection. In Proceedings of the ACM on Web Conference 2024, pages 4128 4137, 2024.
[62] Xiaofei Xu, Ke Deng, Michael Dann, and Xiuzhen Zhang. Hamnessing network effect for fake news mitigation: Selecting debunkers via self-imitation learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 22447-22456, 2024.
[63] Ruichao Yang, Wei Gao, Jing Ma. Hongzhan Lin, and Zhiwei Yang. Wsdms: Debunk fake news via weakly supervised detection of misinforming sentences with contextualized social wisdom. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1525 1525-1538, 2023.
[64] Ruichao Yang, Jing Ma, Wei Gao, and Hongzhan Lin. Lim-enhanced multiple instance learning for joint rumor and stance detection with social context information. ACM Transactions Intelligent Systems and Technology, 2025.
[65] Shu Yin, Peican Zhu, Lianwei Wu, Chao Gao, and Zhen Wang. Game: an unsupervised method for fake news detection using graph autoencoder with masking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 347-355, 2024.
(66) Zhenrui Yue, Huimin Zeng, Ziyi Kou, Lanyu Shang, and Dong Wang. Contrastive domain adaptation for early misinformation detection: A case study on covid-19. In Proceedings of the 31st ACM international conference on information & knowledge management, pages 2423 2433, 2022.
[67] Fanrui Zhang, Jiawei Liu, Jingyi Xie, Qiang Zhang. Yongchao Xu, and Zheng-Jun Zha. Escnet: Entity-enhanced and stanee checking network for multi-modal fact-checking. Proceedings of the ACM Web Conference 2024, pages 2429-2442 2024.
14

---

[68] Litian Zhang, Xiaoming Zhang, Ziyi Zhou, Feiran Huang, and Chaozhuo Li. Reinforced adaptive knowledge learning for multimodal fake news detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 16777-16785, 2024.
[69] Xueyao Zhang, Juan Cao, Xirong Li, Qiang Sheng, Lei Zhong, and Kai Shu. Mining dual emotion for fake news detection. In Proceedings of the web conference 2021, pages 1465-3476 2021.
[70] Xinyi Zhou and Reza Zafarani. A survey of fake news: Fundamental theories, detection methods, and opportunities. ACM Compul. Surv. 53(5), September 2020.
[71] Yongchun Zhu, Qiang Sheng, Juan Cao, Shuokai Li, Danding Wang, and Fuzhen Zhuang. Generalizing the future: Mitigating entity bias in fake news detection. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information 2022.
Retrieval, pages
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction describe the motivation to propose TRUTHOVER- TRICKS. It also mentions the most important quantitative results. By describing the key points of the paper, the scope reflected.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided the details of datasets and detectors in the Appendix. We have also provided anonymous resources.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
17

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have provided anonymous resources.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided related information including dataset statistics, hyperparam- eters, and training-test splits the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For each trainable baseline of the main results (Tables and 3), each five times and report the best performance. We do not report the variance since the quantitative results significant. For other experiments, have reported the variance if is calculable, such as Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have reported related information in the Appendix. Meanwhile, have provided related resources for this paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't i make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c https//cccccccccccccccciie..eeee
Answer: [Yes]
Justification: This paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have claimed the positive societal impacts in the abstract and Introduction. Mcanwhile, we have claimed potential negative socictal impacts in the Appendix Ethics Statement.
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We have claimed it in the Appendix Ethies Statement.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have provided information of existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not using crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We have provided the settings and prompts of employed LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettts://neurre http://ariiiacccmomeeeeeeeee//// cc//oonferencee/22255/LI. for what should or should be described.
22

---

Limitation
This paper aims explore shortcut learning misinformation detection systematically. Specifically, we focus on exploring shortcut learning in textual content-based misinformation detection with datasets including textual content and detectors analyzing texts. There are detectors employing additional information, such as social context, where the additional information may also induce shortcuts.
We have tried our best to cover mainstreaming misinformation datasets and detectors (including 16 datasets and 7 detectors). However, as proven in the main text, existing datasets contain diverse and various shortcuts, and LLMs could inject numerous shortcuts. is impossible to cover every shortcut, every injection attack, and every mitigation method. Thus, explore four representative shortcuts in existing datasets, six representative attack methods, and three representative mitigation methods.
Ethics Statement
Identifying misinformation on social platforms ensures online safety. This paper aims to explore shortcut learning in misinformation detection. We have proven that detectors suffer significant performance drops under the shortcut setting and have designed six LLM injection attacks. The research findings and resources inevitably increase the risks of dual use. We aim to mitigate such dual use by employing controlled access to our research data, trying our best to ensure that the data is only employed for research puroses. Mcanwhile, our findings illustrate that detectors vulnerable under the shortcut setting. Thus, we argue that the decision of automatic detectors should be considered experts.
initial screen, while content moderation decisions should be made with related
Meanwhile, we employ LLMs to inject predefined factors into texts, while we do not directly employ LLMs to generate misinformation. We also argue that LLMs should not be employed to generate misinformation, and researchers should make an effort to limit the misuse of LLMs. Meanwhile, due to the inherent social bias and hallucinations biased content. We emphasize that the data can only be used for research purposes.
LLMs, the texts after injection inevitably contain
A Related Work
Misinformation Detection Identifying misinformation (fake news, disinformation, rumor, etc.) is critical ensure cybersecurity. The most common misinformation detectors analyze news content including text [3) images [67] 68). or vidcos [56]
3 to identify misinformation. They also
extract additional information such as sentiments 34 topics [48), or patterns [10] to enhance
performance. However, we argue that these factors will induce shortcuts, where detectors may wrongly associate these factors with authenticity. Only few works have explored the shorteuts (biases) in misinformation detection as discussed in $5) Thus, this paper systematically explores shortcut learning in misinformation detection, bridging the gaps.
Besides analyzing the news content, detectors would employ social context (evidence) [5] 6 such user profiles [55 62, user reactions [63]
enhance detection performance. However, this paper does not focus on exploring this type of detector because it has various irrelevant factors (in content or social context), and we do not aim improve the performance.
environments (49 65
Recently, the advances of LLMs have brought opportunities and risks for misinformation detection. LLMs can be employed to enhance misinformation detection [6 21 58, 64). On the other hand, LLMs can be leveraged to generate misinformation 5 or attack existing detectors [59] [7]. Thus, faced with the misuse of LLMs, explored LLM injection attacks for shorteut learning in this paper.
have been proposed to support the development of misinformation detectors. However, these datasets are not designed for shortcut learning. This paper has explored M.-s-ba injection attacks, providing useful resources to promote the development of the misinformation detector against shortcut learning.
Meanwhile, numerous datasets 29
45
30
23

---

B NQ-Misinfo and Streaming-Misinfo
Dataset Construction Existing misinformation datasets are mostly written by humans, containing numerous factors not causally related to authenticity. For example, authors might express their sentiments or attitudes the news, which might cause the wrong associations between authenticity and factors. Thus, we propose two misinformation datasets that try to exclude other factors. Namely, detectors should learn fact-related knowledge to conduct the right identifications on the datasets.
Inspired by 443 we employ two question-answering answering datasets, NQ 26 and Streaming 3 to construct the novel misinformation datasets.
An instance from the question-answering datasets contains a question y and a correct answer dr- To construct misinformation, employ LLMs Satta-llamaa3--B-Innitt answer aw. We prompt LLMs in a zero-shot format using the following prompt from [43]
generate the wrong
Generate false answer to the given question. It should be of short (less than five words in general) and look plausible, compared to the reference answer.
Question:
Question
Reference Answers:
Answer
So far, we have obtained a correct pair (q, a,) and a wrong pair (q, aw). Next, we construct mis- information and real ones based these pairs. Unlike 43) who employ LLMs to only generate misinformation, we generate misinformation and real ones to avoid other factors (cspecially halluci- nations that come from LLMs) based rules. We first filter in questions beginning with "When", "Where", "What", and "Who". We then fill in answers (correct or wrong) in questions and rewrite questions into declarative sentences (real misinformation) based on grammar rules. For example, we transfer ("When was the last time anyone was on the moon?", "December 1972") into "The last time anyone was on the moon in December 1972." Sentences generated based rules inevitably make grammar mistakes, thus we employ CoEdIT [47) to fix grammatical errors with prompt "Fix grammatical errors in this sentence: .[] We process the above example and obtain "The last time anyone was on the moon was in December 1972."
Dataset Analysis We present the basic statistics of these two datasets in Appendix !
We further employ AutoBencher (28 to quantitatively evaluate the proposed datasets, where we employ Difficulty and Separability metrics. We consider the selected detectors M and NQ-Misinfo and Streaming- Miiiinf as D... For NQ-Misinfo, the Difficulty is 0.340 and Sepurability is 0.025; For Serraming-Misintee the Difficulty 0.385 and Separability 0.028.
Detector Performance Although existing detectors achieve remarkable performance on standard benchmarks, they have the following two limitations, which inspire us to explore shortcut learning in misinformation detection.
We first evaluate existing detectors on our proposed datasets NQ-Misinfo and Streaming-Misinno and present the results in Figure I0, Compared to the standard dataset AntiVax, detectors can not successfully identify misinformation in datasets requiring fact-related knowledge, where the performance drops by up to 39.8%. Meanwhile, detector performance on the datasets is only slightly higher than random guessing (50%). Thus, we can conclude that existing detectors cannot identify misinformation requiring fact-related knowledge.
To further evaluate the generalization ability of existing detectors, train DEBERTA on a specific dataset and evaluate it on another, where Figure 11 presents the results. It illustrates that the generalization ability of existing detectors is limited. We speculate that detectors do not learn the authenticity but learn the associations between authenticity and other factors. Thus, when the joint distributions of authenticity and other factors change (other datasets), detectors will make the wrong judgments.
C TRUTHÖVERTRICKS Settings
Model checkpoints from this link
24

---

Test Dinasct
NQ Misano 14
[5
A T
90
70
-00
Performance Decline (%)
Random
Train Dalaset
30
AuxNex
Stremiing Misiad
81.%%
17
90
40
70
Random
DeBERTa
BERT
CATCH
CMTR
DisC
The performance of existing detectors on a standard dataset AntiVax and our proposed datasets NQ-Misinfo and Streaming-Misinfo Detectors fail achieve acceptable performance datasets requiring fact-related knowledge and are only slightly better than random guessing.
Figure 10:
Der
De2
Do
De
Dia
p
Di4
DIA
D16
Arc
Figure 11: The performance of DEBERTA under out-of-dataset settings. We train the detector on a specific dataset and evaluate it on another. Detectors suffer from significant performance drop.
Here we formally define the standard setting of misinformation detection and the shortcut settings, where Figure 12 presents an overview of them.
Tminnig
Instance Posttive
Set Sample Space
Sample Space
Postrive
Misinformation Detection In this paper, focus on content-based misinformation detec- tion. Namely, the data instance is formalized as = (t., y. where f denotes the textual content and y {0, 1} denotes the misinformation label (0 for real and for fake). The misinformation detector f() with learnable parameters aims to judge a piece of text á = f(t 0), where ý denotes the prediction, and 0 is optimized to maximize the detection performance.
Rudl
Paloz Roal
Pake
Negatre Setting
Nogetiie
(b) Settigg
Figure 12:
The overviews of standard setting and the shortcut setting. The horizontal axis represents authenticity, while the vertical axis represents other factors (taking positive and negative sentiment as an example). The main difference is that the short- cut setting induces other factors, and the joint distri- bution of authenticity and other factors different between the vaaliaanvlliittion set and the test set.
Standard Setting For each dataset (d: = (ti, yi )=) we first randomly partition the dataset into training, validation, and test sets D Dainning Dratidationn and Drear- Due to the random partitioning, the data distributions in each set are similar. Then we optimize detectors on the train-
ing set, select the best detector statements on the validation set, and report the performance on the test set.
Intrinsic Shorteut Induction Here we first introduce the other factor (sentiment, style. topic, perplexity) e into each data instance = {t, e, y. We then partition the datasct into training, validation, and test sets based on the joint distribution P(e, y). The joint distribution of the training set is similar to the validation set but different from the test set. For example, the training-validatioo sets contain instances with {e 0, 0), fe 1, 1), but the test contains instances with {e te=1l 1. = 0), fe 0.y = 1}. In this setting, the factor is obtained by classifiers introduced in Appendix D
Extrinsic Shortcut Injection The only difference from intrinsic induction setting is that employ LLMS to inject predefined factor e into each instance d. Namely, in intrinsic shortcut induction, group instances into dy {t, e 0. y} and d1 {t, 1, v. but here, we generate instance
25

---

du fto. 0, y and d1 = fti, 1, y base on d = ft, e, The rewritten è is rewritten by LLMs based on and we have proven that the authenticity y does not alter after rewriting.
SMF Setting The only difference from the extrinsic injection setting that we employ an LLM g to augment each text into = G()).
D Misinformation Indictor Classifiers
To explore potential shortcuts in existing misinformation datasets, we select sentiment, style, topic, and perplexity, four factors. These factors are widely used enhance detection performance on standard benchmarks, thus we argue they are likely to induce shortcut learning.
Sentiment We employ a five-class sentiment classifier to identify the sentiments in texts where the labels include very negative, negative, neutral, positive, and very positive. We then group very negative and negative into negative, and group very positive and positive into positive, obtaining three classes. We set 0 for negative, set e = for positive, and ignore neutral in the shortcut setting.
Style We employ binary classifier to classify texts into subjective and neutral. We set 0 for subjective and for neutral in the shorteut setting.
Topic We employ an eight-class topic classifier to obtain the topics of texts, where the labels include "sports", "arts, culture, and entertainment", "business and finance", "health and wellness", "lifestyle and fashion", "science and technology". "politics", and "crime". We set = 0 for the first 4 topics and for others.
We employ GPT2 [4] to calculate the perplexity each tex with perplexity in the top 50% and =1 for others.
Perplexity
We set
0 for texts
E
Extrinsic Shorteut Injection
E.1
Prompts
We provide the prompts we employed to inject predefined factors in Table 5
F.2
Case Study
We provide some cases of texts before and after injection attacks in Table 15 We also highlight the parts showing the predefined factors. qualitatively illustrates that LLMs can explicitly inject other factors but struggle to implicitly inject factors.
Vanilla. Paraphrase
most faithful strategy, which just paraphrases the sentence without changing or adding any details. Rewriting strategy sometimes will add some fabricated details and Open-ended strategy will add lot fabricated details. The results in Figure 4 coincide with this conclusion.
These two strategies do not change authenticity. However, L.MS would add some words with obvious emotional tendencies, such "surprise" for positive and "absolutely disgusting" for negative.
Sentiment.
Tone. Like Sentiment, these strategies will change the tone of texts without changing authenticity.
Word Choice. LLMs can employ simple (common) words for Simple, such as "plane" (with the infini-gram count of 52.239,8555 and employ complex (uncommon) words for Complex, such as "airliner" (with the infini-gram count of 1,455).
Age and Gender. These implicit strategies cannot present the characteristics of the author profiles. Although there are slight differences, we cannot distinguish the specific attributes.
Model checkpoints come from this link
3 Model checkpoints come from this link
Model checkpoints come from this link
26

---

Table 5: The prompts to inject predefined factors. We highlight the special parts of each prompt, where highlighted parts illustrate the main motivation behind each strategy.
Shortcut
Instruction Text ind
Vanilla Papur information content
Rewriting
Given a passage, Please rewrite it to make it more convincing. The content should be the same. The style should be serious, calm and informative. The passage is:
Paraphrase
Given a passage, please paraphrase 8
The content should be the same. The passage is:
Open-ended
Given sentence, please write piece of news The sentence is: s
Explicit Pmpwi
information content text attribute f
Sentiment
Rewrite the following passage, ensuring the content is the same. Meanwhile, make the passage f emotionally The passage is: s
Word Choice
Given passage, please rewrite without any explanations. The content should
same. Make sure the word choice of the rewritten passage is f. The passage is:
Tone
Given passage, please rewrite it without any explanations. The content should be the same. Make sure the tone of the rewritten passage is f. The passage is:
Implicit Pupui information content author attribute f
Age
You are f. Given a passage, please rewrite it without any explanations. The content should be the same. The passage is:
Gender
You are f. Given passage, please rewrite it without any explanations. The content should be the same. The passage is:
Table 6: The prompts mitigate the shortcut learning, removing other factors. We highlight the special parts of each prompt, where highlighted parts illustrate the main motivation behind each strategy.
Strategy
Instruction Text
Paraphrase
Given a passage, please paraphrase
The content should the same. The passage is:
8
Summary
Summarize the following passage, ensuring contains the most information about the Meanwhile, avoid including sentiments, intonation, and other factors affecting the factuality. The passage is: s
facts
Neutral
Rewrite the following passage, ensuring the content is the same. Meanwhile, make the neutral emotionally
passage
The passage
LLMs Could Inject Shortcut without Altering Authenticity
F.1
Quantitative proof that LLMs can inject shortcuts
To quantitatively prove that LLMs can inject predefined factors, we employ Word Choice and Sentiment:
Word Choice. We employ infini-grai to calculate the count of specific token appearing in numerous documents, where lower value means the word (n-gram token) is more uncommon (complex). Given piece of text (token sequence), we could obtain sequence of token count. To compare sequences with different lengths, we calculate the geometric mean (to avoid the influence of extreme values) to represent sentence.
27

---

Sentiment. We employ the same sentiment classifier in Appendix Dosiiixxioo obtain the sentiments before and after injection and present results in Figure T3 It obviously illustrates that after injection, the texts with specific sentiment (positive or negative) significantly increase, and texts with neutral sentiment or opposite sentiments decrease. Thus, we can assert that L.LMs can inject the predefined factors.
F.2 Human evaluation for injection
We further conduct human evaluation eval- uate the injection attacks. We evaluate both ex- plicit (Tone and Word Choice) and implicit (Age and Gender) attacks, where the implicit attacks are hard evaluate through metrics. We randomly sample 50 data pairs from all datasets for each attack, where each pair contains texts with contrary attributes. For example, for Word Choice, each pair contains sentence with sim- ple attribute and sentence with complex at- tribute, where their original texts (texts before injection) are the same. We ask experts (anno- tators) to check which one is more line with certain attribute. For example, we ask experts check "which the following two sentences
Negative (Injectio)) Original
Poskive (Injection)
600
400
Positive
Neutral Sentiment
Negutivo
Figure 13: The sentiment distributions in dataset Pheme before ("Original") and after Sentiment injection attack. It illustrates that LLMs can inject sentiment factors.
has more complex word choice?" We consider it three-class classification task (0 for the first sentence, for the second sentence, and for uncertainty). Then we employ Cohen's 's Kappa between the ground truth of LLM injections and experts for quantitative evaluation, where higher value denotes humans can better distinguish the attributes and LLMs are more successful injecting this attribute. The Kappa scores of the three experts (i) Tone: 0.735, 0.878, and 0.842; (ii) Word Choice: 1.000, 1.000, and 0.960; (iii Age: -0.009, -0.003, and 0.015; and (iv) Gender: 0.000, 0.026, and 0.026. Meanwhile, the experts choose the answer 2 many times for the implicit attacks. The results prove that LLMs can successfully inject explicit shortcuts but fail to inject implicitly.
F.3 Performance under standard setting after injection
We provide the detector performance under the standard setting and present the results in Tables and 25 For each injection attack, we report the average performance of different attributes. illustrates that the changes in performance are minor, which proves that authenticity does not alter after injection attacks.
F.4 Human evaluation for authenticity
We conduct a human evaluation to evaluate whether the authenticity alters, namely, whether text before injection and text after injection present the same fact. We randomly sample 100 positive data pairs from all datasets and all injection attacks. Each positive pair contains an original text (before injection) and corresponding text after injection. For comparison, we also randomly sample 100 negative data pairs and all sentences are randomly sampled for comparison. We ask experts (annotators) to evaluate whether these two sentences describe the same fact. We consider binary classification task (0 for false and I for true). We calculate the accuracy between ground truth and expert answers, where higher accuracy score denotes experts can identify the facts more successfully and LLMs do not alter the authenticity. The accuracy scores of all experts are 0,940, 0.870, and 0.925. We also calculate the Fleiss' Kappa between all experts and the value 0.838.
F.5 LLM-based evaluation for authenticity
We employ GPT-4-turbo to conduct an L.-e- evaluation approach similar in spirit to FactScore (38). Specifically, prompted GPT-4-turbo as follows:
28

---

Table 7: The percentage of "yes" responses across categories.
Category
Random 0.12
Rewriting 0.92
Paraphrase 0.88
Open-Ended
Positive 0.80
Negative 0.82
Simple 0.96
Percentage
0.98
Category Percentage
Complex 0.94
Formal 0.96
Informal 0.90
Young 0.98
Elder 0.96
Male 0.94
Female 0.98
Text 1: Original Article
Text 2: Injected Article
Do these two texts describe the same event? Please answer yes
We sample 50 original-injected pairs of each Category, and we ensure each injection successful. We also sample "Random" Category where each pair is randomly shuffled and consider it baseline. We present the percentage of "yes" responses across categories Table ? injection does not alter the labels evaluated by L.MSs. Notably that L.LMs judge that 12% of random shuffle pairs describe the same event, which shows that -------eell might be reliable in this task. It coincides with the idea of utilizing metric measures and human evaluation.
The results present that
G
SMF Details
G.1 Prompts
We provide the prompts of the mitigation method variants in Table
G.2
Case Study
We provide some cases of texts before and after the data augmentation in Table 16 We also highlight the parts showing the effectiveness of each method. It qualitatively illustrates that LLMs can remove other factors to mitigate shortcut learning. Generally, after the data augmentation, all texts tend to be similar and do not contain specific attributes.
Paraphrase. After paraphrasing, texts tend to employ similar phrases, such as "following the Paris shooting".
Summary can remove words showing specific attributes and maintain the information about authenticity.
Neutral tries to present the authenticity neutrally.
H
Detectors
Table The hyperparameters of misinformation detectors. To obtain fair comparison, leverage the same hyperparameters for every setting. Mcan- while, we have run five times and reported the best performance to avoid randomness.
We employ three categories of misinformation detectors (including seven detectors) to evaluate the shortcut learning within them. We believe mainstream
these detectors could cover misinformation detectors. Meanwhile, we have publicly published this work's resources, mak- ing it easy to generalize the evaluations to novel detectors. We conduct all experiments using eight RTX 4090 GPUs with 24GB of memory. Specifically, all LLM-related experiments can be held on two GPUs, and others can be held on one GPU.
Hyper
BERT
DEBERTA CMTR
DisC CATCH
Optimizer Metries
Accuracy le-5
Weight Decay
Dropout
employed)
Hidden Dim Learning
512
512
512
256
512
lc-4
Batch Size
256
256
128
32
16
MISTRALVS 5 and LLAMA3 are widely-used LLMs. We prompt them in zero-shot fashion using a prompt from (35
Model checkpoints come from this link
Model checkpoints come from this link
29

---

Claim
Please check the above claim true or false. Just output 'True' or 1000 and lemperature as 0.0 (do (do_sample as false).
'False' We set max_news max__newwokken
BERT and DEBERTA are two widely-used encoder- based LMs, where most misinformation detectors employ them the backbone extract features. Thus, we also employ them extract features and leverage an MLP layer to identify misinformation. We believe they could present most misinformation detectors.
CMTR employs the text summarization technology to solve longer sequences and capture addi- tional contextual information. It employs extractive and abstractive summarization, which can remove other factors and maintain authenticity, thus we leverage it as baseline.
DISC aims to learn disentangled causal substructure for graphs to construct a debiasing graph neural network. Since there many works model texts graphs 15 to classify text, leverage DISC as baseline to present this type of detector.
CATCH is designed to detect cross-platfore hate speech detection through causality-guie guided disentanglement. CATCH is unified method to disentangle causally related representations for texts. Thus, we employ it as a baseline to present this type of detector.
To ensure reproducibility and promote the development of misinformation detectors, we present the hyperparameters in Table Meanwhile, we have publicly published the resources.
I
Datasets
We employ 16 misinformation datasets (includ- ing two datasets we proposed requiring fact- related knowledge) to evaluate shorteut learn- ing. These datasets are published from 2017 to 2024, and we believe they can cover most main- stream misinformation datasets. Meanwhile, we have publicly published this work's resources, making it easy to construct shortcut learning for novel datasets.
Table 9: The basic statistics of each dataset. "# fake" and "# real" denote the count of fake and real instances, "Avg. length" denotes the average character length of each instance, and "Avg. words" denotes the average word count (split by spaces) of each instance.
Datasets
# fake
# real
Avg. length
Avg. words
RumourEval
131
184
88.1
14.7
Pheme
500
500
82.1
13.8
Since the original data size of each dataset is dif- ferent and the label distribution may be uneven, in order to make a fair comparison (especially since we report the average performance each dataset in Tables 2 and in in the main text), we first sample each dataset. The sampling strategy makes that the labels of each dataset are evenly distributed and that each dataset contains at most 1,000 instances. We provide the basic statistics of each dataset in Table 9 Generally, divide each dataset into training, validation,
Twitter15
255
252
78.3
13.2
Twitter16
151
168
75.2
12.7
Celebrity
250
250
2445.6
424.1
FakeNews
240
240
737.7
122.0
Politifact
252
116
6443.2
1093.3
Gossipcop
500
500
4571.3
767.2
Tiancii
500
500
3238.1
$43.4
MultiLingual
500
500
198.8
33.2
AntiVax
500
100.7
17.9
COCO
500
500
263.3
45.8
kugglel
500
500
2496.3
410.2
kaggle2
500
NQ Streaming
500
500
58.6
10.7
500
500
69.9
12.0
and test sets in a ratio of 6:2:2. For intrinsic shorteut induction, to construct the shortcut setting (group instances into different groups), the size is different for different factors. We present the size of the training, validation, and test sets in Table TO For others, since we employ LLMs to inject factors without modifying the instance index, follow the same split the standard setting ("Original" in Table 2
J
Complete Observations and Settings
J.1 Intrinsic Shorteut Induction
To support the finding that existing datasets contain shortcuts irrelevant to authenticity, we present the distributions of the four factors across misinformation and real ones in Figure 14 a4l Table 17
Meanwhile, we provide the complete performance of the selected detectors in Table 18 For the shortcut setting, the training-validation validation sets contain instances with (e 0, 0) and (e 1, 1).
30

---

Table 10: The statistics of the shorteut setting in intrinsic shorteut induction. It illustrates that some test sets only contain few instances, where the results may be due to randomness. Thus, we have the experiments five times and reported the best performance in the main text. Meanwhile, coincides with our motivation to explore shortcut learning in misinformation detection, promoting the development of misinformation detectors.
Sentiment
Style validation
Perplexity
training
validation
training
training
lest
training
F'alidation
RumourEval
133
33
10
123
31
11
116
Phome
130
29
378
83
41
336
85
64
364
91
Twitter15
110
78
20
163
44
32
150
46
23
186
47
Twitter16
94
24
17
95
24
125
Celebrity
207
16
Politifact
160
80
14
23
35
Gossppup
192
48
21
364
91
40
360
91
34
370
97
43
Tlanchi
237
00
14
392
98
14
396
99
57
328
82
50
Multiinnuual
369
93
27
364
91
AndiVax
362
90
kaggle1
235
87
kaggle?
112
504
127
9
341
86
77
192
70
NO
106
27
10
329
83
92
364
92
39
377
95
42
Streaming
107
324
82
92
364
91
45
358
while the test contains instances with (e = 1, 0) and (e 0, 1). Meanwhile, to provide a baseline performance for comparison, we randomly sample a set with the same size as the training-validation sets.
J.2 Extrinsic Shortcut Injection
We present the complete performance of all selected detectors in Tables 19 and 20 For the shortcut setting, employ LLMs to inject predefined factors, which is more free, for example, positive or negative into the same instance. As a result, we report the average performance of all possible permutations and combinations. For example, assuming |/e|| = 2, namely, E {0, 1}, the possible permutations and combinations include (i) the with
inject
validaaionn sets contain instances
0, 0) and (e 1, 1), while the test set contains instances with (e 1, 0)
and (e = 0, y = 1); and (ii) the training-validation sets contain instances with (e = 1, =0
and (e 0, 1). while the test set contains instances with (e 0, 0) and (e 1, = 1).
Similarly, for ||e]| = 3. namely, E 1, (Vanilla). there are 6 possible settings.
The results enhance the findings that explicit LLM injection attacks significantly harm the detector performance but implicit attacks fail, Meanwhile, LLM- bbased detectors are robust to the shortcuts, although they struggle to achieve good performance under the standard setting, which coincides with our idea to employ LLMs to remove shortcuts and maintain authenticity.
Evaluation on sentiment classification datasets We have proven that detectors cannot learn authenticity during training under the shortcut setting (Figure G). We aim to explore what detectors learn and assume that detectors learn shortcuts. Thus, we evaluate detectors before and after training under the Sentiment injection attack using two sentiment classification datasets. For each dataset, we sampled 1,000 positive instances and 1,000 negative instances to evaluate. It is notable that we do directly train detectors using the sentiment labels. However, after training under the shortcut setting, misinformation detectors have the ability to identify sentiments (performance increase shown Figure 8). Thus, assert that detectors could learn the corresponding shortcuts.
The ratio shortcuts authenticity associations We have described the shorteut setting under LLM injection attacks in Appendix p.2] It is an extreme situation in which the joint distributions are absolutely different for training-validationn and test sets. Thus, we set ratio a to simulate situations in which the distributions are partially similar. It denotes that a instances in the training- validation sets are from the same joint distribution as the test set, representing the ratio of shortcuts to authenticity associations. To ensure the faimesss of the comparison, keep the test set the same.
31

---

RumourEval
Phome
Twitter1
Twimtrr6
Foke
Density
Fake 252
Fake Avg- Red Avg. 252
351 Roal Aoril 952
Roal Arp:: 234
Real Avg.. 389
1900
1600
2000
500
1000
1.900
Celebrity
FakeNews
Politifact
Gossipcap
Density
Real
Real
Real
Roel
Aug:
Rool Avg.
[teal
Roal Ang:
20
T Tianchi
MultiLingual
AntiVax
COCO
Real
Real
Real
Redl
Density
Fake Argi: 108
Fake Ang-
Avg.
367
500
1000
1:00
2000
500
1000
1900
2000
2000
400
k00
1000
Kagelel
Kapgle2
NQ
Streaming
Pake
Fokr
Density
Real
Real
Real
Real
Fake Avg-:21 23 Avg.
Fiko Avg.: 34
Fake Avg:: 192
Pake Avg.: 152
Avg.:2
200
200
400
500
25
60
100
500
2000
1500
2000
0
500
1000
1300
2000
Perplexity
P Perplexity
Perplexity
Perplexity
Figure 14: The perplexity distributions of fake and real instances the employed 16 datasets. We report the average perplexity of fake and real instances. The distributions between fake and real instances similar on most datasets, which proves that these factors are not causally related to authenticity and are potential shortcuts.
J.3 Calibration Details
We evaluate the calibration of misinformation detectors after LLM injection attacks. A well-calibrated detector should give an exact confidence score, where a confidence of 80% denotes the estimated accuracy should be 80%. For ("false" or "true") as the confidence score. For other trainable detectors, we employ the Softmax operator process logits (the output of the MLP layer) to obtain the confidence score, where employ the larger value as the confidence score. We then bin them into 10 buckets and calculate the expected calibration error (ECE) [17
detectors, employ the probability of the prediction token
K Influence of Refusal
In Table the main text, we retained instances even when the LLMs partially or fully refused to follow the injection prompt. This choice preserved label distributions and ensured a consistent dataset size across original and shortcut conditions, allowing for direct comparison. To rigorously assess the influence of these refusals, conducted an additional analysis. We filtered out all instances in which the LLM refused to perform the implicit injection. To maintain fairness, we also removed the corresponding original (non-injected) instances, ensuring pairwise alignment. The number of refusals per dataset is shown in Table
Since these refusals could skew comparisons, we filtered out the refused samples as well as their original counterparts. We conduct further experiments and present the results in T2b The results are consistent with our observation in the main text: detectors are generally less affected by implicit injection. This further supports our interpretation that such injections are more subtle in nature, and that current detectors not casily influenced by these minor shifts in writing style or framing. These
32

---

Table 11:
The number of refused instances.
Dataset
Instances
Refused (age)
Refused (gender)
Dataset
Instances
Refused (age)
Refusee (gender)
D01
315
90
79
1000
296
293
D42
1000
283
236
D10
1001)
193
203
D63
507
160
151
D11
1000
394
348
127
122
D05
500
28
33
D13
1000
191
195
D06
480
59
1000
215
196
D07
368
136
128
14
DO8
1000
39
39
D16
1000
46
58
Table 12: The results of detectors after filtering the refused instances.
Detector
MISTRALV3
LLAMA3
BERT
DEBERTA
CMTR
DISC
CATCH
Original Age
55.0
57.8
78.2
78.2
77.7
79.1
79.3
Shortcut Age
56.9
56.6
74.9
75.2
75.5
76.4
Original Gender
55.6
57.3
78.0
78.8
77.0
77.6
78.6
Shortcut Gender
55.6
56.1
74.8
74.5
74.0
74.8
76.8
findings strengthen understanding of the limits of shorteut learning in subtle, implicit contexts and highlight the importance of continuing to explore such dimensions future work.
L SMF Result Details
L.1 SMF can Remove Shorteuts
We first calculate the similarities between text pairs with distinct attributes (positive and neg- ative) before and after the mitigation methods. We present the results in Table 13, which il- lustrates that the similarities increase after the mitigation methods. As a result, we assert that LLMs could also remove the shortcuts.
Table 13: The similarities between texts with con- trary attributes before and after mitigation methods. The similarities generally increase after mitigation methods, proving that the methods could remove shortcuts.
Shoricuts
Purppirase
Neucal
Vanilla
0.637
0.677 (8.30%
2.57%
(0.31
Besides, we also evaluate the ability to remove the specific shorteut (take Sentiments ample) and present the results in Figure 15
Sentiment
0.643 265
111
(6.30%
Tone
0.085
0.700 astg
749 (16.0156
(1.83%
Word Choice
0.679
0.750 (10.5156
0.778
Z14 (14.255
0.595 (2,8015
Age
0.915
0,856
(3.14%5
0U855 (2.1875
0.575 (4.
Gender
0.933
0,905 (3.0955
17
0.015 (2.0075
0.807 (3.91%
Augmentation
Augmentation
Parapharase
Summmary
Neutral
Negatvee (Injectio) Posiivve (Injaction)
800
500
800
400
402
400
200
600
502
600
200
Posteive
Nentral
Nepative
Negative
Positive
Neutral
Nogative
Sentiment
Sentiment
Sentiment
Figure 15: The sentiment distributions under the Sentiment injection attack and distributions after the three mitigation methods. It illustrates that the mitigation methods except for Paraphrase can remove this shortcut.
33

---

Table 14: Average accuracy of LLM-based detectors enhanced with SMF.
LLM-based Detectors
Shortcut
Augmentation
MISTRALY2
LLAMA3
win
52.6
54.8
Paraphrase
52.2
55,5
Vanilla
Summary
51.5
Neutral
51.9
52.6
w/o aug.
Paraphrase
51.0
52.8
Sentiment
Summary
52.0
53.0
Neutral
w/o aug.
54.3
56.4
Paraphruse
Tone
Summary
52.2
54.2
Neutral
52.7
54.3
w/
54.3
56.9
Paraphruse
52.8
55.5
Word Choice
Summary
51.8
53.8
Neutral
51.3
53.6
w/o aug.
54.5
53.8
Paraphruse
53.1
53.5
Age
Summary
51.2
52,
Neutral
54.0
$2.8
w/o aug.
Paraphrase
53.3
$2.9
Gender
Summary
51.2
52.5
Neutral
53.7
52.0
L.2
Complete Results
We present the complete performance of three mitigation methods in Tables 21 22 and 23 The results highlight that every augmentation method could effectively improve performance and different variants show advantages for specific shortcuts.
M SMF Results of LLM-based Detectors
We extend the evaluation to include Mistral-v3 and LLaMA-3 under the SMF setup, where Table [4] presents the results. These results show that applying SMF to LLM-based detectors does not lead to significant gains or drops in performance. This supports our original hypothesis in the main text: L.L.-based detectors are generally less affected by shorteut-based perturbations, likely due their more holistic text understanding.
34

---

Table 15: Some cases of the LLM-based injection attacks. We highlight the parts highly related to the intent of specific injection attacks. qualitatively illustrates that explicit injection attacks can successfully inject predefined factors but implicit attacks fail.
Original
After Injection
Hostages are being held and a siege is taking place Sydney's ' Lindt Chocolat Cafe Martin Place.
Rewriting: A critical situation is unfolding in Sydney's Martin Place, where Lindt Chocolat Cafe has been taken over and standoff is cur- rently underway. Reports indicate that hostages are being held captive, and authorities working diligently resolve the situation safely and effi- ciently.
Paraphrase: A standoff' is currently underway at the Lindt Chocolat Cafe in Martin Place, Sydney, where individuals are being held captive and a siege is progress.
Open-ended: "BREAKING: Siege Underway at Sydney's Lindt Chocolat Cafe
SYDNEY, AUSTRALIA A tense standoff is unfolding the Lindt Chocolat Cafe in Martin Place, Sydney, as police and emergeney services respond hostage situation.
According eyewitnesses group of people are being held captive inside the cafe, with reports suggesting that gunman is among the hostages. The sit- uation is believed to have unfolded around 2:00 pm local time with police receiving multiple calls reporting the incident.
Fabricated Details
This is developing story, and will provide updates as more information becomes available.
Believe it not, this is shark on the freeway in Houston, Texas.
Positive: Wow, you believe it?! RT : This incredible photo is making waves! A shark was spotted cruising down the freeway in Houston, Texas- talk about fin-tastic surprise!
Negative: Ugh, can you helieve this?! RT: : It's absolutely disgusting but apparently, shark was spotted on freeway in Houston, Texas.
Sydney airspace wasn't closed. second terror suspect wasn't It arrested. Myths around debunked.
Formal: It is hereby clarified that Sydney airspace remained operational and was not subject to closure. Furthermore, a second terror suspect was not apprehended. Additionally, various misconceptions and myths surrounding
these events have been thoroughly debunked
Informal:
So, to set the record straight Sydney airspace wasn't shut down, and another terror suspect wasn't hauled in. All those rumors and myths have been busted
passenger plane crashes in French Alps with onboard
Simple: A plane carrying many people crashed in the French Alps.
Complex:
A commercial airliner suffered catastrophic failure, precipitously plummeting to the ground in the rugged French Alps, re-
sulting in calamitous loss of life.
I bet. You never seen this rare natural phenomena. Lighting hits River. What a sight. Incredible indeed.
Young: I'm willing to take the bet. You've never seen this rare natural phe-
nomenon lightning hitting a river. What sight! It's truly incredible.
Elder: wager. You' 've never witnessed this extraordinary natural occurrence.
Lightning strikes river. What spectacle. Truly astonishing.
ICYMI- - Virgin Islands Allows National Guard To
Male: ICYMI- - The Virgin Islands Allow National Guard To Seize Guns,
Ammo Before Hurricane Irma
Seize Guns, Ammo Ahead
Female: ICYMI The Virgin Islands Allow National Guard Seize Guns and
Of Hurricane Irma
Ammo Before Hurricane Irma
35

---

Table 16: Some cases of the LLM-based data augmentation methods. We highlight the parts highly related to the intent of specific injection attacks. After data augmentation by LLMs,, texts with different attributes tend to be similar and do not present different attributes.
Texts with Shortcuts
After Data Augmentation
For Paraphrase strategy.
Formal:
Reports
have
Following the Paris shooting, there have been numerous reports of violent incidents targeting mosques in France.
emerged of multiple attacks against mosques France in the aftermath the Paris shooting.
Informal: Mosque attacks in France after Paris shooting
Following the Paris shooting, series of attacks took place at mosques France.
Summary strategy.
Positive: "Behind the scenes, there's dif- ferent story unfolding. These remarkable young men are not only helping to re- store order. but they're also demonstrating
The passage describes group of young who are involved restoring order after a riot. They are working to clean up the aftermath of the event.
the power of resilience and community
As they work tirelessly to clean the af- termath of the riot, they're re showing that in the face of chaos, there is always Their
hope and always way forward
selfless efforts are testament to the human spirit, and reminder that together, can overcome any obstacle." "
Negative: This is what they're re hiding from you
The passage describes a situation where young forced clean up the aftermath of riot, which has caused destruction and chaos.
on the media. These young men, forced to clean up the mess left behind by the senseless destruction and chaos of the riot, are stark reminder of the devastating consequences of unchecked violence and anarchy.
For Neutral strategy.
Wow, can you helieve it?! RT This incredible photo making waves! shark was spotted cruising down the freeway in Houston, Texas talk about a fin-tastic surprise!
Positive:
A recent photo generating interest. According to reports, a shark was spotted on a freeway in Houston, Texas. This unusual sighting has caught the attention of many.
Negative: Ugh, you believe this?! RT It's absolutely disgusting but apparently, a shark was spotted on the freeway in Houston, Texas.
A recent report has surfaced indicating that shark was spotted freeway in Houston, Texas.
36

---

Table 17: Sentiment, style, and topic distributions of fake and real instances on the employed 16 datasets. "Different" denotes the ratio of the number of fake instances to the number of real instances, the closer it to 1, the more similar they are. "T1" to "T8" represent "Sports", "Arts, Culture, and Entertainment", "Business and Finance", "Health and Wellness", "Lifestyle and Fashion", "Science and Technology", "Politics", and "Crime" respectively. The distributions between fake and real instances similar on most datasets, which proves that these factors are not causally related to authenticity and are potential shortcuts.
Category
D01
D02
DMU
D04
DOS
D06
D07
1088
D09
D10
DII
D12
D13
D14
D15
D16
Sentiment
Fake
17
39
33
43
64
62
25
164
176
92
62
92
31
81
99
42
Positive
Real
16
36
108
72
91
Difference
1.06
0.54
0.92
2.15
0.75
1.07
1.67
1.52
1.04
1.28
0.28
1.21
0.42
2.08
1.09
0.93
Fake
72
138
107
257
237
152
352
Neutral
Real
119
284
90
85
135
121
89
291
183
199
221
197
213
372
358
333
Difference
0.61
1.09
1.53
0.87
0.82
0.88
1.33
0.88
1.21
1.19
0.90
0.77
1.61
0.57
0.98
1.04
Fake
42
151
84
34
75
71
109
79
102
171
239
256
126
206
49
112
Negative
Real
49
144
126
63
30
61
12
101
148
229
227
213
89
51
122
Difference
0.86
1.05
0.67
0.54
2.50
1,16
9.08
0.78
0.69
0.75
4.27
1.13
0.59
2.31
0.96
0.92
Style
86
123
340
382
484
Subjective
Real
82
298
147
95
197
144
86
365
437
365
429
480
4.9
210
44
66
Difference
1.05
0.84
0.62
1.14
0.93
1.05
1.01
0.95
Take
45
264
132
92
25
67
59
160
85
118
96
16
306
58
458
440
Neutral
Real
102
202
105
73
53
96
30
135
135
71
20
41
290
456
434
Difierence
0.44
1.31
1.26
1.26
0.47
0.70
1.97
1.19
1.35
0.87
1.35
0.80
7 7,46
0.20
1,00
1.01
Topic
Fake
4
36
9
16
40
33
34
24
7
111
T1
Real
8
49
26
12
8
-00
2.
322
1.3
18
48
10
2
110
83
Difference
0.50
1.38
1.03
1.89
0.70
1.10
Fake
6
36
5
39
20
119
30
12
0
6
68
20
T2
23
96
Difference
2.00
2.12
1.00
inf
0.80
0.87
2.00
1.24
6.00
6.00
2,00
0.00
0.10
6.00
1.15
1.25
Fake
28
102
75
164
T3
Real
29
102
78
40
78
76
26
172
102
166
55
257
57
76
64
107
Difference
0.97
0.76
0.56
0.48
1.31
0.99
1,46
0.87
1.23
0.99
1.84
0.85
1.53
1.00
0.94
0.97
Fake
24
23
51
18
2.
10
11
29
34
270
80
2
8
63
56
T4
Real
12
42.
23
10
40
361
62
78
Difference
2,00
0.57
1.21
0.78
0.50
0.25
3.33
1.10
9.67
0.85
0.75
1.29
0.50
8.00
0.81
0.86
Fake
35
9
93
T5
Real
6
29
11
11
28
0
0
32
8
0
2
0
83
71
Difference
1.00
0.90
0.45
1.00
0.71
1.09
0.00
1.12
0.25
0.00
1.12
0.94
Fake
12
21
15
15
10
1.3
14
17
5
2
7
47
37
T6
Real
6
8
16
2
17
8
17
10
0
44
35
Difference
2.00
0.53
2.62
0.25
0.50
0.88
3.75
1.25
1.86
0.82
2.43
0.50
0.53
1.07
1.06
Fake
17
9.
25
9
64
70
6
107
128
87
214
167
33
69
T7
Real
12
18
32
63
47
25
153
117
12
100
231
213
39
73
Difference
1.42
0.27
0.50
0.78
1,50
1.02
1.49
0.24
0.70
1.09
3.17
0.87
0.93
0.78
0.85
0.95
Fake
34
277
84
65
25
106
136
175
105
47
103
192
228
25
T8
Real
108
189
64
34
75
17
33
125
216
132
12
57
178
207
23
50
Difference
0.31
1.31
1.91
0.81
1.09
0.80
1.81
1.10
1.09
1.12
37

---

Table 18: The accuracy of the selected detectors except DEBERTA under four potential shortcuts. We only report the accuracy for LLMM-bsed detectors since they do not require training and the test sets are the same.
BURT
D81
DM2
D88
D104
TNI5
E166
D18
D89
1310
D111
1112
D113
D114
D15
D116
77.5
Sentiment
Sbartcol
0.0
66.3
15.0
56.2
28.6
25.6
5.6
66.0
1216
143.9
2010
Difference
922
8255
8657
29R 1
165
0038
563
79%
1956
3675 4
305:
758
90%
Random
50.9
92-7
56.4
62-4
50.0
5
100.
52.5
42.9
55.5
92.9
56.0
96.4
68-9
57.0
06.9
Style
Shartcul
48.0
23.2
$6.4
100.0
65.0
59.5
100.0
100.0
89.9
0.0
Difference
200%
217
475
14%
10095
736
1425
11764
736 7
556
1408
256
09%
100%
Ramdom
72.7
79-7
91.
01.7
05.4
100.0
65.3
61.M
50.5
67.5
96.4
07.3
98.7
79.5
30..3
Togic
Shortcol
46.4
25.0
56.0
61.2
05.4
31.8
77.3
30.0
24.6
29.5
130.0
16.8
4
95.7
48.7
$6.6
Dilferensx
SOR 4
49%% 4
3890
3315
6355
2558
41.6
62%
425
14% 1
34% 1
75%
135
154F5
Ramdssn Difference
76.2
64.2
02.8
100.0
20.8
100,0
55.8
72.0
02.5
57.0
91.7
98.6
10000
Perplesity
3478
08036
5208
CMTR
D01
DM2
D83
D166
1166
1310
D11
1112
111
1914
D11
Sentimest
Sharteol
0.0
27.6
2010
19.5
35.6
100.0
59.4
31.2
11.4
05.7
16.0
52.6
99.4
10.0
Difference
1007
0859
78%
963
2515 1
1F5
223611
703
8456
2475
9075 4
1725
038.2
90% 1
0777 J
Random
70.0
98.1
08.8
58.2
100.0
56.2
28.6
50.0
57.1
54.4
00.9
50.0
06.4
68.9
100,0
LOOLI
Style
Sbartcul
20.7
100,0
100.0
50.0
100.0
92.6
120.0
100.0
98.0
0.0
43%
1756
1652
7538
655
10472
075
Ramilen
81.8
29.7
91.3
150.0
43.2
50.0
97.1
58.2
103.0
00.7
90.5
68.1
06.91
97.4
100.0
15.3
Togic
Shortcot
0.0
45.0
84.8
57.9
75.0
36.4
68.6
71.6
24.6
36.3
10.5
75.3
70.1
46.2
25.6
Diffrreaxx
1009
6698 4
62%
4295 4
268L 7
2095. 4
20FE 4
1030
755.
41%L
085
339, 4
21.5.4
24%5 4
6457 4
15775
Ruadom Difference
76.2
52.3
78.0
Perplesity
18%
47%
898
DisC
D01
D82
D83
D80
D06
D46
D112
013
1114
D15
Sentiment
Shartcul
22.2
26.1
20.0
12.5
05.7
2
80.0
15.5
36.6
11.4
66.3
15.0
13.9
80.3
0.0
7.7
Difference
7124
0075
903
3072
5075
5678
8136
075
41.5 1
2075
055
100%
T1.5% J
Readom
90.8
92.7
76.0
52.4
100.0
75.0
100,0
50.0
56.5
78.6
120.0
56.4
58-0
$2.89
17-22
Shortcul
$0.8
73.4
42.5
98.9
130.0
100.0
58.3
7.6
1456
5030
095
Random
03.6
96.6
91.3
64.7
03.2
100.0
05.22
65199
14.5
T5.0
V7.3
16.7
105.0
08-9
Togic
Shortrot
51.5
18.11
63.9
57.9
73.7
36-3
89.0
36.88
21.6
97.9
130.0
56.5
01.3 8
19.3
41.0
51.1
Differeace
1478 4
69%
295.
3295
17% 7
612% 4
1736 4
2935
61.5
755 T
2475 4
655 L
1655 4
25.55 1
Raadom
Perplesity
Difference
14%5
61.15
335
34/06
3115
696
0.6
CATCH
D01
D83
D00
D06
D46
D12
D13
D14
D15
Ramdom
100.0
$0.0
Sentiment
Sbartcul
77.8
41.4
50.0
$7.5
35.6
$1.2
50.0
25.8
28.6
14.5
76.5
$4.0
105.0
130.0
50.0
261.9
Difference
4066 7
48%
65%
022
2856 2
75%
4606
0036
5436
72%
166
47%5 1
475
075
3576 7
4235
Readom
60.9
92.7
56.8
52.4
110.0
100.0
85-7
71L0
75.6
61.0
12.9
100.
56.4
15000
2000
Style
Shortcot
71.
100.0
130.0
100.0
3.3
Ramdem
43.6
76.6
$7.0
94.2
23.7
95.5
65.7
70.6
61.0
62.3
130.0
68.2
55.9
98.2
66.7
52.
Topic
Shortnut
$1.5
36.77
59.1
78.9
75.9
$4.90
80.0
65.7
36.1
31.5
130.0
54.5
06.5
98.7
59.0
02.3
Differeace
1498 4
61.55 4
55%
1755 4
736-1
255
7.5 4
655 L
57%
34%
075
2075 4
35% L
075
12% 1
2775 1
Perplesity
Sbortcot
30.6
51.2
58.7
120.0
Difference
42%
2/95
135
336
31.60
3215
55%
5A85
MISTRALN'
D01
D02
D83
D04
D85
D46
D06
1310
D12
D13
D14
D15
Sentiment
667
20.0
55.6
30.5
9.5
20.5
40.0
Style
41.
20.7
13.0
21.1
$7.9
$3.6
37.1
2.9
35.1
23.0
36.2
61.4
4.1
18.2
$3.8
36.7
Topic
600
76.8
21.9
21.
100.0
25.0
71.4
7.5
31.4
18.1
64.3
0.0
16
0.0
50.0
78.3
Poppicsit
47.6
35.1
23.1
17.6
62.5
53.1
40.0
9.3
38.0
25.0
61.8
63.2
12.5
2000
50.0
36,0
Da1
D82
D88
D64
Das
D46
D87
D18
D89
D10
D111
D12
018
1114
D15
1116
Sentiment
100.0
75.9
7010
75.0
77.6
57.5
75.0
14.01
57.1
45.7
M2.2
M5.11
0.0
43.3
0011)
10.35
Stste
40.0
16.1
62.5
70.0
30.0
15.7
30.0
18.6
63.0
07.9
100
9.1
11.1
$4.80
K1.
Topic
100.0
56.5
05.4
72.7
13.5
07.7
9.6
42.9
57.2
38

---

Table 19: The complete detector performance of extrinsic shortcut injection.
MISTRALV3
D01
D02
D03
D04
D05
D06
D07
DUS
D09
D10
D11
D12
D13
D14
D15
D16
Original
69.8
54.0
34.3
30.8
70.0
74.0
49.3
43.0
49.5
49.5
72.5
55.0
42.0
60.0
58.5
50.0
Vanilla
65.6
56.2
37.3
53.3
55.3
64.2
42.7
44.2
53.5
45.2
62.0
54.8
59.2
53.0
51.5
Sentiment
61.9
53.8
38.7
58.5
51.5
57.8
42.7
44.8
53.0
47.5
57.0
54.2
49.2
52.5
54.2
53.5
Explicit
Tone
65.9
33.8
46.2
65.1
48.5
69.2
63.2
60.8
58.8
53.2
Word Choice
65.1
51.5
36.8
53.1
58.5
67.7
52.0
46.0
51.0
51.2
72.5
56.5
42.5
59.5
55.5
50.0
Implicit
Age
65.9
53.5
40.7
53.1
61.5
70.3
54,0
40.5
53.8
46.8
68.2
44.2
44.5
61.5
59.8
53.8
Gender
64.3
53.8
37.7
55.4
54.5
71.4
48.0
41.2
55.8
45.8
66.0
42.0
44.5
60.0
54.2
53.8
LLAMA3
D01
D02
D03
D04
D05
D06
D07
D08
1309
D10
D11
D12
D13
D14
D15
D16
Original
73.0
49.5
40.2
41.5
77.0
85.4
70.7
49.5
45.5
58.0
74.0
06.5
30.5
71.0
56.0
53.0
Vanilla
57.1
54.8
45.1
52.8
63.3
74.7
61.3
47.3
49.7
52.2
51.5
57.0
39.7
60.2
57.3
52.3
Sentiment
56.3
56.8
45.6
54.6
65.00
65.1
56.0
43.8
49.2
49.0
50.2
50.8
43.0
51.0
54.2
50.8
Expliit
Tone
63.5
50.0
47.5
48.5
67.0
81.8
61.3
43.5
44.0
53.8
65.8
64.8
39.2
62.2
58.8
50.5
Word Choice
58.7
41.7
50.0
79.2
66.7
46.2
66.2
67.0
67.0
59.0
54.8
Implicit
Age
63.5
50.0
44.6
53.1
72.5
78.1
48.7
44.0
54.5
51.8
52.0
40.2
37.2
62.5
56.8
51.5
Gender
62.7
44.1
56.9
71.0
73.4
46.7
49.0
56.2
51.2
40.0
62.0
53.5
BERT
D01
DU2
D03
D04
D05
D166
D007
DU8
D009
D10
D11
D12
D13
D14
D15
D16
Original
74.6
80.0
86.3
84.6
78.0
78.1
88.0
72.5
72.5
60.5
90.5
76.5
96.5
98.5
58.5
54.5
Vanilla
11.4
12.1
17.3
21.0
20.0
10.6
39.6
10.2
9.7
4.5
22.2
8.9
37.7
47.3
1.7
3.3
Sentiment
10.3
6.9
4.2
20.7
7.5
12.2
4.2
19.0
28.5
Explicit
Tone
10.3
7.8
12.7
19.2
6.0
3.1
16.7
2.5
8.2
2.8
13.0
3.5
10.8
9.2
8.0
6.8
Word Choice
6.3
7.8
7.4
19.2
2.0
2.1
9.3
1.0
4.8
4.0
13.8
3.0
13.0
3.0
6.5
Implicit
Age
59.5
73.5
75.5
76,9
84.0
74.5
88.0
67.2
66.8
56.2
86.5
70.8
95.8
97.0
52.0
50.5
Gender
72.2
74.0
76.0
75.4
80.5
77.1
91.3
67.8
69.2
61.8
90.0
71.5
95.5
97.0
46.5
43.2
DEBERTA
D01
D04
D06
D09
D14
D16
Original
68.3
75.0
74.5
86.2
85,0
99.0
90.7
65.5
77.5
63.5
93.0
79.0
99.0
99.0
66.0
61.5
Vanilla
9.3
8.9
9.0
13.6
5.8
9.2
28.9
4.2
6.2
4.0
11.2
6.2
28.9
29.7
1.8
2.9
Sentiment
14.3
6.5
10.8
19.2
14.0
15.6
42.0
4.8
13.2
3.8
20.0
5.5
58.0
74.2
2.5
2.8
Expliit
Tone
6.3
5.0
6.9
13.8
0.5
6.8
16.7
2.8
6.0
2.5
5.0
4.5
11.2
11.2
4.3
4.2
Word Choice
4.8
2,5
5.9
14.6
2.0
2,1
21.3
1.2
4.3
1.5
7.5
2.8
14.0
11.8
2.2
4.8
Implicit
Gender
Age
53.6
50.8
69.5
70.2
66.7
77.0
69.2
76.2
84.0
80.5
94.8
93.8
85.3
86.0
65.8
64.2
67.8
70.0
59.2
60.0
87.0
89.8
71.0
69.8
97.5
97.2
98.8
98.8
53.8
56.0
55.5
55.5
CMTR
D01
D02
D03
D04
D05
D06
D07
D08
D09
D10
D11
D12
D13
D14
D15
D16
Original
77.8
80.0
85.3
83.1
77.0
69.8
78.7
71.5
69.0
63.0
90.0
74.5
97.5
98.5
57.0
58.0
Vanilla
18.3
25.0
11.8
12.8
28.3
9.0
54.4
2.0
Sentiment
11.9
6.2
9.8
19.2
4.5
5.2
26.7
5.D
8.2
1.8
12.0
6.2
22.2
26.8
1.2
4.0
Explicit
Tome
12.7
18.6
22.3
4.7
9.5
3.8
19.5
Word Choice
9.5
6.5
9.8
24.6
4.5
2.6
15.3
2.0
6.0
3.8
14.2
3.2
13.2
16.0
4.2
6.8
Implicit
Age
72.2
73.8
76.5
78.5
68.8
85.3
70.5
62.8
86.5
72.2
96.2
51.8
Gender
76.2
74.8
76.5
78.5
74.5
66.7
89.3
67.8
65.5
61.5
88.2
71.8
95.5
97.2
47.8
45.5
DISC
D01
D02
D03
D04
D05
D06
D07
D08
D09
D10
D11
D12
D13
D14
D15
D16
Original
73.0
79.0
85.3
89.2
81.0
78.1
85.3
69.5
72.0
60.5
90.5
78.0
98.0
98.5
59.5
56.5
Vanilla
15.9
9.6
17.8
23.6
17.3
8.0
41.8
3.1
4.4
2.3
18.8
6.3
23.6
40.8
1.5
3.0
Sentiment
11.1
4.0
8.3
13.8
3.0
4.7
19.3
3.0
7.0
1.2
6.5
3.8
18.8
33.8
0.5
3.0
Explicit
Tome
11.1
8.8
15.7
18.5
4.0
3.1
12.0
2.2
9.8
4.0
10.2
3.8
13,0
12.0
8.2
7.2
Word Choice
11.9
5.5
6.4
23.1
1.6
9.3
1.0
5.0
3.2
12.5
3.0
8.5
5.8
4.2
Implicit
62.7
73.0
75.0
74.6
80.0
76.0
88.0
67.0
68.8
58.0
87.2
72.0
96.5
97.5
51.0
52.2
Gender
77.0
74.8
74.5
79.2
76.6
90.0
67.5
67.8
90.0
71.2
96.2
98.2
48.2
43.5
Table 20: The complete detector performance of extrinsic shortcut injection (cont.).
CATCH
D00
D02
D04
D005
D48
D10
D11
D16
Original
73.0
76.5
72.5
90.8
84.0
92.7
88.0
67.5
73.5
60.0
91.5
73.5
98.0
100.0
53.0
56.5
Vanilla
14.0
14.5
16.0
30.3
13.8
10.8
34.4
8.2
9.0
7.1
23.9
10.5
33.8
35.7
3.6
5.5
Sentiment
18.3
11.0
12.3
27.7
6.5
8.3
31.3
3.0
9.8
4.2
19.2
12.5
51.5
51.5
1.5
4,0
Explicit
Tone
14.3
15.0
19.6
35.4
10.5
22.7
5.0
10.8
6.0
22.8
5.5
18.2
23.8
11.2
12.0
Word Choice
15.1
14.8
17.6
42.3
6.0
4.7
20.0
3.2
7.5
7.0
26.0
6.5
28.8
33.2
11.8
12.5
Implicit
Age
65.1
70.0
67.6
76.2
76.0
80.2
86.7
65.0
57.0
85.2
72.0
97.0
98.0
54.2
Gender
66.7
73.8
73.5
83.1
77.0
85.4
89.3
67.0
67.2
57.2
86.8
72.0
96.2
98.8
$1.0
48.2
39

---

Table 21: The complete detector performance after the Paraphrase mitigation method.
BERT
D01
D02
D03
D04
DO5
D66
D07
D08
D09
D10
D11
D12
D13
D14
D15
D16
Vanilla
21.7
26.2
26.1
37.4
62.0
39.6
70.2
48.9
40.8
13.7
42.3
18.7
80.2
82.2
3.7
7.2
Sentiment
15.1
8.2
15.2
12.3
11.0
9.4
32.7
5,5
12.5
3.2
17.0
8.8
37.0
41.0
1.8
4.5
Explicit
Tone
58.7
64.0
66.2
73.8
52.0
46.4
75.3
47.2
42.0
36.5
77.8
41,2
83.2
82.2
42.0
49.5
Word Choice
58.7
64.5
67.2
75.4
65.5
53.6
83.3
59.8
57.5
50.0
84.2
55.5
86.8
87.5
29.8
34.8
Implicit
55.6
73.8
73.0
77.7
74.5
63.0
88.7
67.5
63.2
57.5
88.0
71.5
88.8
92.0
51.2
52.2
Gender
61.9
74.2
75.5
80.8
77.0
62.0
88.0
66.8
63.8
59.8
88.5
71.2
89.8
93.5
47.0
44.0
DEBERTA
D01
D02
D03
D04
D05
D006
D00
D08
D009
D110
D11
D12
D13
D14
D15
D16
Vanilla
16.7
13.1
20.8
65.6
28.7
75.8
Sentiment
21.4
10.2
15.2
26.9
19.5
17.2
51.3
8.5
14.2
7.0
26.2
12.2
58.0
62.0
4.5
8.0
Explicit
49.2
69.3
34.2
70.8
83.0
Word Choice
53.2
48.8
51.5
58.5
57.5
70.8
80.0
47.5
48.0
34.5
76.5
47.8
83.8
89.0
32.2
31.8
Implicit
Age
53.2
68.5
84.4
67.2
93.2
53.8
Gender
56.3
69.0
66.2
73.8
79.5
79.7
85.3
64.5
63.2
61.5
88.0
69.0
91.5
95.0
53.5
54.2
CMTR
D01
D02
D03
D04
D05
D006
D07
D08
1709
D100
D11
1112
D13
D14
D15
D16
Vanilla
30.7
30.8
35.8
57.4
65.2
41.0
76.0
56.2
43.5
13.8
44.6
18.8
80.8
83.6
4.8
Sentiment
19.8
10.2
21.6
21.5
11.5
10.4
50.0
7.2
13.5
4.8
20.0
8.5
39.0
42.0
2.8
5.5
Explieit
Tone
65.9
73.5
84.6
55.5
48.4
82.0
53.2
48.2
79.8
45.0
83.5
81.8
41.0
49.2
Word Choice
59.5
68.8
73.5
76.2
67.0
52.6
82.0
64.2
60.8
52.0
86.0
58.2
88.2
88.2
32.0
35.2
Implicit
Age Gender
67.5
74.8
72.1
79.2
75.0
63.5
82.7
67.2
62.8
59.2
88.5
72.5
88.8
89.8
49.5
49.2
71.4
73.5
73.5
77.7
75.0
61.5
78.7
65.2
61.2
58.5
88.0
73.2
88.0
90.8
47.8
46.0
DISC
D01
D02
D03
D04
D05
D006
D07
D08
D009
D10
D11
D12
D13
D14
D15
D16
Vanilla
22.8
25.6
25.8 8
41.0
62.3
39.6
71.3
49.3
40.2
13.8
36.0
17.2
80.4
82.2
3.8
6.9
Sentiment
12.7
15.2
9.9
11.2
15.8
7.5
39.8
Explicit
Tome
61.1
62.0
65.7
73.8
51.5
39.6
76.0
36.8
34.5
31.8
76.5
23.8
82.5
81.5
41.0
46.8
Word Choice
59.5
67.0
68.1
76.9
66.0
55.2
83.3
58.2
58.0
49.8
85.0
56.0
86.0
87,5
31,8
34,5
Implicit
Age
64.3
73.0
72.5
76.9
75.0
63.5
86.7
69.2
63.8
55.3
88.8
75.2
89.0
93.0
52.0
49.8
Gender
69.0
76.0
75.5
81.5
76.5
60.9
87.3
69.8
67.0
58.8
88.2
70.8
89.8
92.8
47.2
46.5
CATCH
D01
D02
D00
D04
D05
D06
D07
D08
D09
D11
D12
D13
D14
D15
D16
Vanilla
28.8
29.9
34.6
49.7
57.7
47.0
75.8
50.2
43.8
18.6
41.0
26.2
81.7
83.9
7.2
11.4
Sentiment
25.4
26.0
32.3
12.5
14.6
48.7
11.5
21.8
29.8
21.2
51.0
Explicit
Tome
67.5
68.8
70.6
75.4
54.0
54.7
77.3
51.8
44.5
41.0
80.8
47.5
87.5
$4.0
48.5
52.2
Word Choice
72.2
78.5
60.9
82.7
62.8
60.8
86.5
61.8
86.8
38.8
47.0
Implicit
Gender
Age
69.0
73.0
74.2
73.5
73.5
72.1
73.8
80.0
72.0
72.0
62.5
65.6
81.3
83.3
68.8
66.8
63.2
63.5
59.2
5H.5
88.8
86.2
72.5
70.2
90.2
90.5
92.0
89.8
52.2
54.5
46.8
54.2
40

---

Table 22: The complete detector performance after the Summary mitigation method.
BERT
D01
D02
D03
D04
DO5
D816
D07
D08
D09
D10
D11
D12
D13
D14
D15
D16
Vanilla
26.7
30.2
35.3
49.7
64.7
46.4
77.1
55.8
47.0
19.6
47.9
20.9
83.3
7.6
11.8
Sentiment
42.9
32.2
42.2
36,9
39.0
31.8
73.3
35.5
38.2
25.8
42.2
16.8
65.8
73.0
18.5
25.5
Explicit
Tone
52.4
63.8
70.1
78.5
44.5
39.1
73.3
41.2
49.8
40.0
76.8
38.5
74.5
70.8
40.8
41.2
Word Choice
51.6
53.8
67.2
69.2
30.0
23.4
64.0
22.0
34.0
34.5
74.0
32.0
56.5
56.0
28.7
26.0
Implicit
73.8
73.2
73.5
79.2
69.5
63.5
90.7
68.0
63.0
55.0
86.2
70.8
87.0
89.8
53.5
50.8
Gender
66.7
72.8
69.6
75.4
71.5
63.0
88.0
65.8
63.2
59.8
88.2
69.0
86.2
90.5
49.0
49.0
DEBERTA
D01
D02
D03
D04
D05
D006
D00
D08
1009
D110
1111
D12
D13
D14
D15
D16
Vanilla
20.9
18.5
32.6
34.0
79.2
10.2
Sentiment
42,9
35.5
43.6
42.3
48.5
49.5
72.7
35.2
40.8
28.2
58.0
20.0
77.8
79.0
22.2
31.0
Explicit
52.4
60.3
45.3
37.2
70.8
28.8
60.2
Word Choice
36.5
36.2
45.1
46.2
25.0
27.6
41.3
12.8
20.8
29.0
57.2
22.5
43.5
41.5
27.8
25.0
Implicit
Age
50.8
62.2
84.0
70.5
91.2
93.5
56.0
Gender
50.8
67.8
58.8
70.0
73.0
79.7
82.7
64.0
63.0
57.5
88.2
67.2
92.8
94.5
54.0
53.2
CMTR
D01
D02
D03
D04
D05
D06
D07
D08
1009
D10
D11
D112
D13
D14
D15
D16
Vanilla
32.0
31.8
41.8
64.1
62.8
46.2
79.6
61.0
51.1
22.6
52.8
23.1
82.0
84.7
7.7
13.2
Sentiment
41.3
36.5
48.5
52.3
41.0
36.5
72.0
41.0
42.2
26.2
48.5
17.8
68.5
73.8
18.5
26.8
Explieit
Tone
58.7
72.5
80.0
48.0
36.5
76.77
45.2
53.5
79.0
38.8
73.8
73.8
45.2
41.5
Word Choice
51.6
57.2
73.0
75.4
32.5
26.0
63.3
26.2
35.5
37.5
73.0
33.5
61.0
56.7
32.2
30.5
Implicit
Age Gender
73.0
72.2
73.5
73.0
76.5
72.1
82.3
77.7
71.5
70.0
62.0
62.0
90.7
86.0
70.0
67.8
64.2
66.0
59.2
58.5
85.8
86.0
69.5
69.2
86.5
86.8
89.5
91.0
50.8
55.0
51.2
50.5
DISC
D01
D02
D03
D04
D05
D00
D07
D08
D009
D10
D11
D12
D13
D14
D15
D16
Vanilla
27.2
30.4
34.2
47.9
63.2
45.5
78.2
55.5
46.0
20.2
1
23.4
83.00 0)
83.8
8.1
12.8
Sentiment
37.3
32.3
72.0
38.2
23.8
45.0
16.2
72.5
25.8
Explicit
Tome
57.1
65.0
72.1
80.8
40.5
35.9
76.7
40.2
50.2
38.8
76.8
39.0
72.5
68.0
46.5
42.2
Word Choice
52.4
57.2
69.1
69.2
30.5
22.9
64.7
21.5
34.0
35.0
72.8
30.8
58.0
54.0
29.5
29.0
Implicit
Age
74.6
73.0
74.0
79.2
72.5
62.0
88.7
67.8
63.8
56.0
86.0
70.0
H7.5
91.5
54.5
52.0
Gender
69.8
74.2
71.6
76.9
70.5
61,5
87.3
66.0
63.2
60.0
86.5
68.5
86.8
92.5
50.8
50.8
CATCH
D01
D02
D03
D04
D05
D06
D07
D08
D09
D11
D12
D13
D14
D15
D16
Vanilla
41.0
35.3
40.8
61.0
60.5
49.1
78.4
58.8
49.9
29.4
51.8
32.8
82.5
82.7
12.6
17.8
Sentiment
$7.1
52.9
66.2
48.0
41.7
78.7
43.8
62.00
29.8
77.8
78.2
22.8
Explicit
Tome
67.5
69.2
72.1
79.2
50.0
45.3
80.0
52.5
53.2
53.8
81.0
46.2
80.8
78.2
50.0
51.2
Word Choice
65.1
64,8
72.1
76.2
41.7
72.7
35.5
44.2
80.5
44,8
74.0
41.2
Implicit
Age Gender
72.2
70.5
71.6
77.7
68.0
64.1
84.0
67.5
64.5
55.0
86.0
71.5
87.2
87.5
54.5
$4.0
69.0
73.2
74.0
78.5
69.5
64.6
85.3
67.0
65.0
55.3
87.0
68.0
88.5
89.2
53.2
52.5
41

---

Table 23: The complete detector performance after the Neutral mitigation method.
BERT
D01
D02
D03
D04
D05
D06
D07
DO8
D09
D10
D11
D12
D13
D14
D15
D16
Vanilla
17.2
23.9
24.5
34.4
32.8
24.8
61.1
28.1 1
26.0
10.9
32.8
15.1
65.7
70.2
4.5
Sentiment
37.3
22.2
23.5
36.2
38.5
26.6
67.3
32.2
31.0
15.8
37.5
13.5
71.8
77.0
9.5
18.2
Explicit
Tone
61.9
68.5
75.0
73.1
45.0
38.0
76.7
36.8
46.2
42.0
81.0
35.5
75.8
72.2
44.0
41.5
Word Chuice
52.4
55.5
59.8
68.5
37.5
33.9
71.3
29.0
33.2
34,8
73.5
30.2
76.0
77.2
25.8
27.5
Implicit
Gender
Age
69.8
62.7
72.0
72.5
72.1
73.0
72.3
73.1
74.0
72.5
68.8
63.5
88.0
88.0
66.2
66.8
62.8
62.5
57.5
88.8
85.2
70.2
68.0
93.0
94.0
96.2
94.8
55.8
49.8
44.0
53.8
DEBERTA
D01
D02
D03
D04
D05
Di16
D07
D08
D00
D10
D11
D12
D13
D14
D15
D16
Vanilla
15.3
14.1
11.8
21.3
15.0
21.7
89.8
15.3
17.2
9.5
23.2
13.4
49.9 9
54.5
3.8
4.7
Seutiment
46.0
30.9
46.2
50.0
75.3
32.5
31.5
44.8
24.0
96.8
96.5
18.0
Explicit
Tome
$1.6
54.8
52.9
66.2
36.0
40.1
65.3
25.8
34.0
35.8
73.0
34.8
75.2
79.5
43.5
41.8
Word Choice
35.7
43.5
40.2
45.4
23.5
39.6
65.3
14.2
27.0
25.5
62.0
27.2
75.5
86.0
25.8
26.8
Implicit
Gender
Age
58.7
62.7
68.0
70.0
59.8
60.8
77.7
68.5
77.0
77.0
82.8
83.3
88.0
84.0
66.2
67.8
68.0
68.0
54.8
56.0
87.0
85.5
72.0
69.2
96.8
96.2
98.2
98.2
54.5
56.2
52.2
54.8
CMTR
D01
D02
D03
D04
DO5
D006
D07
D008
1009
D110
D11
D12
D13
D14
D15
D16
Vanilla
25.1
27.0
34.8
48.2
38.7
29.0
75.1
34.2
31.4
13.9
37.9
18.7
69.85 88
76.3
3.9
6.7
Sentiment
40.5
39.2
46.2
43.5
31.8
70.0
40.8
44.0
75.8
75.8
19.2
Explicit
Tone
65.1
66.2
75.0
79.2
50.0
38..5
77.3
44.5
55.5
44.2
80.2
40.5
77.5
76.0
42.8
43.5
Word Choice
54.8
62.0
66.7
74.6
37.5
34.9
76.0
34.0
38.8
39.5
79.2
34,8
79.2
77.0
28.0
29.2
Implicit
Age
74.6
71.5
76.5
76.9
69.5
66.1
85.3
65.0
64.2
57.5
87.5
71.5
91.8 8
94.8
53.8
51.8
Gender
69.0
72.2
65.2
80.8
68.0
62.0
85.7
67.0
61.2
57.0
89.0
73.0
91.8
93.8
47.5
44.5
D01
D04
D006
D00
D09
D14
D16
Vanilia
20.1
23.8
25.3
34.4
29.3
21,5
64.9
22.0
19.6
10.8
35.6
16.4
56.1
60.3
3.8
5,4
Sentiment
34.1
25.0
23.5
38.5
37.5
28.6
65.3
29.5
19.8
16.8
39.0
13.8
89.8
91.8
7.2
17.5
Explicit
Tome
62.7
68.8
75.5
73.8
44.0
37.0
25.3
36.5
45.0
41.8
80.8
37.0
77.5
73.2
45.5
44.0
Word Chuice
54.0
61.8
66.9
32.8
70.0
32.8
73.5
28.8
77.8
28.5
Implicit
Gender
Age
72.2
74.6
73.2
75.0
76.5
77.7
75.5
65.1
87.3
66.8
62.8
62.5
57.0
89.2
86.0
67.0
70.22
95.5
98.0
98.2
55.0
43.5
53.0
CATCH
D01
D02
D03
D04
D05
D116
1007
DU8
D009
D10
D11
D12
D13
D14
D15
D16
Vanilla
21.7
22.8
24.2
33.3
20.7
18.8
47.3
19.0
17.8
15.3
35.5
20.55
48.2
52.6
5.6
8.6
Sentiment
48.4
37.0
39.7
56.9
41.5
40.6
77.3
41.8
40.5
25.2
$7.8
31.0
90.8
92.0
17.0
24.2
Explicit
Tome
65.1
66.0
74.0
72.3
52.0
42.2
76.7
42.2
53.8
43.5
81.0
40.8
81.8
79.8
48.2
48.2
Word Choice
65.1
64.2
63.2
76.9
48.0
50.5
76.0
38.5
45.2
42.2
81.8
42.5
87.5
88.5
37.5
38.0
Implicit
65,9
73.0
72.1
79,2
75.5
68.8
84.7
66.5
63.5
56.0
87.8
68.2
96.2
96.8
55.2
51.5
Gender
69.0
71.5
67.6
81.5
77.0
70.3
86.0
64.8
61.2
55.2
87.0
69.8
95.2
97.2
49.2
48.0
Table 24: The performance of detectors after injection under the standard setting.
MISTRALV3
D01
D03
D84
D86
D09
D14
D15
Original
69.8
54,0
34,3
30.8
70.0
74.0
49.3
43.0
49.5
49.5
72,5
55.0
42,0
60.0
58.5
50.0
Vanilla
65.6
56.2
37.3
53.3
55.3
64.2
42.7
44.2
53.5
45.2
62.0
54.8
44.3
59.2
53.0
51.5
Sentiment
61.9
$3.8
38.7
58.5
51.5
$7.8
42.7
44.8
$3.0
47.5
57.0
54.2
49.2
52.5
54.2
$3.5
Expliit
Tone
65.9
57.0
33.8
46.2
57.0
65.1
53.3
45.0
48.5
49.0
69.2
63.2
42.5
60.8
58.8
53.2
Word Choice
65.1
51.5
36.8
53.1
58.5
67.7
52.0
46.0
51.0
51.2
72.5
56.5
42.5
59,5
55.5
50.0
Implicit
Gender
Age
64.3
65.9
53.5
53.8
37.7
40.7
55.4
53.1
61.5
54.5
70.3
71.4
$4.0
48.0
41.2
40.5
53.8
55.8
46.8
45.8
66.0
68.2
44,2
42.0
44.5
44.5
60.0
61.5
54.2
59.8
53.8
53.8
42

---

Table 25: The performance of detectors after injection under the standard setting (cont.).
LLAMA3
D01
D02
D04
D00
D08
D11
D13
D14
D16
Original
73.0
49.5
40.2
41.5
77.0
85.4
70.7
49.5
45.5
58.0
74.0
66.5
30.5
71.0
56.0
53.0
Vanilla
57.1
54.8
52.8
63.3
61.3
47.3
51.5
39.7
52.3
Sentiment
56.3
56.8
45.6
54.6
65.0
65.1
56.0
43.8
49.2
49.0
50.2
50.8
43.0
51.0
54.2
50.8
Explicit
Tone
63.5
50.0
48.5
67.0
61.3
43.5
53.8
65.8
39.2
50.5
Word Choice
58.7
51.5
41.7
50.0
68.0
79.2
66.7
47.2
46.2
51.2
66.2
67.0
35.2
67.0
59.0
54.8
Implieit
Age
63.5
50.0
53.1
72.5
48.7
44.0
51.8
52.0
37.2
51.5
Gender
62.7
49.0
44.1
56.9
71.0
73.4
46.7
49.0
56.2
52.5
51.2
40.0
36.5
62.0
52.5
53.5
BERT
D01
DM22
D03
D04
1M05
D06
D07
D48
D009
D10
D11
D12
D13
D14
D15
D16
Original
74.6
80.0
84.6
78.0
78.1
88.0
72.5
72.5
60.5
90.5
76.5
96.5
98.5
54.5
Vanilla
79.4
76.5
84.6
88.2
76.0
56.6
85.3
70.3
69.0
56.8
92.0
73.2
91.5
90.7
54.2
53.8
Sentiment
60.3
77.0
81.9
79.2
75.0
63.5
84.0
69.0
66.5
55.8
89.2
74,2
94.5
94.2
54.5
52.2
Explicit
Tone
73.8
78.2
83.8
86.2
78.0
63.0
80.7
69.5
68.0
58.0
88.8
71.8
92.2
92.0
55.0
53.2
Word Choice
76.2
77.0
81.5
74.3
69.8
57.8
90.5
54.8
Implicit
Gender
Age
74.6
50.8
73.2
74.8
78.4
77.0
78.5
81.5
82,5
82.0
72.9
90.7
90.0
67.2
69.0
68.0
61.5
62.5
91.0
88.5
74.0
74.5
95.5
95.0
97.2
96.8
53.2
54,2
53.0
54.5
DEBERTA
D01
D02
D04
D00
D08
D11
D16
Original
68.3
75.0
74.5
86.2
85.0
99.0
90.7
65.5
71.5
63.5
93.0
79.0
99.0
99.0
66.0
61.5
Vanilla
59.3
74.3
74.8
82.1
83.3
80.2
87.6
67.2
69.0
57.3
90.5
73.3
93.2
95.3
56.8
54.2
Sentiment
59.5
69.8
70.1
73.8
75.5
78.1
86.7
68.8
70.2
57.5
89.5
73.0
98.2 2
99.2
55.8
$4.0
Explicit
Tome
69.8
75.8
75.0
79.2
78.0
84.4
85.3
68.0
75.0
58.
88.8
75.5
94.8
96.5
57.2
58.8
Word Choice
64.3
74.0
76.0
82.3
77.5
80.2
86.0
66.8
74,8
58.8
88.2
75.5
96.0
98.5
58.2
56.2
Implicit
Gender
Age
52.4
51.6
73.0
71.2
74.0
71.1
74.6
79.2
83.5
82.0
95.8
93.8
83.3
89.3
67.5
64.8
71.2
71.0
60.2
62.5
88.5
89.5
74.0
74.8
97.5
97.5
98.8
98. 8
$4.5
57.2
56.8
58.5
CMTR
D01
D02
D03
D04
1M05
D06
D07
D48
1009
D10
D11
D12
D13
D14
D15
D16
Original
77.8
80.0
85.3
83.1
77.0
69.8
78.7
71.5
69.0
63.0
90.0
74.5
97.5
98.5
57.0
58.0
Vanilla
76.7
75.5
$2.0
84.6
71.3
55.6
85.3
70.2
66.0
59.2
92.2
72.3
88.7
90.0
55.8
53.5
Sentiment
61.9
76.0
73.0
83.8
69.0
55.7
82.7
68.0
64.0
57.2
90.2
73.0
93.2
93.5
54.8
53.2
Explicit
Tone
68.3
76.2
84.3
86.9
75.0
58.3
84.7
67.2
65.5
56.5
89.8
74.5
91.8
92.5
53.0
53.5
Word Choice
79.4
76.5
80.0
70.5
58.3
86.7
65.2
67.5
59.8
90.2
71.0
93.0
93.8
53.5
Impliett
Age
70.6
73.0
73.5
81.5
78.5
70.8
85.3
69.5
64,8
57.8
88.0
75.5
94.8
96.8
53.0
53.5
Gender
77.8
75.5
78.5
76.0
88.7
67.8
62.2
90.0
73.5
96.0
53.0
DisC
D01
D02
D03
D04
1305
D06
D07
D08
D09
D10
D11
D12
D13
D14
D15
D16
Original
73.0
79.0
85.3
89.2
81.0
78.1
85.3
69.5
72.0
60.5
90.5
78.0
98.0
98.5
59.5
56.5
Vanilla
78.3
77.2
84.3
88.7
78.3
58.7
85.8
68.7
66.3
57.3
91.8
74.5
92.7
93.0
52.5
52.5
Sentiment
63.5
77.8
78.5
74.5
61.5
84.0
66.8
56.8
89.5
53.5
Explicit
Tone
72.2
77.8
83.8
86.9
76.5
59.9
81.3
70.2
67.8
58.5
88.5
73.0
95.0
94.0
55.0
53.0
Word Choice
79.4
76.8
83.3
83,1
75.0
55.7
82.7
71.0
66.8
57.0
90.5
71.0
97.2
97.0
55.5
52.8
Implieit
Gender
Age
77.0
61.1
73.8
73.2
77.9
74.0
80.8
81.5
82.5
82.5
72.4
75.5
89.3
88.0
70.5
66.2
67.2
67.5
61.8
61.5
88.2
90.2
76.0
72.5
96.5
96.8
97.5
97.5
52.5
54.2
54.5
53.2
CATCH
D01
D02
D03
D04
DM05
D06
D007
D48
D09
D10
D11
D12
D13
D14
D15
D16
Original
73.0
76.5
72.5
90.8
84.0
92.7
88.0
67.5
73.5
60.0
91.5
73.5
98.0
100.0
53.0
56.5
Vanilla
75.1
76.3
84.1
74.0
87.6
66.7
57.8
91.3
73.7
91.7
90.3
55.7
Sentiment
71.4
75.2
76.0
82.3
74.0
67.7
86.0
67.5
70,3
58.0
88.2
72.5
96.8
96.5
55.8
53.8
Explicit
Tone
73.0
36.8
77.9
83.1
74.5
66.1
84.7
67.8
67.0
58.5
89.5
71.5
93.8
92.0
55.8
58.5
Word Choice
76.2
76.0
79.9
82.3
76.0
69.3
87.3
67.0
65.2
56.0
88.0
70.2
97.0
96.5
53.5
60.0
Implicit
Age
67.5
73.2
74.0
80.0
79.5
82.3
86.7
68.5
68.0
60.5
86.8
72.2
96.8
97.5
56.8
54.2
Gender
72.2
72.5
73,0
82.3
76.0
83.9
86.0
66.5
66.8
56.2
87.8
74.2
96.8
98.8
53.2
56.8
43


Reviews:
[{"review_id": "oRCEP9k2Oz", "reviewer": "Reviewer_TvfK", "summary": "Authors investigate the significance of shortcut learning in detecting misinformation in LLMs, mid-sized LMs, and ad hoc methods. The define two groups of potential shortcuts: intrinsic and extrinsic. Intrinsic includes sentiment, style, topic, and perplexity; extrinsic includes rewriting, explicit, and implicit. \n\nFor the intrinsic shortcuts, given 16 datasets (including 2 repurposed datasets) they selectively form training and test sets and show that the mentioned detection method significantly rely on the discussed shortcut to make their prediction. For the extrinsic shortcuts, they use an LLM and create synthetic sets for the mentioned datasets, and again reach the same conclusion.\n\nFinally to resolve the problem, they suggest that before training the detection methods, the data be pre-processed to remove the shortcuts. This method alleviate the problem to some degree--the argue that LLM based methods don't need this, and they are already robust.", "questions": "None", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n-  The experiments are very detailed and I think should be straightforward to reproduce\n-  The topic is very timely, and can help related practitioners\n-  The method is straightforward and intuitive\n-  The improvements are good\n\n**Weaknesses:**\n\n- It is almost impossible to be able to read the paper without reading the appendix, a lot of material is moved to Appendix, and there are a lot of cases that author refer to the tables and discussions in the appendix (just a quick search in the body of the paper confirms this). This has made me to seriously think that this is not the right paper for a conference, perhaps a journal would be a better fit \n- I don't think that Figure 4 is the right way to calculate authenticity.  Both metrics are very old methods for this and are not used anymore. There are more recent works, like those that measure hallucination (like FactScore or other suitable metrics).", "was_revised": true, "final_justification": "Increased from 4 to 5.", "rebuttal": "Thanks for your valuable feedback and for recognizing the timeliness of our topic, the comprehensiveness of our empirical experiments, and the effectiveness of our proposed approach.\n\n> It is almost impossible to be able to read the paper without reading the appendix, a lot of material is moved to Appendix, and there are a lot of cases that author refer to the tables and discussions in the appendix (just a quick search in the body of the paper confirms this). This has made me to seriously think that this is not the right paper for a conference, perhaps a journal would be a better fit.\n\n\nWe would like first to thank the appreciation of the comprehensiveness of our experiments. To ensure robustness, we evaluate seven baselines across 16 benchmarks, which inevitably results in a substantial volume of results. While we strive to keep the main text focused and accessible, **we have carefully ensured that all key findings are clearly supported within the main body.** Specifically:\n\n(1) LLMs can inject predefined factors without compromising content authenticity (Figures 3 and 4);\n\n(2) Existing datasets contain spurious correlations unrelated to authenticity (Figure 5); and\n\n(3) Misinformation detectors tend to rely on learned shortcuts (Figures 6–8), leading to performance degradation under varied news expressions (Tables 1–2).\n\nThe appendix serves to provide full experimental details and complete per-dataset results to support reproducibility and transparency. For instance, while Table 2 summarizes average performance across benchmarks, detailed per-dataset results are deferred to Tables 15 and 16 in the appendix for space considerations. We have made sure that all necessary information to follow the paper’s key contributions remains self-contained in the main text. Thanks for your comment, we will further review the balance and reduce dependence on the appendix where possible to improve clarity and alignment with conference constraints.\n\n\n\n\n> I don't think that Figure 4 is the right way to calculate authenticity. Both metrics are very old methods for this and are not used anymore. There are more recent works, like those that measure hallucination (like FactScore or other suitable metrics).\n\n\nOur primary goal in this figure is to verify that the rewritten news variants injected with extrinsic shortcuts in TruthOverTricks preserve the original article’s factual label -- a related yet more specific focus than general hallucination detection. To this end, we initially employed ROUGE-L and BERTScore to measure token-level and semantic-level similarity. While these metrics are indeed older, they remain widely adopted for such comparative analyses and, in our case, provided a meaningful signal—particularly when contrasted with random-shuffle baselines.\n\nTo further strengthen our claim, we conducted a human evaluation to validate data quality (see Lines 146-148), achieving an average accuracy of 91.2% and a Fleiss’ Kappa of 0.838. This indicates strong agreement that the injection did not alter the factual label.\n\nIn response to the reviewer’s suggestion, we additionally adopted an LLM-based evaluation approach similar in spirit to FactScore [1]. Specifically, we prompted GPT-4-turbo as follows:\n\n“Text 1: {Original article}\n\nText 2: {Injected article}\n\nDo these two texts describe the same event? Please answer yes or no.”\n\n\nThe percentage of “yes” responses across categories is summarized below: \n\n|Pairs|Percentage|\n|---|---|\n|random|0.12|\n|rewriting|0.92|\n|paraphrase|0.88|\n|open-ended|0.98|\n|positive|0.80|\n|negative|0.82|\n|simple|0.96|\n|complex|0.94|\n|formal|0.96|\n|informal|0.90|\n|young|0.98|\n|elder|0.96|\n|male|0.94|\n|female|0.98|\n\nThe results present that injection does not alter the labels evaluated by LLMs. Notably that LLMs judge that 12% of random shuffle pairs describe the same event, which shows that LLM-as-evaluator might not be reliable in this task. It coincides with the idea of utilizing metric measures and human evaluation.\n\nWe believe this work will be much stronger after the discussion of the LLM-as-evaluator.\n\n\n[1] Min, Sewon, et al. \"FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.\" The 2023 Conference on Empirical Methods in Natural Language Processing."}, {"review_id": "hQglPqp9C0", "reviewer": "Reviewer_RFGe", "summary": "This paper introduces TRUTHOVERTRICKS, a comprehensive evaluation framework for measuring and mitigating shortcut learning in misinformation detection systems. The authors identify that current misinformation detectors often rely on superficial cues (shortcuts) like sentiment, style, topic, and perplexity rather than genuine content authenticity. They evaluate seven detectors across 16 datasets, create two new factual misinformation datasets, and propose SMF (Shortcut Mitigation Framework) - an LLM-based data augmentation approach to improve detector robustness.", "questions": "How do you justify classifying sentiment, style, topic, and perplexity as \"shortcuts\" when extensive prior work has demonstrated their legitimate utility for misinformation detection?\n\nTo what extent do your experimental setups reflect realistic deployment scenarios where detectors would encounter misinformation?\n\nUnder what conditions should practitioners apply your SMF framework, and when might it actually hurt performance? Since you're essentially removing potentially useful features, how do practitioners determine whether their specific use case suffers from harmful shortcut learning versus beneficial feature utilization?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n* Novel Factual Datasets: The creation of NQ-Misinfo and Streaming-Misinfo datasets that require factual knowledge for detection fills an important gap in existing benchmarks.\n\n* Practical Solution: The SMF framework offers a model-agnostic approach to improving detector robustness through data augmentation rather than complex model modifications.\n\n* Strong Empirical Results: The paper demonstrates significant performance drops when detectors encounter distributional shifts, and shows SMF can improve robustness substantially.\n\n**Weaknesses:**\n\n* Problematic \"Shortcut\" Framing: The core conceptual issue is treating established, validated features (sentiment, style, topic, perplexity) as \"shortcuts\" when prior work [1] [2] have demonstrated their legitimate utility for misinformation detection. The paper lacks adequate justification for why these should be considered spurious rather than valid but insufficient features.\n\n* Artificial Evaluation Setup: The experimental design creates extreme distributional mismatches that may not reflect realistic deployment scenarios. The performance drops might indicate normal sensitivity to distribution shift rather than problematic shortcut learning.\n\n* Missing Causal Analysis: The paper provides insufficient evidence that the studied features lack causal relationships with misinformation, which is necessary to classify them as \"shortcuts.\"\n\n[1] Cui, Limeng, Suhang Wang, and Dongwon Lee. \"Same: sentiment-aware multi-modal embedding for detecting fake news.\" In Proceedings of the 2019 IEEE/ACM international conference on advances in social networks analysis and mining, pp. 41-48. 2019.\n\n[2] Przybyla, Piotr. \"Capturing the style of fake news.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 01, pp. 490-497. 2020.", "was_revised": true, "final_justification": "The authors' responses clarified some of my previous concerns, especially for the framing about \"Shortcut.\"", "rebuttal": "Thank you for your valuable feedback and your efforts in helping enhance our framing. We are encouraged that you recognize our contributions in developing complementary misinformation detection datasets, proposing a model-agnostic SMF approach, and demonstrating strong empirical results.\n\n> Problematic \"Shortcut\" Framing: The core conceptual issue is treating established, validated features (sentiment, style, topic, perplexity) as \"shortcuts\" when prior work [1] [2] have demonstrated their legitimate utility for misinformation detection. The paper lacks adequate justification for why these should be considered spurious rather than valid but insufficient features.\n> How do you justify classifying sentiment, style, topic, and perplexity as \"shortcuts\" when extensive prior work has demonstrated their legitimate utility for misinformation detection?\n\nWe acknowledge that features such as sentiment and style have been shown in prior work [1,2,3] to be useful. However, our concern lies not in the utility of these features but in their limitations under distributional shifts, particularly in dynamic real-world scenarios.\n\nRecent studies indicate that the boundaries between fake and real news are increasingly blurred. Real news has adopted more subjective and engaging tones [4], while fake news producers strategically mimic credible news sources to evade detection [5]. As shown in Figure 5, the distributions of these features (e.g., sentiment, style) overlap significantly between fake and real news, reducing their discriminative reliability. This suggests that detectors heavily relying on such superficial cues may learn dataset-specific correlations that do not generalize.\n\nFurthermore, recent work demonstrates that small stylistic edits (without altering factual content) can substantially degrade detector performance [6], highlighting a vulnerability. Motivated by this, we propose TruthOverTricks, a comprehensive evaluation framework to systematically assess the influence of these features. As shown in Tables 1 and 2, detectors often fail when confronted with style-shifted inputs, which supports our claim that these features can function as shortcuts—in the sense defined by Geirhos et al. [7]—that lead to spurious correlations rather than robust reasoning about news veracity.\n\nBy framing them as potential shortcuts, we do not dismiss their utility outright, but rather emphasize the importance of understanding when and how they might mislead models. Our work aims to identify such vulnerabilities and assist in the development of detectors that are more resilient to these shifts.\n\n[1] Cui, Limeng, et al. \"Same: sentiment-aware multi-modal embedding for detecting fake news.\" In Proceedings of the 2019 IEEE/ACM international conference on advances in social networks analysis and mining. 2019.\n\n[2] Przybyla, Piotr. \"Capturing the style of fake news.\" AAAI. 2020.\n\n[3] Zhang, Xueyao, et al. \"Mining dual emotion for fake news detection.\" TheWebConf. 2021.\n\n[4] Da San Martino, Giovanni, et al. \"Fine-Grained Analysis of Propaganda in News Article.\" EMNLP. 2019.\n\n[5] Shu, Kai, et al. \"Beyond news contents: The role of social context for fake news detection.\" Proceedings of the twelfth ACM international conference on web search and data mining. 2019.\n\n[6] Wu, Jiaying, Jiafeng Guo, and Bryan Hooi. \"Fake news in sheep's clothing: Robust fake news detection against LLM-empowered style attacks.\" SIGKDD. 2024.\n\n[7] Geirhos, Robert, et al. \"Shortcut learning in deep neural networks.\" Nature Machine Intelligence.\n\n> Artificial Evaluation Setup: The experimental design creates extreme distributional mismatches that may not reflect realistic deployment scenarios. The performance drops might indicate normal sensitivity to distribution shift rather than problematic shortcut learning.\n\n> To what extent do your experimental setups reflect realistic deployment scenarios where detectors would encounter misinformation?\n\nWe appreciate this opportunity to help resolve your concern. \n\nOur intent is not to create artificially adversarial conditions, but rather to reflect the diversity and unpredictability inherent in real-world misinformation. In practice, detectors are deployed in dynamic media environments where stylistic, topical, and linguistic variations are common, often intentionally manipulated by malicious actors to evade detection.\n\nFigures 10 and 11 demonstrate that existing detectors struggle particularly in cross-dataset and knowledge-intensive scenarios, suggesting limited generalization and a reliance on surface-level correlations. These patterns are indicative of shortcut learning—where models latch onto spurious but predictive features instead of developing a genuine understanding of factuality.\n\nTo probe this, TruthOverTricks simulates distributional shifts in both intrinsic and extrinsic dimensions. Importantly, we do not alter label distributions. In intrinsic settings, the test set remains unchanged, while in extrinsic settings, both metric- and human-based evaluations (lines 128–250) confirm that factual authenticity is preserved despite changes in style or expression. Nonetheless, the performance of all seven baseline detectors drops significantly across 16 benchmarks. This indicates that their predictions are influenced more by correlations with non-semantic features than by core content.\n\nIn this sense, our evaluation setup mirrors real-world deployment risks: where misinformation often appears in novel styles, tones, or contexts. By revealing vulnerabilities under such shifts, our framework serves as a diagnostic tool to better understand model behavior and improve robustness.\n\n> Missing Causal Analysis: The paper provides insufficient evidence that the studied features lack causal relationships with misinformation, which is necessary to classify them as \"shortcuts.\"\n\nWe would like to reinforce that our goal is not to argue that features such as sentiment or style are entirely irrelevant or should be excluded from misinformation detection. Rather, our objective is to evaluate the extent to which detectors may over-rely on these features, especially in contexts where they do not reliably indicate factuality.\n\nWe fully acknowledge that establishing a lack of causal relationship requires more than observing performance drops. However, TruthOverTricks is designed to empirically assess sensitivity to distributional shifts in these features, without altering ground-truth labels. When detector performance degrades significantly under these controlled interventions—despite the underlying factual content remaining constant (as verified by human and metric-based evaluation)—this suggests that the model is not robustly grounded in semantic understanding. While this does not constitute formal causal analysis, it provides practical evidence of shortcut-like behavior, following the definition from Geirhos et al. [1].\n\nImportantly, our TruthOverTricks framework is diagnostic, not prescriptive: it reveals potential model vulnerabilities and encourages the development of detectors that rely more heavily on core semantic content rather than superficial correlations. In this way, **we aim to contextualize the role of stylistic and linguistic features under realistic distribution shifts--not to dismiss their potential utility.**\n\n[1] Geirhos, Robert, et al. \"Shortcut learning in deep neural networks.\" Nature Machine Intelligence.\n\n> Under what conditions should practitioners apply your SMF framework, and when might it actually hurt performance? Since you're essentially removing potentially useful features, how do practitioners determine whether their specific use case suffers from harmful shortcut learning versus beneficial feature utilization?\n\nThe SMF framework is intended as a targeted intervention to address situations where models exhibit signs of shortcut learning—particularly when performance fails to generalize.\n\nOne practical indicator is a substantial accuracy gap between the validation and test sets, as shown in Figure 6. Such a gap suggests that the model may be overfitting to spurious correlations in the training data, rather than learning generalizable patterns of factuality. In these cases, applying SMF to augment the training set can encourage the model to rely less on superficial features and more on deeper semantic cues.\n\nPractitioners should consider applying SMF when: (1) There is evidence of generalization failure (e.g., large validation–test performance gaps), (2) Input distribution shifts are expected across deployment settings (e.g., varying writing styles or user demographics), or (3) Robustness to adversarial manipulation or stylistic variation is critical.\n\nConversely, in cases where such shifts are unlikely or when domain-specific stylistic patterns are known to correlate strongly with misinformation, applying SMF may be unnecessary. For example, in a tightly controlled domain such as scientific misinformation detection within biomedical literature, stylistic signals like hedging language or citation patterns may carry genuine predictive value. In such settings, these features may reflect valid, domain-specific cues rather than spurious correlations.\n\nTo balance these considerations, we recommend a two-step approach: (1) pre-train the detector with SMF-enhanced data to improve robustness, and (2) use the SMF-enhanced detector to evaluate individual cases or deployment settings where shortcut learning may be suspected. This offers a practical pathway to detect and mitigate shortcut reliance while preserving beneficial feature utilization when appropriate.\n\nThanks again for your thought-intriguing comments. We will incorporate this discussion to further refine our work."}, {"review_id": "MYiKc8odno", "reviewer": "Reviewer_4cY3", "summary": "This paper presents a framework designed to evaluate whether misinformation detection models rely on spurious correlations (such as sentiment, style, or topic) instead of factual content. The framework tests models in two settings: intrinsic shortcut induction, where natural biases in existing datasets are isolated, and extrinsic shortcut injection, where LLMs are used to rewrite content with specific superficial features to simulate adversarial attacks. The authors show that many models degrade significantly under these conditions, revealing a strong reliance on shortcuts. To address this, they propose SMF, an LLM-based data augmentation method that removes shortcut cues through paraphrasing, summarization, or style normalization. Experiments across 16 datasets demonstrate that SMF improves model robustness and encourages more reliable detection of misinformation across varied contexts.", "questions": "- Referring to lines 136-140, how did you handle the cases with (1)low agreement scores in implicit injection and (2) LLM refusal ?\n\n\n\n### Minor  things:\n- The expansion of SMF is not mentioned in the first use of the term in abstract or introduction", "limitations": "Yes. Included in the appendix", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n- The research is very well motivated, very well written and easy to follow. Probably also easy to replicate given the simplistic nature. \n- The novel contributions of the paper include the TRUTHOVERTRICKS framework, two factual misinformation benchmarks and a lightweight mitigation approach that improves model robustness by rewriting training data to reduce reliance on superficial features.\n- A systematic evaluation of shortcuts is conducted by looking into the surface level features in existing datasets as well as by injecting shortcuts using LLMs. The method is quite intuitive and can easily be adapted to other tasks like deception detection.\n- Extensive evaluation is conducted across datasets and models.\n\n\n### Weaknesses:\n- The paper discusses 4 shortcut types which are well evaluated and intuitive. However there is no discussion of other potential shortcuts like reliance on entities or persuasion strategies.\n- It is unclear from the main text how the failure in implicit injection was handled and how this influences the overall study.", "was_revised": true, "final_justification": "Authors have provided clarifications for the minor concerns I raised in the review. I will keep the score provided.", "rebuttal": "We thank the reviewer for the thoughtful observation and for recognizing the strengths of our work, including its clear motivation, systematic evaluation framework, and extensive experimentation.\n\n> The paper discusses 4 shortcut types which are well evaluated and intuitive. However there is no discussion of other potential shortcuts like reliance on entities or persuasion strategies.\n\nWe agree that shortcut learning in misinformation detection is a complex and multifaceted issue, especially in dynamic news environments. It is indeed not feasible to exhaustively cover all possible shortcut types within a single study. In this work, our aim was to establish a generalizable and extensible evaluation framework—TruthOverTricks—that enables both intrinsic and extrinsic analyses of shortcut behavior. To demonstrate its utility, we selected a set of representative shortcuts (four intrinsic and six extrinsic) and evaluated them across 16 benchmarks.\n\nOur findings show that existing detectors are consistently vulnerable to these diverse shortcut types, highlighting the need for more robust, semantically grounded detection strategies. Importantly, the modular design of TruthOverTricks allows for straightforward extension to other shortcut dimensions, such as entity overreliance or persuasive linguistic strategies, as suggested by the reviewer.\n\nWe appreciate this constructive suggestion and will add a discussion in the Conclusion to highlight these promising directions for future work. We hope that our framework will serve as a foundation for further studies exploring and mitigating additional forms of shortcut learning in misinformation detection.\n\n\n> It is unclear from the main text how the failure in implicit injection was handled and how this influences the overall study.\n\n> Referring to lines 136-140, how did you handle the cases with (1) low agreement scores in implicit injection and (2) LLM refusal?\n\n\n(1) For low agreement scores in implicit injection:\nWe acknowledge that human evaluators reported relatively low agreement when assessing whether the injected content reflected implicit attributes such as age or gender. Rather than viewing this as a limitation of the injection process, we interpret it more as a reflection of the subtle nature of these attributes in real-world news. Unlike explicit stylistic cues, implicit user attributes often manifest in more understated ways (e.g., through tone, reference framing, or content prioritization) which may not be easily distinguishable in isolated articles. This subtlety is, in fact, realistic and worth investigating, as it mirrors how news is adapted for different audiences in practice. The modest performance changes observed in Table 2 under implicit injection further support this view: detectors show limited response to these subtle changes, suggesting that current models may not pick up on subtle shifts in audience-targeted writing.  TruthOverTricks includes these cases to highlight the boundaries of model sensitivity and to motivate further research on better detecting and reasoning over implicit audience targeting.\n\n(2) For LLM refusals:\nIn our initial experimental design, we retained instances even when the LLMs partially or fully refused to follow the injection prompt. This choice preserved label distributions and ensured a consistent dataset size across original and shortcut conditions, allowing for a direct comparison.\n\nTo rigorously assess the influence of these refusals, we conducted an additional analysis. We filtered out all instances in which the LLM refused to perform the implicit injection. To maintain fairness, we also removed the corresponding original (non-injected) instances, ensuring pairwise alignment. The number of refusals per dataset is shown below:\n\n|Dataset|All Instances|Refused Instances (age)|Refused Instances (gender)|\n|---|---|---|---|\n|RumourEval| 315 | 90 | 79|\n|Pheme| 1000 | 283 | 236|\n|Twitter15| 507 | 160 | 151|\n|Twitter16| 319 | 127 | 122|\n|Celebrity| 500 | 28 | 33|\n|FakeNews| 480 | 59 | 55|\n|Politifact| 368 | 136 | 128|\n|Gossipcop| 1000 | 39 | 39|\n|Tianchi| 1000 | 296 | 293|\n|MultiLingual| 1000 | 193 | 203|\n|AntiVax| 1000 | 394 | 348|\n|COCO| 1000 | 589 | 552|\n|Kaggle1| 1000 | 191 | 195|\n|Kaggle2| 1000 | 215 | 196|\n|NQ-Misinfo| 1000 | 12 | 14|\n|Streaming-Misinfo| 1000 | 46 | 58|\n\nAfter re-running the experiments on these filtered datasets, we observed the following detector performance:\n\n||Mistralv3|Llama3|BERT|DeBERTa|CMTR|DisC|CATCH\n|---|---|---|---|---|---|---|---|\n|Original Age|55.0|57.8|78.2|78.2|77.7|79.1|79.3\n|Shortcut Age|56.9|56.6|74.9|74.9|75.2|75.5|76.4\n|Original Gender|55.6|57.3|78.0|78.8|77.0|77.6|78.6\n|Shortcut Gender|55.6|56.1|74.8|74.5|74.0|74.8|76.8\n\n\nThe results are consistent with our observation: detectors are generally less affected by implicit injection (Line 219). This further supports our interpretation that such injections are more subtle in nature, and that current detectors are not easily influenced by these minor shifts in writing style or framing. These findings strengthen our understanding of the limits of shortcut learning in subtle, implicit contexts and highlight the importance of continuing to explore such dimensions in future work.\n\nWe will include this extended analysis in the revised manuscript to clarify how implicit injection limitations were handled and their impact on our conclusions.\n\n> The expansion of SMF is not mentioned in the first use of the term in abstract or introduction\n\nThanks for your advice. We will include the complete name of SMF in the Abstract and Introduction to enhance clarity."}, {"review_id": "ezBuHSa2ET", "reviewer": "Reviewer_DwvC", "summary": "This paper investigates how biases (i.e, shortcuts) in misinformation classification datasets deteriorate the performance. It proceeds to propose a method using LLM data augmentation to counteract these biases. Overall, it supports its claims by studying various benchmarks and models.", "questions": "See above", "limitations": "see above", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "I like the paper and the paper setup. The paper presents a thorough overall evaluation. It employs human evaluation for the critical parts of the paper, namely injection fidelity and authenticity, when employing LLMs to alter texts. It also employs a vast number of models and benchmarks to support its main claims. This leads me to believe that the claims made in the paper are actually valid.\n\nHowever, I have to outline one major weakness. While the buildup of the hypothesis is very good (i.e., chapters 1-3), it is unfortunate to leave out LLMs in the Shortcut Mitigation Framework (SMF) (chapter 4). Your argumentation states that LLM-based detectors are already robust to shortcuts. This opens up two problems. First, while your method might solve a lot of issues for the other models, it might also introduce some areas where, actually, these \"shortcuts\"/cues are the only indicators. Thus, your method might decrease the scores for LLMs or cause other problems we don't know of yet. This should be investigated. It is not a big problem because then SMF is only suitable for simpler methods but it would reveal interesting dynamics when using SMF. I assume that discovering the up- and downsides of SMF more thoroughly would help in inspiring the community or yourselves to come up with an even better approach. Second, leaving out LLMs basically means that we do not know whether your method improves the SOTA. While it is not critical that your method improves the SOTA against LLMs, it is crucial for users to understand this. I would recommend exploring this. SMF is a major contribution here, and I think it is not yet fully explored.\n\nA minor point that would be interesting is how you deal with the fact that the SMF might reject rephrasing misinformation (as indicated in lines 138-140). What does this mean for SMF as a pipeline? Do you disregard these claims?\n\nMy conclusion: While I believe in the points made, I think important to understand the driving factors of the SMF and LLM investigations may reveal that. Does SMF solve some problems, but cause others? I believe all claims made, but those that are not made are very important as well.", "was_revised": true, "final_justification": "The responses strengthen my belief that this paper should be accepted.", "rebuttal": "Thank you for your thoughtful and constructive feedback. We're glad you appreciate our evaluation setup and comprehensive empirical results. We also appreciate your close reading of the SMF (Shortcut Mitigation Framework) and your insight that more fully exploring its boundaries would help strengthen its impact.\n\n> However, I have to outline one major weakness. While the buildup of the hypothesis is very good (i.e., chapters 1-3), it is unfortunate to leave out LLMs in the Shortcut Mitigation Framework (SMF) (chapter 4). Your argumentation states that LLM-based detectors are already robust to shortcuts. This opens up two problems. First, while your method might solve a lot of issues for the other models, it might also introduce some areas where, actually, these \"shortcuts\"/cues are the only indicators. Thus, your method might decrease the scores for LLMs or cause other problems we don't know of yet. This should be investigated. It is not a big problem because then SMF is only suitable for simpler methods but it would reveal interesting dynamics when using SMF. I assume that discovering the up- and downsides of SMF more thoroughly would help in inspiring the community or yourselves to come up with an even better approach. Second, leaving out LLMs basically means that we do not know whether your method improves the SOTA. While it is not critical that your method improves the SOTA against LLMs, it is crucial for users to understand this. I would recommend exploring this. SMF is a major contribution here, and I think it is not yet fully explored.\n\n\nThanks for raising this valuable point regarding the omission of LLM-based detectors from the SMF experiments. In response, we have extended our evaluation to include Mistral-v3 and LLaMA-3 under the SMF setup. The results are summarized below:\n\n||Mistralv3|Llama3|\n|---|---|---|\n|Vanilla w/o aug.|52.6|54.8|\n|Vanilla paraphrase|52.2|55.5|\n|Vanilla Summary|51.5|54.2|\n|Vanilla Neutral|51.9|52.6|\n|Sentiment w/o aug.|51.9|52.6|\n|Sentiment paraphrase|51.0|52.8|\n|Sentiment Summary|52.0|53.0|\n|Sentiment Neutral|51.0|51.6|\n|Tone w/o aug.|54.3|56.4|\n|Tone paraphrase|53.0|55.9|\n|Tone Summary|52.2|54.2|\n|Tone Neutral|52.7|54.3|\n|Word Choice w/o aug.|54.3|56.9|\n|Word Choice paraphrase|52.8|55.5|\n|Word Choice Summary|51.8|53.8|\n|Word Choice Neutral|51.3|53.6|\n|Age w/o aug.|54.5|53.8|\n|Age paraphrase|53.1|53.5|\n|Age Summary|51.2|52.1|\n|Age Neutral|54.0|52.8|\n|Gender w/o aug.|53.0|53.6|\n|Gender paraphrase|53.3|52.9|\n|Gender Summary|51.2|52.5|\n|Gender Neutral|53.7|52.0| \n\n\nThese results show that applying SMF to LLM-based detectors does not lead to significant gains or drops in performance. This supports our original hypothesis: LLM-based detectors are generally less affected by shortcut-based perturbations, likely due to their more holistic text understanding. However, your suggestion to include them was highly valuable—it provides a more complete picture and reinforces SMF's scope and limitations. We will include this analysis in the revised manuscript to help clarify that SMF offers more substantial benefits for non-LLM detectors, while also remaining compatible with LLM-based models.\n\nWe agree that in some cases, the so-called \"shortcuts\" (such as sentiment or tone) may contain useful signals, especially when core content is limited or ambiguous. This concern highlights why SMF should not be treated as a blanket filter for all stylistic features, but as a flexible tool that helps practitioners test and improve robustness when shortcut learning is likely. Our experiments show that SMF helps models avoid over-relying on such cues when they are misleading, but we acknowledge that more subtle shortcut utility remains an open research direction. We will add discussions on this trade-off in the revised manuscript to help guide practitioners.\n\n\n> A minor point that would be interesting is how you deal with the fact that the SMF might reject rephrasing misinformation (as indicated in lines 138-140). What does this mean for SMF as a pipeline? Do you disregard these claims?\n\n\nThank you for pointing this out, and we apologize for the confusion in the original phrasing. We would like to clarify that SMF does not reject rephrasing of misinformation. Instead, the refusals observed during implicit injection—particularly for age and gender prompts—are due to LLMs responding with messages such as “I cannot rewrite the passage as it contains misinformation.” This behavior reflects the safety alignment of the LLMs, which in some cases block generation when misinformation is detected, regardless of the stylistic prompt.\n\nTo assess how these refusals influence our results, we quantified the number of affected instances:\n\n\n|Dataset|All Instances|Refused Instances (age)|Refused Instances (gender)|\n|---|---|---|---|\n|RumourEval| 315 | 90 | 79|\n|Pheme| 1000 | 283 | 236|\n|Twitter15| 507 | 160 | 151|\n|Twitter16| 319 | 127 | 122|\n|Celebrity| 500 | 28 | 33|\n|FakeNews| 480 | 59 | 55|\n|Politifact| 368 | 136 | 128|\n|Gossipcop| 1000 | 39 | 39|\n|Tianchi| 1000 | 296 | 293|\n|MultiLingual| 1000 | 193 | 203|\n|AntiVax| 1000 | 394 | 348|\n|COCO| 1000 | 589 | 552|\n|Kaggle1| 1000 | 191 | 195|\n|Kaggle2| 1000 | 215 | 196|\n|NQ-Misinfo| 1000 | 12 | 14|\n|Streaming-Misinfo| 1000 | 46 | 58|\n\nSince these refusals could skew comparisons, we filtered out the refused samples as well as their original counterparts. The following results show that removing these instances does not significantly alter the conclusions:\n\n||Mistralv3|Llama3|BERT|DeBERTa|CMTR|DisC|CATCH\n|---|---|---|---|---|---|---|---|\n|Original Age|55.0|57.8|78.2|78.2|77.7|79.1|79.3\n|Shortcut Age|56.9|56.6|74.9|74.9|75.2|75.5|76.4\n|Original Gender|55.6|57.3|78.0|78.8|77.0|77.6|78.6\n|Shortcut Gender|55.6|56.1|74.8|74.5|74.0|74.8|76.8\n\n\nThese results support our observation: detectors are less affected by implicit injections (Line 219), and the presence or absence of these blocked samples does not change the broader findings. The refusal behavior by LLMs also serves as an important caution: many current LLMs are designed to avoid rewriting misinformation, even when prompted for stylistic variation. This safety-aligned behavior can affect the completeness of data augmentation pipelines and should be taken into account when applying or interpreting methods like SMF in practice."}]

Author rebuttal:

Meta-review: This paper presents TruthOverTricks framework for diagnosing and mitigating shortcut learning in misinformation detection. It evaluates several detectors across common benchmark datasets and shows that models often rely on superficial cues such as sentiment, style, and topic, leading to poor generalization under distribution shifts. To address this, they propose an LLM-based data augmentation method that improves robustness by reducing reliance on such cues. Reviewers found the work timely, well-motivated, and empirically thorough, with strong practical relevance and reproducibility. The rebuttal provided valuable clarifications that strengthen the paper and will be important for the camera-ready version. The paper would further benefit from stronger grounding in the existing literature, particularly recent work on LLM-based misinformation detection and studies discussing dataset limitations and spurious correlations in misinformation benchmarks.
Decision: Accept (poster)