When Claims Evolve: Evaluating and Enhancing the Robustness of Embedding Models Against Misinformation Edits
Authors: Jabez Magomere, Emanuele La Malfa, Manuel Tonneau, Ashkan Kazemi, Scott A. Hale
Accepted at: acl25

When Claims Evolve: Evaluating and Enhancing the Robustness of Embedding Models Against Misinformation Edits

Jabez Magomere,1 Emanuele La Malfa,1,2 Manuel Tonneau,1,3,4 Ashkan Kazemi,5 Scott A. Hale1,5
1University of Oxford, 2Alan Turing Institute, 3World Bank, 4New York University, 5Meedan
jabez.magomere@keble.ox.ac.uk

Abstract

Online misinformation remains a critical challenge, and fact-checkers increasingly rely on claim matching systems that use sentence embedding models to retrieve relevant fact-checks. However, as users interact with claims online, they often introduce edits, and it remains unclear whether current embedding models used in retrieval are robust to such edits. To investigate this, we introduce a perturbation framework that generates valid and natural claim variations, enabling us to assess the robustness of a wide-range of sentence embedding models in a multi-stage retrieval pipeline and evaluate the effectiveness of various mitigation approaches. Our evaluation reveals that standard embedding models exhibit notable performance drops on edited claims, while LLM-distilled embedding models offer improved robustness at a higher computational cost. Although a strong reranker helps to reduce the performance drop, it cannot fully compensate for first-stage retrieval gaps. To address these retrieval gaps, we evaluate train- and inference-time mitigation approaches, demonstrating that they can improve in-domain robustness by up to 17 percentage points and boost out-of-domain generalization by 10 percentage points. Overall, our findings provide practical improvements to claim-matching systems, enabling more reliable fact-checking of evolving misinformation.

<table>
  <tr>
    <th>Dialect</th>
    <td>Covid-19 nuh more deadly dan di seasonal flu'</td>
    <th>Entity Repl.</th>
    <td>Corona virus is only as deadly as the common cold</td>
  </tr>
  <tr>
    <th>Typos</th>
    <td>COVID-19 is onli as dedli as the sizni flu</td>
    <th>Casing</th>
    <td>COVID-19 IS ONLY AS DEADLY AS THE SEASONAL FLU</td>
  </tr>
  <tr>
    <th>LLM Rewrite</th>
    <td>COVID-19 is no more deadly than the seasonal flu.</td>
    <th>Negation</th>
    <td>Covid-19 is not as deadly as the seasonal flu</td>
  </tr>
  <tr>
    <th>Claim</th>
    <td>Covid-19 is only as deadly as the seasonal flu</td>
    <th>Fact Check:</th>
    <td>COVID-19 has a significantly higher mortality rate than seasonal flu, with greater severity and hospitalizations</td>
  </tr>
</table>

Figure 1: Examples of user-informed misinformation edits observed in fact-checking. Users frequently introduce subtle edits to online misinformation claims that preserve the corresponding fact-check. However, it remains unclear whether sentence embedding models are robust to these edits.

One such approach is claim matching, the task of determining whether a given claim has already been fact-checked or retrieving relevant fact-checks (Shaar et al., 2020; Sheng et al., 2021; Shaar et al., 2022; Kazemi et al., 2021). Claim matching has shown strong potential in reducing the time human fact-checkers spend verifying repeated or previously debunked claims, by ensuring that once a costly human fact-check is written, it can be linked to all relevant instances of that claim to provide context to others encountering it. However, current claim matching approaches are typically trained on datasets composed of previously fact-checked claims (Barrón-Cedeno et al., 2020; Nakov et al., 2022), excluding the vast majority of unverified claims in the wild. This limits models’ exposure to the noisy, diverse, and evolving nature of misinformation as it appears in real-world contexts. As a result, these models may perform well in controlled, in-domain evaluations but generalize poorly to settings characterized by unpredictable
Figure 2: An Overview of Our Approach: (1) We generate candidate rewrites using an LLM As a Perturber, followed by another LLM As a Verifier of these perturbations. (2) Next, we assess the robustness of various embedding models within a multi-stage retrieval pipeline. (3) Finally, we examine the impact of train- and inference-time mitigation approaches on improving robustness for weaker embedding models.

user behaviour.

As users interact with and share misinformation online, they often introduce various edits to claims, such as replacing entities, introducing common social-media slang, or rewriting a claim in a different dialect (Kazemi et al., 2023; Quelle et al., 2025) as shown in Figure 1. While these edits may be subtle and not affect the matching fact-check, Yan et al. (2022) show that mutated claims are more likely to spread widely on social media compared to non-mutated claims. Thus, current claim-matching methods need to be robust to user claim edits to curb the spread of misinformation effectively.

At the core of claim matching tasks is the reliance on sentence embedding models to represent input claims and fact-checks during retrieval. There has been growing interest in developing general-purpose sentence embeddings designed to handle a wide range of tasks (Lee et al., 2024; Su et al., 2023). However, their effectiveness in task-specific applications remains uncertain, as different domains demand varying notions of similarity (Ghafouri et al., 2024). Fact-checking is one such domain where fact-checkers need to retrieve sentences that may be evaluated under the same factual basis, even if they are not strictly semantically equivalent, a nuance that general-purpose embeddings mostly trained on semantic textual similarity datasets (Cer et al., 2017) may struggle to capture.

To tackle the challenge of embedding model brittleness in real-world settings, we study the robustness of sentence embedding models to user-driven misinformation edits and propose mitigation strategies to improve robustness. We introduce a taxonomy of real-world edits drawn from fact-checking scenarios and evaluate model robustness on the downstream task of claim matching (Figure 2). Our analysis shows that widely used embedding models are highly sensitive to user-informed misinformation edits, and that even strong rerankers fail to fully mitigate these weaknesses. However, we demonstrate that both train-time and inference-time mitigation approaches can significantly improve in-domain and out-of-domain robustness. Based on these findings, our key contributions are as follows:1

• We propose a novel perturbation framework for generating valid and natural claim edits, resulting in a new dataset for evaluating robustness of sentence embedding models.
• We conduct a comprehensive evaluation of a wide range of embedding models, including BERT-based, T5-based, and LLM-distilled in a multi-stage retrieval pipeline comprising first-stage retrieval followed by reranking.
• We assess the effectiveness of both train-time and inference-time mitigation approaches in improving the robustness of weaker embedding models, providing insights into their trade-offs and effectiveness.

2 Related Work

Robustness in Misinformation Detection Robustness in NLP is a well-studied area, with research focusing on developing adversarial attacks that can fool models and compromise real-world deployments (Malfa and Kwiatkowska, 2022; Wang

1Code and datasets are available for research purposes.
et al., 2022; Goyal et al., 2023). In the context of misinformation, prior work has explored adversarial robustness in text classifiers for fake news detection (Zhou et al., 2019; Le et al., 2020; Flores and Hao, 2022), veracity prediction (Thorne et al., 2019; Hidey et al., 2020), and propaganda detection (Przybyla et al., 2024). However, fact checkers have evidenced a lack of transparency and credibility on automated fact-checking approaches that assign factuality labels to claims (Glockner et al., 2022; Das et al., 2023; Procter et al., 2023). In contrast, the robustness of claim matching—a core information retrieval task—remains underexplored, with prior research focusing on improving retrieval performance (Barrón-Cedeno et al., 2020; Sheng et al., 2021; Nakov et al., 2021b, 2022). Most studies on robustness in misinformation rely on adversarial techniques, such as character- and word-level substitutions (e.g., synonym replacements; Jin et al., 2019; Li et al., 2020; Garg and Ramakrishnan, 2020), to perturb input claims. However, the practicality of these techniques has been questioned, as recent studies show that such attacks can mislead NLP models but have little impact on human readers (Morris et al., 2020a; Dyrmishi et al., 2023a; Chiang and Lee, 2023). There is also a well-known trade-off between how realistic a local adversarial attack is and its plausibility for humans (Malfa and Kwiatkowska, 2022). This issue is even more critical in misinformation contexts, where the goal is to not only fool a model but also to generate believable claims and spread content among human users; therefore, the perturbations should be both valid and natural (Dyrmishi et al., 2023a). In contrast to adversarial robustness which focuses on exploiting model vulnerabilities, our work focuses on realistic user edits: natural variations in misinformation that arise organically, either intentionally or unintentionally, as claims spread online.

Evaluating Embedding Model Robustness Several studies have examined the robustness of embedding models across different settings. Rafiei Asl et al. (2024) evaluate robustness of embedding models adversarially using TextAttack (Morris et al., 2020b) and propose using gradient-based (Goodfellow et al., 2014) perturbed embeddings to enhance robustness. Chiang et al. (2023) examine how different sentence encoders represent sentence pairs with high lexical overlap, showing that training datasets influence similarity notions. Closely related to our work, Nishimwe et al. (2024) assess LASER (Heffernan et al., 2022) against user-generated content and apply knowledge distillation (Reimers and Gurevych, 2020) to improve robustness. However, prior work largely evaluates embeddings in isolation, despite their role in larger systems: e.g., candidate selection in multi-stage retrieval (Nogueira et al., 2020) or retrieval-augmented generation (RAG) pipelines (Fan et al., 2024). The impact of representation differences on end-to-end system performance remains unclear. For example, we examine whether the robustness of embedding models in first-stage retrieval meaningfully affects the overall performance of a multi-stage retrieval system when a strong reranker is applied, addressing this gap.

3 Data and Methods

3.1 Background

3.1.1 Task Definition
Following Shaar et al. (2020), we define the task of claim matching as a learning-to-rank problem. Given a check-worthy input claim \( q \) and a set of verified claims or fact-checks \( V = \{ v_1, v_2, \ldots, v_n \} \), the objective is to learn a ranking function \( f \) that orders the verified claims such that those that can help verify the input claim \( q \) are ranked higher. \( f \) returns a ranked list \( [v_{\pi(1)}, v_{\pi(2)}, \ldots, v_{\pi(n)}] \), where \( \pi \) is a permutation of indices that sorts the claims in descending order of relevance to \( q \). Model performance on this task is evaluated using Mean Average Precision (MAP) score truncated to rank \( k \) (MAP@k). We compare claims with a sentence embedding model \( \theta \) that encodes the input claim and fact-checks into dense vector representations. These embeddings are then utilized in learning the ranking function \( f \).

3.1.2 Problem Formulation
We consider a scenario where a social media user \( u \) edits an original claim \( q \) to produce a perturbed claim \( q' \). The edits must ensure that \( q' \) is both valid, remaining applicable to the fact-check \( v \) and conveying the same claim as \( q \) and natural, meaning that \( q' \) is something a social media user might write. These edits are modeled as a set of transformations \( \mathcal{T} \), each applied with a noise level \( b \) drawn from a budget \( B_t = [b_{low}^t, b_{high}^t] \) for \( t \in \mathcal{T} \). We vary the noise levels to capture the spectrum of user edits, from minimal edits (low noise) to substantial rewording (high noise). A set of constraints \( \mathcal{C} \) enforces the validity and naturalness of \( q' \). Formally,
this process is described by

\[
q' = t_b(q) \quad \text{with} \quad C(q, q', v) = \text{True}
\]

where given an input claim \( q \), the baseline edit is defined as \( q_{\text{base}}^b = t_{b_{\text{low}}}^b(q) \) and the worst-case edit as \( q_{\text{worst}}^b = t_{b_{\text{high}}}^b(q) \). This formulation systematically examines how varying noise levels within \( B_t \) impact sentence embedding models in claim matching tasks.

3.1.3 Perturbations as Claim Edits
We introduce a taxonomy of common misinformation edits observed in real-world fact-checking, which serves as a template for the transformations \( t \in \mathcal{T} \). The applied transformations and noise levels are as follows:

Casing A subtle transformation observed in claims involves changes in casing, such as upper-casing an entire claim or parts of it to add emphasis. In other cases, casing differences may be artefacts of the noisy nature of social media text (Ritter et al., 2011; Baldwin et al., 2015). To capture casing variations, we apply TrueCasing (Lita et al., 2003)\footnote{A preprocessing technique that converts all-lowercase text to properly capitalized text and introduces variations in text.} to the original claim as the minimum amount of noise (\( b_{\text{low}} \)) and UPPERCASE the entire input claim as the maximum noise level (\( b_{\text{high}} \)).

Typos User edits to a claim may introduce spelling errors, common social media abbreviations and slang (Van Der Goot et al., 2018; Sanguinetti et al., 2020). We define a transformation budget based on the Levenshtein edit distance between the unperturbed and perturbed input claims. The baseline budget, \( b_{\text{low}} \), corresponds to the perturbed claim with the smallest edit distance, while the worst-case budget, \( b_{\text{high}} \), represents the largest edit distance to the unperturbed claim.

Negation Social media users may negate claims, for instance as part of counterspeech (Gligoric et al., 2024), though the underlying claim to be verified remains unchanged. To evaluate embedding models on such cases, we apply negation transformations to original claims, defining the minimum noise level (\( b_{\text{low}} \)) as a single negation and the maximum noise level (\( b_{\text{high}} \)) as a double negation. For example, the claim “Covid-19 is only as deadly as the seasonal flu” with a single negation becomes “Covid-19 is not as deadly as the seasonal flu”. With double negation the same claim becomes “It is not true that Covid-19 is not as deadly as the seasonal flu”.

Entity Replacement One common pattern in mutated claims is the substitution of entities in the original text with synonymous references, such as replacing a well-known person’s name with a nickname or changing entities to tailor a claim to a new audience; for example, changing “According to a Harvard study . . .” to “According to an Oxford study . . .” while retaining the original meaning. We define the transformation budget as the number of named entities to be replaced in a claim, with \( b_{\text{low}} = 1 \), meaning at most one entity is replaced, and \( b_{\text{high}} = All \), meaning all entities are replaced.

LLM rewrites Chen and Shu (2024) demonstrate that LLMs can be used to rewrite a claim, either to conceal authorship or to make the claim more deceptive and harder to detect. We prompt an LLM to rewrite an original claim, setting the transformation budget based on the Levenshtein distance, where \( b_{\text{low}} \) is the rewrite with the fewest changes, and \( b_{\text{high}} \) is the rewrite with the most changes.

Dialect Changes A social media user may rewrite a claim in a different dialect to fit a specific target audience. However, most NLP models are primarily trained on American and British English (Longpre et al., 2024; Tonneau et al., 2024a), leading to notable performance drops when applied to other English dialects (Jurgens et al., 2017; Ziems et al., 2023; Tonneau et al., 2024b). To study the effect of dialectal variations, we rewrite claims in African American English, Nigerian Pidgin English, Singlish (an English variant spoken in Singapore) and Jamaican Patois, an English-based creole spoken in the Caribbean.\footnote{We treat all dialects equal and assume similar noise levels.}

3.2 Experimental Setup

3.2.1 Perturbation Generation Framework
To generate perturbations that meet the constraints outlined in Section §3.1.2, we implement a two-stage, multi-prompting framework leveraging two separate LLMs: one for perturbing input claims and another for verifying them. In the perturbation step, the perturber LLM takes an input claim \( q \) and its corresponding fact-check \( v \) as inputs to generate an initial set of \( N \) candidate rewrites intended to satisfy the transformation requirements and constraints. To select valid perturbations, we implement a verification step, where another LLM
assumes the role of verifier and checks whether the generated rewrites adhere to all constraints. Each perturbation is assigned a binary label (1 for valid, 0 for invalid), and invalid perturbations are discarded. We set the initial candidate size \( N \) to 5 and use GPT4o as both the perturber and verifier. A human evaluation of the perturbation framework shows that the perturber applies perturbations with 96.70% accuracy, while the verifier achieves 100% precision in selecting valid perturbations. However, the verifier’s lower recall of 55.67% suggests that it may be overly conservative, potentially filtering out some valid instances. Further details on the human evaluation and prompts are provided in Sections §B and §I, respectively.

3.2.2 Datasets
We apply perturbations to two claim-matching datasets: CheckThat22 (Nakov et al., 2022) and the dataset proposed by Kazemi et al. (2022), referred thereafter as FactCheckTweet. CheckThat22 matches tweets with short, verified claims, whereas FactCheckTweet matches claims with full fact-check articles. We apply perturbations to the test splits only (see Appendix §A). To assess generalization i.e. ability to improve performance on novel, out-of-distribution claims, we test our mitigation approaches on an out-of-domain (\( OOD \) Dataset) composed of novel claim–fact-check pairs provided by Meedan\footnote{The non-profit Meedan supports fact-checkers running misinformation tiplines on WhatsApp and other platforms.} and primarily sourced through five fact-checking organizations. Table 1 summarizes the descriptive statistics of these datasets.

<table>
  <tr>
    <th>Dataset</th>
    <th># Claim</th>
    <th># FC</th>
    <th>Claim Len</th>
    <th>FC Len</th>
  </tr>
  <tr>
    <td>CheckThat22</td>
    <td>2,592</td>
    <td>14,231</td>
    <td>42.3</td>
    <td>19.2</td>
  </tr>
  <tr>
    <td>FactCheckTweet</td>
    <td>3,578</td>
    <td>59,951</td>
    <td>37.9</td>
    <td>682.1</td>
  </tr>
  <tr>
    <td><i>OOD</i> Dataset</td>
    <td>721</td>
    <td>1,261</td>
    <td>150.1</td>
    <td>80.4</td>
  </tr>
</table>

Table 1: Statistics of datasets in this work. FC denotes target fact-checks; lengths are measured in words.

3.2.3 Multi-Stage Retrieval Pipeline
To evaluate the robustness of various embedding models, we adopt a common multi-stage retrieval pipeline (Nogueira and Cho, 2019; Nogueira et al., 2019a; Ma et al., 2024) consisting of a computationally efficient retriever model to return top-\( j \) relevant fact-checks followed by a more computationally intensive reranker that refines the retrieved candidates to improve ranking quality. We describe the models we evaluate for each stage below:

Retrievers In the retrieval stage, we evaluate embedding models with varying dimensions, pre-training approaches, and backbone architectures. This includes:
BERT-based embedding models (Devlin et al., 2019; Reimers and Gurevych, 2019): all-MiniLM-L12-v2, all-mpnet-base-v2, and all-distilroberta-v1. T5-based embedding models (Ni et al., 2022; Raffel et al., 2020): sentence-t5-base, sentence-t5-large, instructor-base, and instructor-large—instruction-finetuned embeddings based on GTR models (Su et al., 2023).
Decoder-only LLM-distilled embeddings: SFR-Embedding-Mistral (Meng et al., 2024) and NV-Embed-v2 (Lee et al., 2025),\footnote{NV-Embed-v2 was the top-performing model on MTEB (Muenninghoff et al., 2022) at the time of this study. https://huggingface.co/spaces/mteb/leaderboard} both initialized from Mistral-7B. To study the effect of finetuning, we finetune all-mpnet-base-v2 and sentence-t5-large on the CheckThat22 training set, and denote the finetuned models respectively as all-mpnet-base-v2-ft and sentence-t5-large-ft. We also implement BM25 as a baseline to explore the robustness of lexical-based sparse retrieval approaches. We provide further details on embedding models and finetuning in §C of the Appendix.

Rerankers With multiple combinations of embedding models (\( m = 12 \)), perturbation types (\( p = 14 \)), and rerankers (\( r = 7 \))—totalling \( m \times p \times r = 1,176 \) cases, we evaluate a wide range of rerankers and select the best-performing one for further experiments. While RankGPT (Sun et al., 2023) using GPT4o achieves the highest performance, we select bge-reranker-v2-gemma (Chen et al., 2024) for its comparable accuracy and significantly lower inference cost. See §D.1 in the Appendix for reranker evaluations.

4 Results

4.1 RQ1: How do perturbations affect the robustness of embedding models in first-stage retrieval?
We begin by evaluating the robustness of embedding models used for first-stage retrieval under various perturbation types. For each embedding model and perturbation type, we compute the \( \Delta_{retrieval\ gap} \) by comparing the MAP@k performance difference between the unperturbed and
<table>
  <tr>
    <th rowspan="2">Stage Model</th>
    <th rowspan="2">Casing</th>
    <th colspan="2">TrueCore</th>
    <th colspan="4">Epper</th>
    <th colspan="5">Logic</th>
    <th colspan="2">Negation</th>
    <th colspan="4">Entity Attribution</th>
    <th colspan="4">LLM Rewrite</th>
    <th colspan="2">Dialect</th>
    <th rowspan="2">Simplist</th>
    <th rowspan="2">PADEM</th>
  </tr>
  <tr>
    <th>Upper</th>
    <th>Lower</th>
    <th>Neg</th>
    <th>Mot</th>
    <th>Shallow</th>
    <th>Englishable</th>
    <th>Clause</th>
    <th>Motive</th>
    <th>Mojor</th>
    <th>Misinforo</th>
    <th>MAP</th>
    <th>AEP</th>
    <th>Nonverbal</th>
    <th>VF</th>
    <th>POS</th>
    <th>PP</th>
    <th>AEP</th>
    <th>Mention</th>
    <th>Entity</th>
    <th>Cause</th>
    <th>Literary</th>
    <th>Motivation</th>
    <th>DeepL</th>
  </tr>
  <tr><td rowspan="10">CheckThat22</td><td rowspan="2">MML</td><td>±0.0</td><td>±0.0</td><td>-15.2</td><td>-15.0</td><td>+0.5</td><td>±6.7</td><td>-0.9</td><td>0.8</td><td>0.8</td><td>+0.8</td><td>-0.0</td><td>0.2</td><td>-0.0</td></tr>
  <tr><td colspan="26" rowspan="26"></td></tr>
  <tr><td>all-distil/roberta-v1</td><td>-5.4</td><td>-5.3</td><td>-7.8</td><td>-7.9</td><td>4.8</td><td>-2.6</td><td>-0.4</td><td>-4.8</td><td>3.5</td><td>-0.9</td><td>0.4</td><td>-7.3</td><td>-0.0</td><td>+0.0</td></tr>
  <tr><td>all-multit4.1-v2</td><td>-4.0</td><td>-4.0</td><td>-3.5</td><td>-3.5</td><td>8.6</td><td>-2.5</td><td>-0.8</td><td>2.5</td><td>2.9</td><td>-0.9</td><td>1.1</td><td>-7.0</td><td>0.1</td><td>+0.1</td></tr>
  <tr><td>all-embed-v2</td><td>-4.0</td><td>-4.0</td><td>-8.3</td><td>-8.4</td><td>-0.2</td><td>-3.1</td><td>-1.9</td><td>+18.8</td><td>+2</td><td>-2.1</td><td>-3.8</td><td>5.2</td><td>-3.8</td><td>-1.3</td><td>+4.1</td></tr>
  <tr><td>all-npsmt-base-v2</td><td>-5.1</td><td>-5.2</td><td>-5.3</td><td>-5.4</td><td>+6.2</td><td>-1.1</td><td>1.8</td><td>+8.0</td><td>+4.6</td><td>-0.5</td><td>0.5</td><td>8.1</td><td>-0.9</td><td>+0.0</td></tr>
  <tr><td>all-npsmt-base-v2-ft</td><td>-5.1</td><td>-5.2</td><td>-2.1</td><td>-1.4</td><td>-5.2</td><td>-2.7</td><td>1.4</td><td>2.7</td><td>-6.7</td><td>0.8</td><td>0.2</td><td>-2.5</td><td>-1.3</td><td>0.7</td></tr>
  <tr><td>sentence-t5-base</td><td>-2.3</td><td>-2.4</td><td>-3.0</td><td>-3.0</td><td>+8.0</td><td>+2.1</td><td>-2.2</td><td>+1.0</td><td>6.6</td><td>4.6</td><td>4.5</td><td>+1.6</td><td>+1.2</td><td>-4.6</td><td>0.1</td></tr>
  <tr><td>sentence-t5-large</td><td>-1.0</td><td>-0.9</td><td>-2.1</td><td>-5.4</td><td>+2.2</td><td>-14.7</td><td>-4.9</td><td>-5.4</td><td>+1.8</td><td>+1.7</td><td>-0.8</td><td>+3.4</td><td>-3.2</td><td>+2.2</td></tr>
  <tr><td>sentence-t5-large-ft</td><td>-0.6</td><td>-0.7</td><td>+8.0</td><td>+7.7</td><td>+3.6</td><td>+11.7</td><td>-2.6</td><td>-1.8</td><td>+2.8</td><td>+0.4</td><td>+0.3</td><td>+0.2</td><td>+1.8</td><td>-1.2</td><td>-5.6</td></tr>
  <tr><td>instructor-base</td><td>+1.1</td><td>+1.2</td><td>+3.2</td><td>+4.1</td><td>-3.8</td><td>+1.1</td><td>0.2</td><td>+0.1</td><td>+8.8</td><td>+3.2</td><td>+1.1</td><td>+0.2</td><td>+1.2</td><td>+1.0</td><td>4.6</td></tr>
  <tr><td>instructor-large</td><td>+0.7</td><td>+0.7</td><td>+1.1</td><td>-1.7</td><td>+0.4</td><td>+9.6</td><td>+1.2</td><td>-0.0</td><td>0.2</td><td>0.1</td><td>0.1</td><td>+0.1</td><td>0.0</td><td>0.9</td><td>-0.3</td><td>-1.2</td></tr>
  <tr><td>SFR-EmbeddingMistral</td><td>-0.6</td><td>-0.6</td><td>-0.1</td><td>+1.0</td><td>+0.2</td><td>+0.5</td><td>-0.2</td><td>+0.1</td><td>0.1</td><td>0.0</td><td>-0.1</td><td>0.1</td><td>0.0</td><td>+1.1</td><td>0.0</td><td>-0.2</td></tr>
  <tr><td>NV-Embed-v2</td><td>-0.0</td><td>-0.0</td><td>-0.0</td><td>-0.0</td><td>0.0</td><td>-0.0</td><td>+0.2</td><td>-0.0</td><td>+0.0</td><td>+0.0</td><td>+0.0</td><td>+0.1</td><td>+0.2</td><td>-0.2</td><td>0.0</td><td>+0.2</td></tr>
  <tr><td rowspan="10">FactCheckTweet</td><td>BMQE</td><td>+10.9</td><td>+10.7</td><td>+42.9</td><td>+50.0</td><td>+46.1</td><td>+7.2</td><td>+10.7</td><td>+19.8</td><td>+9.8</td><td>+9.3</td><td>+7.8</td><td>+8.9</td><td>+41.5</td><td>+0.6</td><td>+11.1</td></tr>
  <tr><td>all-distil/roberta-v1</td><td>+2.0</td><td>+1.8</td><td>+6.2</td><td>+7.1</td><td>+14.1</td><td>+1.8</td><td>+2.1</td><td>+2.3</td><td>+0.1</td><td>+0.2</td><td>+1.6</td><td>+2.9</td><td>+1.0</td><td>+0.1</td><td>+2.5</td></tr>
  <tr><td>all-multit4.1-v2</td><td>+1.3</td><td>+1.2</td><td>+6.8</td><td>+10.2</td><td>+3.2</td><td>+4.5</td><td>+1.2</td><td>+2.0</td><td>+1.8</td><td>+1.7</td><td>+1.3</td><td>+3.4</td><td>+1.6</td><td>+0.3</td><td>+1.4</td></tr>
  <tr><td>all-npsmt-base-v2</td><td>+1.8</td><td>+1.9</td><td>+1.2</td><td>+8.8</td><td>+12.3</td><td>+5.1</td><td>+2.0</td><td>-0.5</td><td>4.9</td><td>+15.3</td><td>0.9</td><td>4.4</td><td>+6.8</td><td>0.1</td><td>-3.7</td></tr>
  <tr><td>all-npsmt-base-v2-ft</td><td>+5.9</td><td>+6.3</td><td>+10.0</td><td>+11.5</td><td>+0.9</td><td>+3.3</td><td>+3.4</td><td>+13.2</td><td>+11.8</td><td>+0.8</td><td>+0.2</td><td>+0.7</td><td>+0.8</td><td>1.0</td><td>-2.2</td></tr>
  <tr><td>sentence-t5-base</td><td>+5.4</td><td>+5.0</td><td>+12.4</td><td>+17.8</td><td>+0.0</td><td>+13.8</td><td>+15.5</td><td>+4.0</td><td>+20.6</td><td>+13.8</td><td>+12.3</td><td>+15.5</td><td>+13.8</td><td>+12.3</td><td>+4.0</td></tr>
  <tr><td>sentence-t5-large</td><td>+13.6</td><td>+14.4</td><td>+29.9</td><td>+24.5</td><td>+13.6</td><td>+2.1</td><td>+13.6</td><td>+14.4</td><td>+2.1</td><td>+13.6</td><td>+14.4</td><td>+21.3</td><td>+4.0</td><td>+29.9</td></tr>
  <tr><td>sentence-t5-large-ft</td><td>+3.5</td><td>+3.4</td><td>+8.7</td><td>+1.8</td><td>+8.7</td><td>+4.8</td><td>+7.3</td><td>+26.8</td><td>+0.3</td><td>+2.7</td><td>+5.3</td><td>+6.9</td><td>+8.8</td><td>+5.3</td><td>+8.8</td></tr>
  <tr><td>instructor-base</td><td>+5.4</td><td>+5.2</td><td>+5.3</td><td>+2.8</td><td>+0.3</td><td>+5.8</td><td>+5.7</td><td>+2.8</td><td>+0.3</td><td>+5.3</td><td>+5.4</td><td>+0.8</td><td>+0.6</td><td>+0.8</td><td>+4.0</td></tr>
  <tr><td>instructor-large</td><td>+2.4</td><td>+2.4</td><td>+2.5</td><td>+2.4</td><td>+2.1</td><td>+1.6</td><td>+1.8</td><td>+2.6</td><td>-0.5</td><td>-1.2</td><td>+1.3</td><td>+0.1</td><td>-1.0</td><td>+0.8</td><td>+0.8</td></tr>
  <tr><td>SFR-EmbeddingMistral</td><td>-2.4</td><td>-2.6</td><td>-2.2</td><td>-2.3</td><td>-2.2</td><td>-2.2</td><td>-1.3</td><td>-2.8</td><td>-2.8</td><td>-2.2</td><td>-2.8</td><td>-2.2</td><td>-2.8</td><td>-2.3</td></tr>
  <tr><td>NV-Embed-v2</td><td>-1.5</td><td>-2.0</td><td>-2.3</td><td>-2.8</td><td>-1.9</td><td>-0.2</td><td>+0.7</td><td>-1.7</td><td>+1.1</td><td>+2.8</td><td>+2.1</td><td>-3.5</td><td>+0.6</td><td>-0.8</td></tr>
  <tr><td rowspan="10">Pipeline</td><td>BQME</td><td>+2.2</td><td>+2.0</td><td>+3.6</td><td>+6.0</td><td>±1.9</td><td>0.3</td><td>-5.0</td><td>-1.3</td><td>+1.3</td><td>+1.1</td><td>-2.6</td><td>+4.5</td><td>±1.0</td><td>-3.6</td></tr>
  <tr><td>all-distil/roberta-v1</td><td>+0.8</td><td>+0.8</td><td>-0.3</td><td>-0.4</td><td>-1.8</td><td>-1.2</td><td>0.1</td><td>1.0</td><td>-1.1</td><td>-4.7</td><td>+6.1</td><td>+0.8</td><td>+0.2</td><td>+0.9</td></tr>
  <tr><td>all-multit4.1-v2</td><td>-5.2</td><td>-5.4</td><td>-2.1</td><td>-2.1</td><td>-2.3</td><td>-1.8</td><td>-0.8</td><td>+0.4</td><td>0.5</td><td>+2.2</td><td>+1.7</td><td>-1.6</td><td>-0.9</td><td>-7.4</td></tr>
  <tr><td>all-npsmt-base-v2</td><td>-4.7</td><td>-5.4</td><td>-7.0</td><td>-8.3</td><td>-0.3</td><td>-1.2</td><td>+0.1</td><td>0.1</td><td>+4.2</td><td>+2.6</td><td>+1.7</td><td>+1.9</td><td>+4.2</td><td>+2.5</td><td>+1.7</td></tr>
  <tr><td>all-npsmt-base-v2-ft</td><td>-1.2</td><td>-2.0</td><td>-1.2</td><td>-2.1</td><td>+6.0</td><td>+0.3</td><td>+6.0</td><td>0.3</td><td>+6.3</td><td>+1.3</td><td>+6.6</td><td>+4.6</td><td>+4.0</td><td>+0.0</td><td>+5.3</td></tr>
  <tr><td>sentence-t5-base</td><td>+0.7</td><td>+0.2</td><td>+5.9</td><td>+1.2</td><td>+2.0</td><td>+1.3</td><td>+2.0</td><td>+1.3</td><td>+2.4</td><td>+1.9</td><td>0.4</td><td>+2.4</td><td>+2.2</td></tr>
  <tr><td>sentence-t5-large</td><td>-2.1</td><td>-2.0</td><td>-2.3</td><td>-2.0</td><td>-15.8</td><td>0.2</td><td>-2.2</td><td>-1.4</td><td>+1.7</td><td>+1.5</td><td>+0.0</td><td>+1.8</td><td>+4.8</td><td>-0.7</td><td>-4.0</td></tr>
  <tr><td>instructor-base</td><td>+2.8</td><td>+1.2</td><td>+1.</td><td>+1.0</td><td>+1.9</td><td>+6.8</td><td>+5.5</td><td>+2.7</td><td>+3.0</td><td>+2.2</td><td>+1.0</td><td>+1.5</td><td>1.1</td><td>+2.0</td><td>+1.1</td></tr>
  <tr><td>instructor-large</td><td>+1.8</td><td>+2.0</td><td>+2.4</td><td>+1.7</td><td>+2.1</td><td>+1.6</td><td>+6.4</td><td>+9.0</td><td>+6.4</td><td>+8.1</td><td>+1.7</td><td>+2.5</td><td>+1.7</td><td>+1.7</td><td>+1.7</td></tr>
  <tr><td>SFR-EmbeddingMistral</td><td>0.5</td><td>-0.2</td><td>-1.6</td><td>+0.9</td><td>-1.9</td><td>-1.2</td><td>+1.4</td><td>-1.1</td><td>0.0</td><td>-0.1</td><td>-1.1</td><td>0.1</td><td>2.2</td><td>-1.2</td><td>-1.0</td></tr>
  <tr><td>NV-Embed-v2</td><td>+2.4</td><td>+2.5</td><td>+2.4</td><td>+1.6</td><td>+4.6</td><td>+1.3</td><td>+1.3</td><td>+2.4</td><td>+1.2</td><td>+4.6</td><td>+1.5</td><td>+0.7</td><td>2.1</td><td>-1.2</td><td>-1.0</td></tr>
</table>

Table 2: Effect of perturbations on CheckThat22 (top) and FactCheckTweet (bottom). First-Stage Retrieval shows the Δretrieval gap between unperturbed and perturbed inputs sets. Reranking Recovery measures the Δrecovery gap for the perturbed input set before and after reranking top-50 candidates, with improvements highlighted in green and drops in red. Overall Pipeline shows the overall gap for combined retrieval and reranking between unperturbed and perturbed inputs. Colors indicate positive or negative deltas compared to unperturbed. MAP@20 is used for all comparisons. Models finetuned are indicated with the postfix ft.

perturbed sets for a given value of k. Results for CheckThat22 and FactCheckTweet with k = 20 are shown in Table 2. We also provide results for additional values of k, which exhibit similar trends, in Appendix §F. From these results, we make the following observations:

LLM-distilled embeddings exhibit superior robustness. Our analysis shows a smaller Δretrieval gap for SFR-Embedding-Mistral and NV-Embed-V2, both based on Mistral-7B (Jiang et al., 2023). We hypothesize that this robustness stems from the increased representational capacity afforded by the larger scale of the backbone model from which the embeddings are derived, a characteristic shown to enhance robustness to adversarial inputs (Howe et al., 2024). This robustness also extends to dialectical variations and social-media typos, which may be due to the increasing importance of social media data in the training corpora of
recent generative models (Longpre et al., 2024; Miranda et al., 2024; Penedo et al., 2024). In contrast, BERT-based and T5-based models are primarily trained on less diverse, standardised corpora, making them less robust to lexical variations.

NLI-Fine-Tuned embedding models such as Sentence-T5 demonstrate low robustness to negation. Sentence embedding models primarily trained on natural language inference (NLI) datasets, such as T5 (Raffel et al., 2020), exhibit a large \( \Delta_{retrieval} \) gap under negation. This may stem from treating contradiction hypotheses as hard-negatives during training with contrastive loss (Ni et al., 2022), making the models highly sensitive to negation (Chiang et al., 2023). In contrast, models trained on more diverse datasets (e.g., all-distilroberta-v1, all-MiniLM-L12-v2, all-mpnet-base-v2) demonstrate greater robustness to negation and, in some cases, even show improvements. BM25, meanwhile, remains robust to negation, as it is affected minimally by small lexical changes. Finally, double negation rewrites claims in a form closely aligned with fact-checks (e.g., “It is not true . . .”), and may lead to retrieval improvements for some embedding models as observed on FactCheckTweet.

Embedding models with case-sensitive tokenizers struggle with casing variations. Embedding models such as all-distilroberta-v1 and T5-based models with case-sensitive tokenizers, which distinguish between tokens like apple and Apple, show significant retrieval performance gaps on casing-perturbed claims. In contrast, embedding models with case-insensitive tokenizers exhibit robustness to casing differences, showing minimal to no retrieval gaps. While this behavior is expected for case-sensitive models, in retrieval tasks where casing differences are irrelevant, applying casing normalization or using a case-insensitive model can be an effective solution.

4.2 RQ2: Can a Strong Reranker Recover the Retrieval Gap?

In Section §4.1, we observe that perturbations introduce significant retrieval gaps in first-stage retrieval. However, it remains unclear whether a strong reranker can mitigate these gaps and improve ranking quality. To test this, we apply a reranker to the top-\( j \) retrieved candidates and measure the \( \Delta_{recovery} \) gap, which quantifies the MAP@\( k \) improvement on the perturbed set after reranking. We rerank the top \( j = 50 \) candidates, as this balances efficiency and performance, achieving the highest MAP@5 across evaluated values (\( j \in \{5, 10, 20, 50, 100, 200\} \)) (see §D.2). We report this gap for \( k = 20 \) and \( j = 50 \) on CheckThat22 and FactCheckTweet using the reranker bge-reranker-v2-gemma in Table 2. Additional results for different \( k \) values are provided in §F of the Appendix. Across different values of \( k \) and evaluation datasets, we observe that:

Reranking improves weaker embedding models but can negatively impact stronger models in short-target tasks. Across both datasets, we observe that reranking is largely beneficial for embedding models that are less robust in first-stage retrieval. However, in CheckThat22, which contains shorter targets (i.e., short fact-check statements), reranking reduces performance for stronger embedding models, as shown by a drop in MAP@20 scores. This decline may stem from noise i.e., inclusion of additional, potentially irrelevant or low-quality candidates introduced when reranking a larger set of top-50 candidates, which can distort the high-quality rankings produced by stronger models. For stronger embedding models, the most relevant candidates are often already ranked highly and expanding the candidate set can introduce noise. When using stronger embedding models in first-stage retrieval, reranking a smaller candidate subset or skipping reranking altogether might mitigate this issue. For FactCheckTweet, the reranker processes the full fact-check article due to its longer context window, unlike the paragraph-based splitting used during retrieval. This allows the reranker to leverage the full fact-check’s information, which we hypothesize contributes to its more pronounced reranking gains, even for stronger models, compared to CheckThat22.

4.3 RQ3: To what extent do perturbations impact the overall pipeline’s performance?

We observe that a strong reranker is helpful in recovering the retrieval gap discussed in Section §4.2. However, whether it fully compensates for this gap remains to be determined. Here, we examine the end-to-end pipeline (i.e., retrieval plus reranking), by comparing performance on perturbed vs. unperturbed sets. This analysis reveals the \( \Delta_{overall} \) gap, which is the MAP@\( k \) difference between the unperturbed and perturbed sets for the full pipeline
of retrieval and reranking the top- \( j \) candidates. We show the results in Table 2 for \( k = 20, j = 50 \), and provide additional results for other \( k \) values in §F of the Appendix. We observe that:

A strong reranker helps, but cannot fully compensate for first-stage retrieval gaps. A strong reranker proves effective in reducing the retrieval gap particularly for less robust embedding models. However, a notable end-to-end performance gap persists, indicating that a strong reranker alone is not sufficient. Although we select a SoTA reranker, we hypothesize that it may also exhibit brittleness to some of the perturbations, which could reduce its effectiveness in reranking and contribute to the observed performance drop. For instance, under the negation perturbation on CheckThat22, the reranker negatively impacts the overall performance of embedding models that initially performed well in first-stage rankings. Additionally, weaker embedding models display significant deficiencies in handling typos and dialect perturbations. Similar limitations are observed with case-sensitive embedding models, where reranking fails to fully recover from retrieval gaps caused by casing perturbations. For entity replacements, reranking helps reduce the retrieval gap on CheckThat22, but offers little benefit on FactCheckTweet. We hypothesize that entity swaps have a smaller impact when matching against shorter targets, as in CheckThat22, compared to longer targets, where the effects of entity changes are more severe.

LLM Rewrites improves overall performance as it acts a form of query rewriting or naïve query expansion, a technique shown to improve performance in retrieval tasks (Nogueira et al., 2019b; Kazemi et al., 2023; Weller et al., 2024).

4.4 RQ4: How effective are mitigation approaches at improving robustness for weaker embedding models, and what trade-offs do they entail?

LLM-distilled embeddings are more robust to misinformation edits (Section §4.1), but they come with the trade-off of increased storage requirements and slower inference, making them less ideal for first-stage retrieval. In contrast, smaller models are computationally efficient but exhibit weaker robustness. In this section, we explore whether train-time and inference-time approaches can improve the robustness of weaker embedding models on typos, negation, dialect changes, and entity replacement, perturbations that remain challenging even for a strong reranker. We test the approaches on the CheckThat22 dataset, which we refer to as the in-domain dataset, as we use its training and development sets. We also evaluate how well these improvements generalize to an out-of-domain (OOD) dataset consisting of novel claims not seen during training on CheckThat22. All mitigations are applied only to the first-stage retrieval, and we report retrieval performance for this stage.6

For train-time interventions, we examine fine-tuning on the CheckThat22 training set and a knowledge distillation (KD) approach inspired by Reimers and Gurevych (2020), which aims to reduce representation differences between unperturbed and perturbed input claims. For KD, we generate parallel sentence pairs using our perturbation generation framework (§3.2.1), applied to the CheckThat22 training set, yielding 11,593 unperturbed–perturbed claim pairs. We further expand this set by pairing different perturbations of the same claim (e.g., typo and dialect), resulting in 70,954 claim pairs. Details of the training setup are provided in §C.1.2. As an inference-time intervention, we investigate a claim normalization approach proposed by Sundriyal et al. (2023), where input claims are rewritten into a standard form using GPT4o prior to retrieval.7 Finally, we evaluate the effect of combining multiple approaches. We select all-mpnnet-base-v2 as a representative weaker—yet computationally efficient—embedding model, as it is widely used for retrieval

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Typos</th>
    <th colspan="2">Neg.</th>
    <th colspan="2">Entity R.</th>
    <th colspan="2">Dialect</th>
  </tr>
  <tr>
    <th>U</th><th>P</th><th>U</th><th>P</th><th>U</th><th>P</th><th>U</th><th>P</th>
  </tr>
  <tr>
    <td>mpnet (baseline)</td><td>0.76</td><td>0.68</td><td>0.78</td><td>0.74</td><td>0.76</td><td>0.66</td><td>0.74</td><td>0.65</td>
  </tr>
  <tr>
    <td>mpnet-robust</td><td>0.83</td><td>0.78</td><td>0.83</td><td>0.83</td><td>0.83</td><td>0.77</td><td>0.80</td><td>0.80</td>
  </tr>
  <tr>
    <td>mpnet-ft</td><td>0.90</td><td>0.77</td><td>0.90</td><td>0.85</td><td>0.89</td><td>0.80</td><td>0.88</td><td>0.80</td>
  </tr>
  <tr>
    <td>mpnet-robust-ft</td><td>0.89</td><td>0.78</td><td>0.88</td><td>0.84</td><td>0.90</td><td>0.81</td><td>0.87</td><td>0.84</td>
  </tr>
  <tr>
    <td>mpnet-CN</td><td>0.77</td><td>0.78</td><td>0.80</td><td>0.79</td><td>0.76</td><td>0.75</td><td>0.74</td><td>0.73</td>
  </tr>
  <tr>
    <td>mpnet-robust-CN</td><td>0.83</td><td>0.83</td><td>0.89</td><td>0.82</td><td>0.84</td><td>0.84</td><td>0.82</td><td>0.82</td>
  </tr>
  <tr>
    <td>mpnet-ft+CN</td><td>0.89</td><td>0.89</td><td>0.85</td><td>0.87</td><td>0.89</td><td>0.87</td><td>0.88</td><td>0.86</td>
  </tr>
  <tr>
    <td>mpnet-robust-ft+CN</td><td>0.90</td><td>0.88</td><td>0.90</td><td>0.85</td><td>0.89</td><td>0.89</td><td>0.89</td><td>0.87</td>
  </tr>
  <tr>
    <td>NV-Embed-v2</td><td>0.98</td><td>0.98</td><td>0.97</td><td>0.94</td><td>0.97</td><td>0.97</td><td>0.97</td><td>0.96</td>
  </tr>
  <tr>
    <td>NV-Embed-v2 + CN</td><td>0.98</td><td>0.97</td><td>0.97</td><td>0.93</td><td>0.97</td><td>0.97</td><td>0.97</td><td>0.97</td>
  </tr>
</table>

Table 3: Effect of mitigation approaches on typos, negation, entity replacement and dialect (Piggin) on CheckThat22 (In-Domain). mpnet refers to all-mpnnet-base-v2, with suffixes denoting mitigation approaches: robust (KD approach), ft (task finetuning), and +CN (claim normalization). Columns marked U and P indicate MAP@20 performance on the unperturbed and perturbed sets, respectively.

6We assume that improving the first-stage retrieval will subsequently improve reranking performance.
7The prompt used is shown in §I.
<table>
  <tr>
    <th>Model</th>
    <th>MAP@1</th>
    <th>MAP@5</th>
    <th>MAP@20</th>
    <th>MAP@50</th>
  </tr>
  <tr>
    <td>mpnet (baseline)</td>
    <td>0.4327</td>
    <td>0.5129</td>
    <td>0.5254</td>
    <td>0.5269</td>
  </tr>
  <tr>
    <td>mpnet-robust</td>
    <td>0.4508</td>
    <td>0.5259</td>
    <td>0.5390</td>
    <td>0.5411</td>
  </tr>
  <tr>
    <td>mpnet-ft</td>
    <td>0.5076</td>
    <td>0.5853</td>
    <td>0.5982</td>
    <td>0.5998</td>
  </tr>
  <tr>
    <td>mpnet-robust-ft</td>
    <td>0.5090</td>
    <td>0.5851</td>
    <td>0.5991</td>
    <td>0.6006</td>
  </tr>
  <tr>
    <td>mpnet+CN</td>
    <td>0.4938</td>
    <td>0.5696</td>
    <td>0.5799</td>
    <td>0.5816</td>
  </tr>
  <tr>
    <td>mpnet-robust+CN</td>
    <td>0.5076</td>
    <td>0.5765</td>
    <td>0.5888</td>
    <td>0.5911</td>
  </tr>
  <tr>
    <td>mpnet-ft+CN</td>
    <td><b>0.5368†</b></td>
    <td><b>0.6109†</b></td>
    <td><b>0.6213†</b></td>
    <td><b>0.6233†</b></td>
  </tr>
  <tr>
    <td>mpnet-robust-ft+CN</td>
    <td>0.5257</td>
    <td>0.6046</td>
    <td>0.6163</td>
    <td>0.6185</td>
  </tr>
  <tr>
    <td>NV-Embed-v2</td>
    <td>0.5520</td>
    <td>0.6306</td>
    <td>0.6402</td>
    <td>0.6418</td>
  </tr>
  <tr>
    <td>NV-Embed-v2+CN</td>
    <td><b>0.5576*</b></td>
    <td><b>0.6375*</b></td>
    <td><b>0.6464*</b></td>
    <td><b>0.6479*</b></td>
  </tr>
</table>

Table 4: Effect of Mitigation Approaches on an Out-of-Domain (OOD) Dataset. mpnet denotes all-mpnet-base-v2, with -ft for task finetuning, -robust for the KD approach, and +CN indicating claim normalization. † indicates the model with the best improvement over the baseline, while * indicates the model with the best overall performance.

tasks.\footnote{all-mpnet-base-v2 had 33.4M downloads in the past month at the time of this study.} Results are shown in Table 3 and Table 4 for the test split of CheckThat22 and the OOD Dataset, respectively. We observe that:

Finetuning, KD and claim normalization all improve robustness for weaker embedding models and can be applied independently or jointly. KD and finetuning are train-time interventions that update model weights to better handle differences between unperturbed and perturbed input claims. KD relies on synthetically generated perturbed input claims that are scalable and inexpensive to produce (Shislberg et al., 2024; Liu et al., 2024). In contrast, finetuning requires claim-factcheck pairs, which are costly to annotate. On the other hand, claim normalization is an inference-time intervention that leverages the innate representational capabilities of weaker embedding models by standardizing perturbed input claims into a form that is easier to represent, without having to update model weights. Although claim normalization is appealing for its “train-free,” inference-time application, its reliance on a strong normalizer (often an LLM) may add a computational overhead to the overall retrieval pipeline, making robust adaptation through knowledge distillation a scalable alternative for scenarios where train-time interventions are feasible.

We expect a significant drop in retrieval performance across all approaches for novel claims in the OOD dataset as they are (a) taken from a different time period, (b) cover different topics, and (c) come from a different platform, with different characteristics than the tweets in CheckThat22 and FactCheckTweet (e.g., longer length). We observe such a drop. Nonetheless, all mitigation approaches improve performance relative to the baseline and combining them yields the largest improvement of 10 percentage points similar to the in-domain setting. This highlights the effectiveness of the mitigation approaches in generalizing to novel out-of-domain claims.

5 Discussion and Conclusion

Real-world claim matching systems must handle not only factual but also edited claims, which often arise as users engage with misinformation online. Embedding models used in these systems must therefore be robust to such user-informed misinformation edits. However, standard benchmarks like MTEB (Muennighoff et al., 2022; Enevoldsen et al., 2025), primarily evaluate sentence embedding models on clean, generalist tasks and fail to capture this challenge. To address this gap, we introduce a perturbation generation framework that produces natural and valid claim edits. This framework creates a more realistic evaluation test bed for sentence embedding models. Although developed for claim matching, these perturbations have broader relevance and can help select embedding models for other downstream tasks that involve user-generated inputs. Our evaluation shows that widely-used embedding models are highly sensitive to user edits, while LLM-distilled embeddings offer greater robustness. Reranking helps, but even strong rerankers cannot fully recover performance under perturbation. To address the retrieval gaps, we show that train- and inference-time approaches can improve embedding model robustness and generalize to out-of-domain settings with different trade-offs. Knowledge distillation and finetuning are train-time interventions that require generating synthetic data or collecting costly human annotated data with additional computational costs for training and re-embedding fact-checks. On the other hand, claim normalization is a “train-free” inference-time mitigation that depends on an LLM and may introduce significant latency and cost to the overall pipeline. Encouragingly, prior work (Ma et al., 2023) shows that rewriting capabilities can be distilled into smaller models, offering a practical compromise. Overall, our study offers practical insights for improving claim matching systems, ultimately empowering human fact-checkers to retrieve relevant fact-checks with greater reliability.
Limitations

While our study makes meaningful strides in evaluating the robustness of embedding models to real-world user edits and exploring mitigation strategies, we acknowledge some limitations:

Dataset Coverage  Our evaluation relies on claim matching datasets from fact-checking organizations. Although these datasets provide a useful starting point, they represent only a fraction of the broader misinformation landscape, as many claims in the wild go unchecked. As such, the taxonomy of common edits proposed in this work is representative but not exhaustive, as it is limited to claims encountered by fact-checkers. This limitation is evident in our out-of-domain evaluations, where models experience notable performance degradation when faced with novel claims spanning different timelines, topics, and platforms. Nonetheless, our proposed interventions generalize well to the out-of-domain setting, improving the performance of baseline models by an average of 10 percentage points.

Language Scope  Our analysis is limited to English-language claims, even though misinformation is a multilingual phenomenon (Kazemi et al., 2021; Quelle et al., 2025). We believe that our approach could be extended to other languages, and pursuing this extension is an important direction for future work.

Experimental Coverage  Although our experiments cover a diverse range of settings, evaluating 12 embedding models, 7 rerankers, and 14 perturbation types across various MAP@k values (\( k \in \{5, 10, 20, 50\} \)) and different top-j reranking candidates (\( j \in \{5, 10, 20, 50, 100, 200\} \))—this selection is not exhaustive. Alternative configurations may yield different robustness outcomes. We evaluate widely used embedding models that represent various pre-training approaches, backbone architectures and embedding dimension sizes. However, we note that other types of embedding models, such as unsupervised embeddings (Gao et al., 2021) or contextual document embeddings (Morris and Rush, 2025), might behave differently. We also do not evaluate closed-source embedding models (e.g., text-embedding-ada-002 from OpenAI9 or Embed v3 from Cohere10) due to the inference costs involved. While we test different rerankers and select the best-performing one, we acknowledge that the reranker we select could also exhibit some brittleness to the perturbations, and we believe that evaluating rerankers against the same perturbation set would be a valuable direction for future work.

Ethical Considerations

We acknowledge the ethical responsibilities inherent in misinformation research given its real-world impact. Our perturbation generation framework relies on widely available LLMs, and our prompts are publicly released for reproducibility; however, this introduces the risk of misuse, whether inadvertently by non-experts or deliberately by malicious actors to generate misinformation or create rewritings that real-world claim matching systems struggle to identify. For a more detailed discussion on the ethical risks associated with LLM-generated misinformation, see Chen and Shu (2024). We also note that the synthetically perturbed claims generated by our work may contain fictional or inaccurate information; as such, these claims should not be used to train models that learn facts from data. Notably, during our claim normalization experiments, GPT-4o refused to process two claims containing sensitive political or religious statements. This incident highlights the critical need for automated systems to differentiate between the use of misinformation (deliberately propagating false information) and the mention of misinformation (discussing or referencing sensitive topics without endorsing them) (Gligoric et al., 2024).

Acknowledgments

We thank Meedan for providing the datasets used in this project. JM is supported by the Rhodes Scholarship. SH is supported by a grant from The Alan Turing Institute for the project entitled “Effective discovery, tracking, and response to mis- and disinformation” and the ESRC Digital Good Network (grant reference ES/X502352/1). We are grateful to Samuel Mensah and Elena Kochkina for their helpful feedback on drafts of this paper.

References

Timothy Baldwin, Marie-Catherine De Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition. In Proceedings
of the workshop on noisy user-generated text, pages 126–135.

Alberto Barrón-Cedeno, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, Fatima Haouari, Nikolay Babulkov, Bayan Hamdan, Alex Nikolov, et al. 2020. Overview of checkthat! 2020: Automatic identification and verification of claims in social media. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22–25, 2020, Proceedings 11, pages 215–236. Springer.

Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada. Association for Computational Linguistics.

Canyu Chen and Kai Shu. 2024. Can LLM-generated misinformation be detected? In The Twelfth International Conference on Learning Representations.

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216.

Cheng-Han Chiang and Hung-yi Lee. 2023. Are synonym substitution attacks really synonym substitution attacks? In Findings of the Association for Computational Linguistics: ACL 2023, pages 1853–1878, Toronto, Canada. Association for Computational Linguistics.

Cheng-Han Chiang, Hung-yi Lee, Yung-Sung Chuang, and James Glass. 2023. Revealing the blind spot of sentence encoder evaluation by HEROS. In Proceedings of the 8th Workshop on Representation Learning for NLP (Repl4NLP 2023), pages 289–302, Toronto, Canada. Association for Computational Linguistics.

Anubrata Das, Houjiang Liu, Venelin Kovatchev, and Matthew Lease. 2023. The state of human-centered nlp technology for fact-checking. Information processing & management, 60(2):103219.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Salijona Dyrmishi, Salah Ghamizi, and Maxime Cordy. 2023a. How do humans perceive adversarial text? a reality check on the validity and naturalness of word-based adversarial attacks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8822–8836, Toronto, Canada. Association for Computational Linguistics.

Salijona Dyrmishi, Salah Ghamizi, and Maxime Cordy. 2023b. How do humans perceive adversarial text? A reality check on the validity and naturalness of word-based adversarial attacks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8822–8836, Toronto, Canada. Association for Computational Linguistics.

Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzeminski, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Diganta Misra, Shreeya Dhakal, Jonathan Rystrom, Roman Solomatin, Ömer Veysel Çagatan, Akash Kundu, Martin Bernstorff, Shitao Xiao, Akshita Sukhlecha, Bhavish Pahwa, Rafal Poświata, Kranthi Kiran GV, Shahwon Ashraf, Daniel Auras, Björn Plüster, Jan Philipp Harries, Loïc Magne, Isabelle Mohr, Dawei Zhu, Hippolyte Gisserot-Boukhlef, Tom Aarsen, Jan Kostkan, Konrad Wojtasik, Taemin Lee, Marek Suppa, Cristina Zhang, Roberta Rocca, Mohammed Hamdy, Andrianos Michail, John Yang, Manuel Faysse, Aleksei Vatolin, Nandan Thakur, Manan Dey, Dipam Vasani, Pranjal A Chitale, Simone Tedeschi, Nguyen Tai, Artem Snegirev, Mariya Hendriksen, Michael Günther, Mengzhou Xia, Weijia Shi, Xing Han Lu, Jordan Clive, Gayatri K. Maksimova Anna, Silvan Wehrli, Maria Tikhonova, He-nil Shalin Panchal, Aleksandr Abramov, Malte Osten-dorff, Zheng Liu, Simon Clematide, Lester James Validad Miranda, Alena Fenogenova, Guangyu Song, Ruqiya Bin Safi, Wen-Ding Li, Alessia Borghini, Federico Cassano, Lasse Hansen, Sara Hooker, Cheng-hao Xiao, Vaibhav Adlakha, Orion Weller, Siva Reddy, and Niklas Muennighoff. 2025. MMTEB: Massive multilingual text embedding benchmark. In The Thirteenth International Conference on Learning Representations.

Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting lms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491–6501.

Lorenzo Jaime Yu Flores and Yiding Hao. 2022. An adversarial benchmark for fake news detection models. arXiv preprint arXiv:2201.00912.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Siddhant Garg and Goutham Ramakrishnan. 2020. BAE: BERT-based adversarial examples for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6174–6181, Online. Association for Computational Linguistics.

Vahid Ghafouri, Jose Such, and Guillermo Suarez-Tangil. 2024. I love pineapple on pizza != I hate pineapple on pizza: Stance-aware sentence transformers for opinion mining. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21046–21058, Miami, Florida, USA. Association for Computational Linguistics.

Kristina Gligoric, Myra Cheng, Lucia Zheng, Esin Durmus, and Dan Jurafsky. 2024. NLP systems that can’t tell use from mention censor counterspeech, but teaching the distinction helps. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5942–5959, Mexico City, Mexico. Association for Computational Linguistics.

Max Glockner, Yufang Hou, and Iryna Gurevych. 2022. Missing counter-evidence renders NLP fact-checking unrealistic for misinformation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5916–5936, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.

Shreya Goyal, Sumanth Doddapaneni, Mitesh M Khapra, and Balaraman Ravindran. 2023. A survey of adversarial defenses and robustness in nlp. ACM Computing Surveys, 55(14s):1–39.

Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178–206.

Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022. Bitext mining using distilled sentence representations for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2101–2112, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Christopher Hidey, Tuhin Chakrabarty, Tariq Alhindi, Siddharth Varia, Kriste Krstovski, Mona Diab, and Smaranda Muresan. 2020. DeSePtion: Dual sequence prediction and adversarial examples for improved fact-checking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8593–8606, Online. Association for Computational Linguistics.

Nikolaus Howe, Ian McKenzie, Oskar Hollinsworth, Michal Zajac, Tom Tseng, Aaron Tucker, Pierre-Luc Bacon, and Adam Gleave. 2024. Effects of scale on language model robustness. Preprint, arXiv:2407.18213.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019. Is bert really robust? natural language attack on text classification and entailment. arXiv preprint arXiv:1907.11932.

David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating dialectal variability for socially equitable language identification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 51–57, Vancouver, Canada. Association for Computational Linguistics.

A Kazemi, Z Li, V Peréz-Rosas, SA Hale, and R Mihalcea. 2022. Matching tweets with applicable fact-checks across languages. In CEUR Workshop Proceedings.

Ashkan Kazemi, Artem Abzaliev, Naihao Deng, Rui Hou, Scott Hale, Veronica Perez-Rosas, and Rada Mihalcea. 2023. Query rewriting for effective misinformation discovery. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 398–407, Nusa Dua, Bali. Association for Computational Linguistics.

Ashkan Kazemi, Kiran Garimella, Devin Gaffney, and Scott Hale. 2021. Claim matching beyond English to scale global fact-checking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4504–4517, Online. Association for Computational Linguistics.

Thai Le, Suhang Wang, and Dongwon Lee. 2020. Malcom: Generating malicious comments to attack neural fake news detection models. In 2020 IEEE International Conference on Data Mining (ICDM), pages 282–291. IEEE.

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. Nv-embed: Improved techniques for training llms as generalist embedding models. Preprint, arXiv:2405.17428.
Jinyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang, and Iftekhar Naim. 2024. Gecko: Versatile text embeddings distilled from large language models. Preprint, arXiv:2403.20327.

Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. BERT-ATTACK: Adversarial attack against BERT using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193–6202, Online. Association for Computational Linguistics.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Lucian Vlad Lita, Abe Ittycheriah, Salim Roukos, and Nanda Kambhatla. 2003. Truecasing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 152–159.

Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinneng Rao, Steven Zheng, Dapeng Peng, Diyi Yang, Denny Zhou, et al. 2024. Best practices and lessons learned on synthetic data for language models. arXiv preprint arXiv:2404.07503.

Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, William Brannon, Robert Mahari, Manan Dey, Mohammed Hamdy, Nayan Saxena, et al. 2024. Bridging the data provenance gap across text, speech and video. arXiv preprint arXiv:2412.17847.

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5303–5315, Singapore. Association for Computational Linguistics.

Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2421–2425.

Emanuele La Malfa and Marta Kwiatkowska. 2022. The King Is Naked: On the Notion of Robustness for Natural Language Processing. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11047–11057. Section: AAAI Technical Track on Speech and Natural Language Processing.

Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. 2024. Sf-embedding-mistral:enhance text retrieval with transfer learning. Salesforce AI Research Blog.

Brando Miranda, Alycia Lee, Sudharsan Sundar, Allison Casasola, and Sanmi Koyejo. 2024. Beyond scale: The diversity coefficient as a data quality metric for variability in natural language data. Preprint, arXiv:2306.13840.

John Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020a. Reevaluating adversarial examples in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3829–3839, Online. Association for Computational Linguistics.

John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020b. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.05909.

John Xavier Morris and Alexander M Rush. 2025. Contextual document embeddings. In The Thirteenth International Conference on Learning Representations.

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022. Meteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316.

Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeño, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021a. Automated Fact-Checking for Assisting Human Fact-Checkers. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4551–4558. International Joint Conferences on Artificial Intelligence Organization.

Preslav Nakov, Giovanni Da San Martino, Firoj Alam, Shaden Shaar, Hamdy Mubarak, and Nikolay Babulkov. 2022. Overview of the CLEF-2022 CheckThat! lab task 2 on detecting previously fact-checked claims. In Working Notes of CLEF 2022—Conference and Labs of the Evaluation Forum, CLEF ’22, Bologna, Italy.

Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeno, Rubén Míguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Nikolay Babulkov, et al. 2021b. The clef-2021 checkthat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news. In Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28–April 1, 2021, Proceedings, Part II 43, pages 639–649. Springer.

Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yifei Yang. 2022. Sentence-t5: Scalable sentence encoders from pretrained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864–1874, Dublin, Ireland. Association for Computational Linguistics.
Lydia Nishimwe, Benoît Sagot, and Rachel Bawden. 2024. Making sentence embeddings robust to user-generated content. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10984–10998, Torino, Italia. ELRA and ICCL.

Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with bert. corr abs/1901.04085 (2019). URL: http://arxiv.org/abs/1901.04085.

Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre-trained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718, Online. Association for Computational Linguistics.

Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019a. Multi-stage document ranking with bert. arXiv preprint arXiv:1910.14424.

Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019b. Document expansion by query prediction. arXiv preprint arXiv:1904.08375.

Guilherme Penedo, Hynek Kydlíček, Loubna Ben al-lal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. RankZephyr: Effective and robust zero-shot listwise reranking is a breeze! arXiv:2312.02724.

Rob Procter, Miguel Arana-Catania, Yulan He, Maria Liakata, Arkaitz Zubiaga, Elena Kochkina, and Runcong Zhao. 2023. Some observations on fact-checking work with implications for computational support. arXiv preprint arXiv:2305.02224.

Piotr Przybyla, Alexander Shvets, and Horacio Saggion. 2024. Verifying the robustness of automatic credibility assessment. Natural Language Processing, page 1–29.

Dorian Quelle, Calvin Yixiang Cheng, Alexandre Bovet, and Scott A. Hale. 2025. Lost in translation: Using global fact-checks to measure multilingual misinformation prevalence, spread, and evolution. EPJ Data Science, 14(1).

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.

Javad Rafiei Asl, Prajwal Panzade, Eduardo Blanco, Daniel Takabi, and Zhipeng Cai. 2024. RobustSen-tEmbed: Robust sentence embeddings using adversarial self-supervised contrastive learning. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3795–3809, Mexico City, Mexico. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics.

Alan Ritter, Sam Clark, Oren Etzioni, et al. 2011. Named entity recognition in tweets: an experimental study. In Proceedings of the 2011 conference on empirical methods in natural language processing, pages 1524–1534.

Manuela Sanguinetti, Bosco Cristina, Cassidy Lauren, Cetinoglu Ozlem, Cignarella Alessandra Teresa, Lynn Teresa, Rehbein Ines, Ruppenhofer Joseph, Seddah Djamel, Zeldes Amir, et al. 2020. Treebanking user-generated content: A proposal for a unified representation in universal dependencies. In Proceedings of the 12th language resources and evaluation conference, pages 5240–5250. ELRA, Language Resources Association.

Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino, and Preslav Nakov. 2020. That is a known lie: Detecting previously fact-checked claims. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3607–3618, Online. Association for Computational Linguistics.

Shaden Shaar, Nikola Georgiev, Firoj Alam, Giovanni Da San Martino, Aisha Mohamed, and Preslav Nakov. 2022. Assisting the human fact-checkers: Detecting all previously fact-checked claims in a document. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2069–2080, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Qiang Sheng, Juan Cao, Xueyao Zhang, Xirong Li, and Lei Zhong. 2021. Article reranking by memory-enhanced key sentence matching for detecting previously fact-checked claims. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5468–5481, Online. Association for Computational Linguistics.
Michael Shliselberg and Shiri Dori-Hacohen. 2022. Riet lab at checkthat!-2022: Improving decoder based re-ranking for claim matching. In CLEF (Working Notes), pages 671–678.

Michael Shliselberg, Ashkan Kazemi, Scott A Hale, and Shiri Dori-Hacohen. 2024. SynDy: Synthetic dynamic dataset generation framework for misinformation tasks. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2801–2805.

Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023. One embedder, any task: Instruction-finetuned text embeddings. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1102–1121, Toronto, Canada. Association for Computational Linguistics.

Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14918–14937, Singapore. Association for Computational Linguistics.

Megha Sundriyal, Tanmoy Chakraborty, and Preslav Nakov. 2023. From chaos to clarity: Claim normalization to empower fact-checking. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6594–6609, Singapore. Association for Computational Linguistics.

Manveer Singh Tamber, Ronak Pradeep, and Jimmy Lin. 2023. Scaling down, lifting up: Efficient zero-shot listwise reranking with seq2seq encoder-decoder models. arXiv preprint arXiv: 2312.16098.

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Evaluating adversarial attacks against multiple fact verification systems. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2944–2953, Hong Kong, China. Association for Computational Linguistics.

Manuel Tonneau, Diyi Liu, Samuel Fraiberger, Ralph Schroeder, Scott Hale, and Paul Röttger. 2024a. From languages to geographies: Towards evaluating cultural bias in hate speech datasets. In Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024), pages 283–311, Mexico City, Mexico. Association for Computational Linguistics.

Manuel Tonneau, Pedro Quinta De Castro, Karim Lasri, Ibrahim Farouq, Lakshmi Subramanian, Victor Orozco-Olvera, and Samuel Fraiberger. 2024b. NaijaHate: Evaluating hate speech detection on Nigerian Twitter using representative data. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9020–9040, Bangkok, Thailand. Association for Computational Linguistics.

Rob Van Der Goot, Rik van Noord, and Gertjan van Noord. 2018. A taxonomy for in-depth evaluation of normalization for user generated content. In Eleventh International Conference on Language Resources and Evaluation, pages 684–688. European Language Resources Association (ELRA).

Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. science, 359(6380):1146–1151.

Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022. Measure and improve robustness in NLP models: A survey. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4569–4586, Seattle, United States. Association for Computational Linguistics.

WEF. 2025. Global risks report 2025: Conflict, environment and disinformation top threats. Accessed: 2025-02-07.

Orion Weller, Kyle Lo, David Wadden, Dawn Lawrie, Benjamin Van Durme, Arman Cohan, and Luca Soldaini. 2024. When do generative query and document expansions fail? a comprehensive study across methods, retrievers, and datasets. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1987–2003, St. Julian’s, Malta. Association for Computational Linguistics.

Muheng Yan, Yu-Ru Lin, and Wen-Ting Chung. 2022. Are mutated misinformation more contagious? A case study of Covid-19 misinformation on Twitter. In Proceedings of the 14th ACM Web Science Conference 2022, pages 336–347.

Zhixuan Zhou, Huankang Guan, Meghana Moorthy Bhat, and Justin Hsu. 2019. Fake news detection via nlp is vulnerable to adversarial attacks. arXiv preprint arXiv:1901.09657.

Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-value: A framework for cross-dialectal english nlp. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada. Association for Computational Linguistics.
A  Dataset Statistics

We report the details of each dataset used in our experiments in this section. The CheckThat22 dataset (Nakov et al., 2021b) is available under a permissible research-use license.11 Similarly, the FactCheckTweet dataset (Kazemi et al., 2022) can be accessed for research purposes.12 The OOD Dataset is provided by Meedan, primarily sourced through five fact-checking organizations operating tiplines on WhatsApp, and our use is consistent with its intended use.

A.1  Dataset Splits

For all experiments, we use only the English subsets of the datasets. In the robustness experiments, we apply perturbations to the test set only. CheckThat22 provides a predefined train/dev/test split, whereas FactCheckTweet does not. We therefore partition the latter into 80% training, 10% development, and 10% test splits. The train and development splits from CheckThat22 are used for finetuning the embedding models or improving robustness through the teacher-student knowledge distillation approach. For the OOD Dataset, we treat the full split as an evaluation set and do not partition it. Table 5 shows the splits for each dataset.

<table>
  <tr>
    <th>Dataset</th>
    <th>Train</th>
    <th>Dev</th>
    <th>Test</th>
  </tr>
  <tr>
    <td><i>CheckThat22</i></td>
    <td>1,195</td>
    <td>1,195</td>
    <td>202</td>
  </tr>
  <tr>
    <td><i>FactCheckTweet</i></td>
    <td>2,504</td>
    <td>537</td>
    <td>537</td>
  </tr>
  <tr>
    <td><i>OOD Dataset</i></td>
    <td>-</td>
    <td>-</td>
    <td>721</td>
  </tr>
</table>

Table 5: Dataset splits for training, development, and testing.

A.2  Perturbed Claims Statistics

We report the number of valid perturbed input claims for each perturbation type in Table 6. All perturbations are applied using the framework detailed in Section 3.2.1, except for Casing, which we apply using the TrueCase Python library.13 While perturbations are initially applied to the full test sets, the number of valid claims varies due to differences in the selection criteria introduced by the LLM as a verifier. When comparing against unperturbed claims, we use only the same subset and not the entire original test set. Overall, perturbations applied at the maximum noise level (\( b_{\text{high}} \)) result in more invalid perturbations, which are filtered out in our generation pipeline.

<table>
  <tr>
    <th>Perturbation</th>
    <th>Type</th>
    <th>CheckThat22</th>
    <th>FactCheckTweet</th>
  </tr>
  <tr>
    <td rowspan="2">Casing</td>
    <td><i>TrueCase</i></td>
    <td>202</td>
    <td>537</td>
  </tr>
  <tr>
    <td><i>UpperCase</i></td>
    <td>202</td>
    <td>537</td>
  </tr>
  <tr>
    <td rowspan="2">Typos</td>
    <td><i>Least</i></td>
    <td>192</td>
    <td>443</td>
  </tr>
  <tr>
    <td><i>Most</i></td>
    <td>189</td>
    <td>441</td>
  </tr>
  <tr>
    <td rowspan="2">Negation</td>
    <td><i>Shallow</i></td>
    <td>181</td>
    <td>405</td>
  </tr>
  <tr>
    <td><i>Double</i></td>
    <td>148</td>
    <td>308</td>
  </tr>
  <tr>
    <td rowspan="2">Entity Rep.</td>
    <td><i>At least 1</i></td>
    <td>181</td>
    <td>327</td>
  </tr>
  <tr>
    <td><i>All</i></td>
    <td>110</td>
    <td>164</td>
  </tr>
  <tr>
    <td rowspan="2">LLM Rewrite</td>
    <td><i>Least</i></td>
    <td>185</td>
    <td>373</td>
  </tr>
  <tr>
    <td><i>Most</i></td>
    <td>185</td>
    <td>371</td>
  </tr>
  <tr>
    <td rowspan="3">Dialect</td>
    <td><i>AAE</i></td>
    <td>195</td>
    <td>449</td>
  </tr>
  <tr>
    <td><i>Jamaican</i></td>
    <td>192</td>
    <td>443</td>
  </tr>
  <tr>
    <td><i>Pidgin</i></td>
    <td>192</td>
    <td>449</td>
  </tr>
  <tr>
    <td><i>Singlish</i></td>
    <td>195</td>
    <td>449</td>
  </tr>
</table>

Table 6: Valid perturbed input claims for each perturbation type across CheckThat22 and FactCheckTweet datasets.

We report the degree of lexical overlap between the unperturbed and perturbed input claims for the CheckThat22 and FactCheckTweet datasets in Table 7 and Table 8, respectively. Following the approach of Chiang et al. (2023), we compute the ROUGE F1 scores (R1, R2, RL) (Lin, 2004), where R1 measures unigrams, R2 measures bigrams, and RL measures the longest common subsequence. We use the rouge14 package for this purpose. A higher ROUGE score indicates greater lexical similarity. We also report the normalized Levenshtein distance between the unperturbed and perturbed input pairs, obtained by calculating the Levenshtein distance between the pair and dividing by the length of the longer sequence. We tokenize the sentences using bert-base-uncased and calculate the distance between the token IDs. A lower Levenshtein distance implies higher lexical overlap. Across all perturbation types, we observe that the worst-case edits—those applied with the maximum noise level (\( b_{\text{high}} \))—result in perturbed claims that have less lexical overlap with their unperturbed counterparts than do the edits applied with the minimum noise level (\( b_{\text{low}} \)). This difference in lexical overlap enables us to capture a wide spectrum of user edits.

11https://gitlab.com/checkthat_lab/clef2022-checkthat-lab/clef2022-checkthat-lab.git
12https://lit.eecs.umich.edu/publications.html
13https://github.com/daltonfury42/truecase
14https://pypi.org/project/rouge/
<table>
  <tr>
    <th>Perturbation</th>
    <th>Type</th>
    <th>R1</th>
    <th>R2</th>
    <th>RL</th>
    <th>Lev.</th>
  </tr>
  <tr>
    <td rowspan="2">Casing</td>
    <td><i>TrueCase</i></td>
    <td>90.20</td>
    <td>83.02</td>
    <td>90.20</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td><i>UpperCase</i></td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td rowspan="2">Typos</td>
    <td><i>Least</i></td>
    <td>60.47</td>
    <td>31.46</td>
    <td>60.47</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td><i>Most</i></td>
    <td>41.67</td>
    <td>10.81</td>
    <td>38.89</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td rowspan="2">Negation</td>
    <td><i>Shallow</i></td>
    <td>56.34</td>
    <td>46.58</td>
    <td>53.52</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td><i>Double</i></td>
    <td>17.78</td>
    <td>12.50</td>
    <td>17.78</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td rowspan="2">Entity Rep.</td>
    <td><i>At least 1</i></td>
    <td>98.00</td>
    <td>94.23</td>
    <td>98.00</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td><i>All</i></td>
    <td>87.50</td>
    <td>74.75</td>
    <td>87.50</td>
    <td>0.14</td>
  </tr>
  <tr>
    <td rowspan="2">LLM Rewrite</td>
    <td><i>Least</i></td>
    <td>58.14</td>
    <td>30.77</td>
    <td>53.49</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td><i>Most</i></td>
    <td>41.46</td>
    <td>16.47</td>
    <td>41.46</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td rowspan="2">Dialect</td>
    <td><i>AAE</i></td>
    <td>39.17</td>
    <td>17.19</td>
    <td>37.97</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td><i>Jamaican</i></td>
    <td>34.33</td>
    <td>12.98</td>
    <td>32.98</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td><i>Pidgin</i></td>
    <td>34.49</td>
    <td>13.32</td>
    <td>33.00</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td><i>Singlish</i></td>
    <td>40.28</td>
    <td>17.81</td>
    <td>38.51</td>
    <td>0.65</td>
  </tr>
</table>

Table 7: Degree of lexical overlap between unperturbed and perturbed input claims for the <i>CheckThat22</i> dataset. The table reports ROUGE F1 scores (R1, R2, RL), and the normalized Levenshtein distance (Lev.). Higher ROUGE scores and lower Levenshtein distances indicate higher lexical overlap.

<table>
  <tr>
    <th>Perturbation</th>
    <th>Type</th>
    <th>R1</th>
    <th>R2</th>
    <th>RL</th>
    <th>Lev.</th>
  </tr>
  <tr>
    <td rowspan="2">Casing</td>
    <td><i>TrueCase</i></td>
    <td>81.25</td>
    <td>66.67</td>
    <td>81.25</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td><i>UpperCase</i></td>
    <td>6.25</td>
    <td>0.00</td>
    <td>6.25</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td rowspan="2">Typos</td>
    <td><i>Least</i></td>
    <td>42.86</td>
    <td>30.77</td>
    <td>42.86</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td><i>Most</i></td>
    <td>42.86</td>
    <td>15.38</td>
    <td>35.71</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td rowspan="2">Negation</td>
    <td><i>Shallow</i></td>
    <td>96.77</td>
    <td>89.66</td>
    <td>96.77</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td><i>Double</i></td>
    <td>77.78</td>
    <td>66.67</td>
    <td>77.78</td>
    <td>0.31</td>
  </tr>
  <tr>
    <td rowspan="2">Entity Rep.</td>
    <td><i>At least 1</i></td>
    <td>93.75</td>
    <td>93.33</td>
    <td>93.75</td>
    <td>0.21</td>
  </tr>
  <tr>
    <td><i>All</i></td>
    <td>46.67</td>
    <td>14.29</td>
    <td>46.67</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td rowspan="2">LLM Rewrite</td>
    <td><i>Least</i></td>
    <td>48.48</td>
    <td>19.35</td>
    <td>48.48</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td><i>Most</i></td>
    <td>43.75</td>
    <td>20.00</td>
    <td>37.50</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td rowspan="2">Dialect</td>
    <td><i>AAE</i></td>
    <td>33.92</td>
    <td>12.11</td>
    <td>32.29</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td><i>Jamaican</i></td>
    <td>29.52</td>
    <td>9.42</td>
    <td>27.92</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td><i>Pidgin</i></td>
    <td>30.37</td>
    <td>10.25</td>
    <td>28.78</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td><i>Singlish</i></td>
    <td>35.45</td>
    <td>14.15</td>
    <td>33.50</td>
    <td>0.71</td>
  </tr>
</table>

Table 8: Degree of lexical overlap between unperturbed and perturbed input claims for the <i>FactCheckTweet</i> dataset. The table reports ROUGE F1 scores (R1, R2, RL) and the normalized Levenshtein distance (Lev.). Higher ROUGE scores and lower Levenshtein distances indicate higher lexical overlap.

B Human Evaluation

We conduct a human evaluation to assess the effectiveness of our perturbation generation framework in producing <b>valid</b> and <b>natural</b> perturbed claims. Our evaluation follows a similar annotation task to that of Dyrmishi et al. (2023b), which assessed adversarial examples in human settings. We randomly sample 20 unique perturbed input claims from three perturbation types—<i>typos, entity replacement,</i> and <i>dialect</i>. Half of these claims are labeled as valid (1) and the other half as invalid (0) by the <i>LLM as a Verifier</i>. We then present both the unperturbed and perturbed claims, along with the annotation instructions provided below, to three annotators, all of whom are authors. No compensation was paid to the author-annotators.

Annotation Instructions

<b>Your task:</b> For each perturbed claim, evaluate the following criteria:

<b>1. Perturbation Accuracy</b> Does the input claim contain the perturbation_type?
☐ Yes   ☐ No

<b>2. Validity and Naturalness</b>

1. Does the fact-check apply to it, i.e., is the fact-check helpful in verifying the perturbed claim? ☐ Yes   ☐ No

2. Does it convey the same main claim as the unperturbed claim? ☐ Yes   ☐ No

3. Does the perturbed claim feel like something a typical social media user might write? ☐ Yes   ☐ No

We first assess whether the perturbed claim contains the intended perturbation type compared to the original. We define this as perturbation accuracy, measured as the proportion of instances in which the LLM as a Perturber successfully applies the intended change. Next, we evaluate validity (and naturalness) by collecting binary labels from three independent annotators, using an all-or-nothing rule where a positive label is assigned only if all annotators agree. For validity, we report precision on the positive class, reflecting the reliability
of the LLM as a Verifier in identifying valid perturbations. Table 9 shows that our framework achieves a perturbation accuracy of 96.70%, indicating that the intended perturbation was applied in nearly all cases. The verifier attained a 100% precision on the positive class, meaning that every claim labelled as valid by the model was confirmed by human annotators. However, the overall recall score for validity is lower (55.67%), suggesting that while the verifier is highly precise, it may be overly conservative, resulting in some valid instances getting filtered out.

<table>
  <tr>
    <th>Type</th>
    <th># Count</th>
    <th>Acc (%)</th>
    <th>Valid (Pr.)</th>
    <th>Valid (Re.)</th>
    <th>Valid (F1)</th>
  </tr>
  <tr>
    <td><i>Typos</i></td>
    <td>20</td>
    <td>100</td>
    <td>100</td>
    <td>52.63</td>
    <td>68.97</td>
  </tr>
  <tr>
    <td><i>Entity Rep.</i></td>
    <td>20</td>
    <td>90</td>
    <td>100</td>
    <td>58.82</td>
    <td>72.07</td>
  </tr>
  <tr>
    <td><i>Dialect</i></td>
    <td>20</td>
    <td>100</td>
    <td>100</td>
    <td>55.56</td>
    <td>71.43</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>60</th>
    <th>96.70</th>
    <th>100</th>
    <th>55.67</th>
    <th>70.82</th>
  </tr>
</table>

Table 9: Perturbation Evaluation: Acc (%) indicates the proportion of instances where the perturbation was correctly applied by the <i>LLM As a Perturber</i>. Valid (Pr.) represents the precision on valid instances, Valid (Re.) is the recall on valid instances, and Valid (F1) is the overall F1 score.

C Embedding Models Details

The list of all the embedding models evaluated in this study; their parameter count and embedding dimensions are shown in Table 10. We use all embedding models off-the-shelf with default hyperparameters, utilizing the SentenceTransformers library.\footnote{https://sbert.net/} Since the <i>FactCheckTweet</i> dataset contains longer articles and most embedding models support only up to 512 tokens, we split articles into paragraphs and compute similarity between each paragraph and the input text, following Kazemi et al. (2021). This approach is also applied to LLM-based embeddings, despite their support for longer sequences, to ensure fair comparison. To run the BM25 evaluation, we used the rank_bm25 implementation.\footnote{https://github.com/dorianbrown/rank_bm25}

<table>
  <tr>
    <th>Embedding Model</th>
    <th># Params</th>
    <th>\( d_{emb} \)</th>
  </tr>
  <tr>
    <td><a href="https://huggingface.co/all-MiniLM-L12-v2">all-MiniLM-L12-v2</a></td>
    <td>82M</td>
    <td>384</td>
  </tr>
  <tr>
    <td><a href="https://huggingface.co/all-distilroberta-v1">all-distilroberta-v1</a></td>
    <td>82M</td>
    <td>768</td>
  </tr>
  <tr>
    <td><a href="https://huggingface.co/all-mpnet-base-v2">all-mpnet-base-v2</a></td>
    <td>110M</td>
    <td>768</td>
  </tr>
  <tr>
    <td><a href="https://huggingface.co/sentence-t5-base">sentence-t5-base</a></td>
    <td>110M</td>
    <td>768</td>
  </tr>
  <tr>
    <td><a href="https://huggingface.co/sentence-t5-large">sentence-t5-large</a></td>
    <td>335M</td>
    <td>768</td>
  </tr>
  <tr>
    <td><a href="https://huggingface.co/hkunlp/instructor-base">hkunlp/instructor-base</a></td>
    <td>86M</td>
    <td>768</td>
  </tr>
  <tr>
    <td><a href="https://huggingface.co/hkunlp/instructor-large">hkunlp/instructor-large</a></td>
    <td>335M</td>
    <td>768</td>
  </tr>
  <tr>
    <td><a href="https://huggingface.co/nvidia/NV-Embed-v2">nvidia/NV-Embed-v2</a></td>
    <td>7B</td>
    <td>4,096</td>
  </tr>
  <tr>
    <td><a href="https://huggingface.co/Salesforce/SFR-Embedding-Mistral">Salesforce/SFR-Embedding-Mistral</a></td>
    <td>7B</td>
    <td>4,096</td>
  </tr>
</table>

Table 10: Embedding models with their parameter counts (# Params) and embedding dimensions (\( d_{emb} \)). Model names are hyperlinked.

Dori-Hacohen (2022) and finetune the embedding models using contrastive learning with the Multiple Negatives Ranking (MNR) loss. MNR loss optimizes the similarity between positive claim–factcheck pairs while reducing the similarity of negative pairs within a batch. It is defined as:

\[
-\frac{1}{n} \sum_{i=1}^{n} \log \frac{\exp(\operatorname{sim}(a_i, p_i))}{\sum_j \exp(\operatorname{sim}(a_i, p_j))}
\]

where \( f(x) \) and \( f(y) \) are the embedding representations of \( x \) (input claim) and \( y \) (factcheck), respectively. To improve generalization, we introduce “hard negatives” by using BM25 to generate a ranked list of candidate negatives and selecting the top-ranked negative factcheck. We use the training scripts provided by Shliseberg and Dori-Hacohen (2022)\footnote{https://github.com/RIET-lab/GenerativeClaimMatchingPipeline/blob/main/src/dynamicquery/candidate_selection/train_sentence_model.py} with the default hyperparameters: learning rate = \( 5e^{-6} \), batch size = 6, max length = 128, temperature = 0.1, and epochs = 1.

C.1.2 Teacher-Student Knowledge Distillation

We provide the details of the knowledge-distillation approach for improving the robustness of weaker embedding models in this section. This approach depends on parallel sentence pairs \( ((s_1, t_1), \ldots, (s_n, t_n)) \) where \( s_i \) is a source sentence and \( t_i \) the target sentence. In our case, \( s_i \) and \( t_i \) are variations of the same claim. We obtain these parallel sentences by generating perturbations of the CheckThat22 training split for typos, negation, dialect, and entity replacement, resulting in an initial set of 11,593 pairs. We then expand this list by pairing instances of the same claim (e.g., dialect
<table>
  <tr>
    <th>Source Sentence (\( s_i \))</th>
    <th>Target Sentence (\( t_i \))</th>
    <th>Pairing (\( s_i \) vs \( t_i \))</th>
  </tr>
  <tr>
    <td>The US drone attack on #Soleimani caught on camera. #IranUsa</td>
    <td>Dem catch US drone strike on Soleimani pon camera, yah. #IranUsa</td>
    <td><i>Unperturbed vs. Jamaican Patois</i></td>
  </tr>
  <tr>
    <td>The American drone strike on #Qassem caught on camera. #TehranWashington</td>
    <td>Dem capture US drone attack wey hit Soleimani for cam. #IranUsa</td>
    <td><i>Entity Replacement vs. Nigerian Pidgin</i></td>
  </tr>
  <tr>
    <td>It is not false that the US drone attack on #Soleimani was caught on camera.</td>
    <td>Wah, US drone attack on Soleimani kenna caught on video leh. #IranUsa</td>
    <td><i>Negation vs. Singlish</i></td>
  </tr>
  <tr>
    <td>US drone strike on Soleimani capt- trd on video. #IranUsa.</td>
    <td>US drone strike pon Soleimani seen on di camera, seen. #IranUsa</td>
    <td><i>Typos vs. Jamaican Patois</i></td>
  </tr>
</table>

Table 11: Examples of parallel sentences generated using various perturbations.

vs. typos), yielding a larger dataset of 70,954 pairs. Examples of parallel sentences are shown in Table 11. We adopt the same training approach as in Reimers and Gurevych (2020) where we train a student model \( \hat{M} \) such that

\[
\hat{M}(s_i) \approx M(s_i) \quad \text{and} \quad \hat{M}(t_i) \approx M(s_i).
\]

For a given minibatch \( B \), we minimize the mean-squared loss,

\[
\mathcal{L}_{\text{MSE}} = \frac{1}{|B|} \sum_{j \in B} \left[ \left( M(s_j) - \hat{M}(s_j) \right)^2 + \left( M(s_j) - \hat{M}(t_j) \right)^2 \right]
\]

We use all-mpnet-base-v2 as both the student (\( \hat{M} \)) and teacher (\( M \)) model. To train the model, we use the script provided by Reimers and Gurevych (2020)\footnote{https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/multilingual/make_multilingual.py} with the following hyperparameters: learning rate = \( 2e-5 \), batch size = 64, max length = 256, and epochs = 20.

D Rerankers Evaluation

D.1 Selecting a strong reranker

Given the many possible combinations between embedding models and rerankers, we evaluate a wide range of reranker models to select a strong-performing one for further evaluation. In our study, we consider several types of rerankers: pairwise rerankers, such as ms-marco-MiniLM-L-6-v2 and monot5-3b (Nogueira et al., 2020); list-wise rerankers, such as Lit5-Distill (Tamber et al., 2023); and LLM-based rerankers, including bge-reranker-v2-gemma (Chen et al., 2024), rank_zephyr_7b_v1_full (Pradeep et al., 2023), and RankGPT (Sun et al., 2023) (for which we use GPT4o as the reranker). We evaluate these rerankers using the first-stage retrieval rankings from all-mpnet-base-v2 on the unperturbed CheckThai22 test set, setting the number of candidates, i.e., *top-j*, to 20. A value of 20 is chosen as it is a feasible size that fits well within the context window for RankGPT. We use RankLLM\footnote{https://github.com/castorini/rank_llm} implementation for the different reranker models.

Table 12 presents the performance results for the different rerankers. RankGPT, using GPT-4o as the reranker, achieves the best performance. However, its results are comparable to the bge-reranker-v2-gemma model, based on gemma-2b. Given that bge-reranker-v2-gemma is lightweight and can be run locally with lower latency and cost, we select it as a representative SoTA reranker over GPT-4o, which increases latency in production retrieval systems (Sun et al., 2023).
<table>
  <tr>
    <th>Model</th>
    <th>MAP@1</th>
    <th>MAP@5</th>
    <th>MAP@10</th>
    <th>MAP@20</th>
  </tr>
  <tr>
    <td>RankGPT(GPT4o)</td>
    <td>0.908</td>
    <td>0.919</td>
    <td>0.919</td>
    <td>0.919</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-gemma</td>
    <td>0.908</td>
    <td>0.918</td>
    <td>0.918</td>
    <td>0.918</td>
  </tr>
  <tr>
    <td>monot5-3b-msmarco-10k</td>
    <td>0.897</td>
    <td>0.912</td>
    <td>0.912</td>
    <td>0.912</td>
  </tr>
  <tr>
    <td>LIT5-Distill-large-v2</td>
    <td>0.876</td>
    <td>0.902</td>
    <td>0.902</td>
    <td>0.902</td>
  </tr>
  <tr>
    <td>bge-reranker-v2-m3</td>
    <td>0.870</td>
    <td>0.891</td>
    <td>0.892</td>
    <td>0.892</td>
  </tr>
  <tr>
    <td>ms-marco-MiniLM-L-6-v2</td>
    <td>0.854</td>
    <td>0.883</td>
    <td>0.884</td>
    <td>0.884</td>
  </tr>
  <tr>
    <td>rank_zephyr_7b_v1_full</td>
    <td>0.778</td>
    <td>0.845</td>
    <td>0.846</td>
    <td>0.846</td>
  </tr>
</table>

Table 12: Performance comparison of rerankers on CheckThat22 (unperturbed test set). The table reports Mean Average Precision (MAP@k) at different cutoffs (\( k = 1, 5, 10, 20 \)) for rerankers applied to first-stage top-20 candidates from all-mpnet-base-v2.

D.2 Selecting the optimal top-j candidates for reranking

To determine the optimal number of candidates for reranking (\( top-j \)), we evaluate MAP@5 across different values of \( j \in \{5, 10, 20, 50, 100, 200\} \), on the CheckThat22 unperturbed test set. We use MAP@5 to ensure comparability across different values of \( j \). We use two rerankers: cross-encoder/ms-marco-MiniLM-L-6-v2 and bge-reranker-v2-gemma, applied to first-stage retrieval rankings from all-mpnet-base-v2. Figure 3 presents the reranking performance of cross-encoder/ms-marco-MiniLM-L-6-v2, while Figure 4 shows that of bge-reranker-v2-gemma. The cross-encoder/ms-marco-MiniLM-L-6-v2 achieves its highest MAP@5 at \( j = 50 \), whereas the bge-reranker-v2-gemma shows minimal improvement beyond this point. Based on this, we set \( top-j \) to 50 for the reranking experiments, balancing performance and efficiency.

![Bar chart showing MAP@5 performance across different top-j values for cross-encoder/ms-marco-MiniLM-L-6-v2 on CheckThat22. The highest MAP@5 is achieved at j = 50.](page_109_1042_495_246.png)

Figure 3: MAP@5 performance across different top-j values for cross-encoder/ms-marco-MiniLM-L-6-v2 on CheckThat22. The highest MAP@5 is achieved at \( j = 50 \).

E Compute Requirements

All embedding model inferences, reranker fine-tuning, and evaluations are conducted on a single NVIDIA A100 80GB GPU.

![Bar chart showing MAP@5 performance across different top-j values for bge-reranker-v2-gemma on CheckThat22. Reranking improves up to j = 50, after which further gains are minimal.](page_779_1042_495_246.png)

Figure 4: MAP@5 performance across different top-j values for bge-reranker-v2-gemma on CheckThat22. Reranking improves up to \( j = 50 \), after which further gains are minimal.

F Additional Results

F.1 Additional Results Across Different MAP@k on CheckThat22

We provide robustness evaluation results across different MAP@k values, \( k \in \{5, 10, 50\} \), for the CheckThat22 dataset. Table 13 reports the evaluation at MAP@5, Table 14 at MAP@10, and Table 15 at MAP@50, with reranking applied to the top 50 candidates in all cases. We observe minimal performance variations across different MAP@k values on the CheckThat22 dataset, indicating that the robustness results hold across various MAP@k values.

F.2 Results for FactCheckTweet

We report evaluation results across different MAP@k values, \( k \in \{5, 10, 20, 50\} \), for the FactCheckTweet dataset. Table 16 presents the evaluation at MAP@5, Table 17 at MAP@10 and Table 18 at MAP@50, with reranking applied to the top 50 candidates in all cases.

G Mitigation Approaches Comparison

We compare our mitigation approaches on the Robust-LASER (RoLASER) model proposed by Nishimwe et al. (2024), who use a similar teacher–student knowledge distillation approach to improve the robustness of embedding models for user-generated content (UGC). In contrast to our work, they focus only on a subset of perturbations—specifically, those in which UGC resembles the Typos perturbation type—and generate an artificial UGC dataset using rule-based methods. For the KD approach, we compare the performance of the original LASER model with that of Robust-LASER (RoLASER), the adapted model. We also
<table>
  <tr>
    <th>Stage</th>
    <th>Model</th>
    <th colspan="2">Casing</th>
    <th colspan="2">Types</th>
    <th colspan="2">Negation</th>
    <th colspan="2">Entity Replacement</th>
    <th colspan="4">LLM Rewrite</th>
    <th>Dialect</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>TrueCase</th>
    <th>Upper</th>
    <th>Least</th>
    <th>Most</th>
    <th>Shallow</th>
    <th>Double</th>
    <th>Entity</th>
    <th>Replace.</th>
    <th>Add</th>
    <th>Leonard</th>
    <th>Most</th>
    <th>Med</th>
    <th>Add</th>
    <th>Jamaican</th>
    <th>Folke</th>
    <th>Simplfie</th>
  </tr>
  <tr><th rowspan="16">First-Stage Retrieval</th><td>BoS5</td><td>-0.06</td><td>+0.00</td><td>-15.33</td><td>-15.19</td><td>-0.57</td><td>-1.99</td><td>-2.79</td><td>-12.83</td><td>-0.85</td><td>-0.25</td><td>-0.18</td><td>-0.64</td><td>-0.50</td><td>-1.71</td></tr>
  <tr><td>all-distilroberta-v1</td><td>-0.64</td><td>+0.43</td><td>-16.90</td><td>-13.80</td><td>+0.75</td><td>+2.15</td><td>+2.97</td><td>+5.09</td><td>+3.70</td><td>+0.44</td><td>+0.50</td><td>+2.13</td><td>-0.01</td><td>-1.67</td><td>+0.89</td></tr>
  <tr><td>all-MiniLM-Lv2-v2</td><td>0.08</td><td>0.00</td><td>-4.81</td><td>-1.64</td><td>-4.81</td><td>-2.01</td><td>-7.67</td><td>-7.67</td><td>-1.18</td><td>-0.96</td><td>-2.01</td><td>-1.18</td><td>-0.26</td><td>-0.71</td><td>-1.25</td></tr>
  <tr><td>all-mnemonics-v2-ft</td><td>0.00</td><td>0.00</td><td>-8.87</td><td>-8.74</td><td>-0.01</td><td>-1.03</td><td>-3.52</td><td>10.11</td><td>0.07</td><td>1.96</td><td>-5.85</td><td>-7.60</td><td>-3.86</td><td>-1.38</td></tr>
  <tr><td>all-mnemonics-v2-ft</td><td>0.00</td><td>0.00</td><td>-8.65</td><td>-8.52</td><td>-0.13</td><td>-1.73</td><td>-9.04</td><td>-9.04</td><td>-1.44</td><td>-1.69</td><td>-4.09</td><td>-4.23</td><td>-8.88</td><td>-3.29</td></tr>
  <tr><td>instructor-base</td><td>-0.70</td><td>-0.95</td><td>+1.77</td><td>+2.15</td><td>-3.25</td><td>15.05</td><td>8.66</td><td>-7.98</td><td>2.41</td><td>1.20</td><td>+12.63</td><td>+11.58</td><td>+10.93</td><td>+1.47</td><td>+0.90</td></tr>
  <tr><td>instructor-large-ft</td><td>-0.78</td><td>-0.95</td><td>+2.55</td><td>+2.88</td><td>-4.80</td><td>+8.68</td><td>+8.10</td><td>+0.81</td><td>+0.45</td><td>+1.67</td><td>+6.86</td><td>+3.53</td><td>-3.06</td><td>+4.19</td></tr>
  <tr><td>instructor-large-ft</td><td>-0.99</td><td>-1.12</td><td>+2.07</td><td>+1.74</td><td>+1.24</td><td>+8.76</td><td>-2.90</td><td>-1.41</td><td>+1.99</td><td>+1.61</td><td>+4.44</td><td>-2.61</td><td>-3.82</td><td>+4.10</td></tr>
  <tr><td>instructor-large-ft</td><td>-1.01</td><td>-1.11</td><td>+3.45</td><td>+3.75</td><td>+1.84</td><td>+11.53</td><td>+6.73</td><td>+5.33</td><td>+1.65</td><td>+4.36</td><td>+9.66</td><td>+4.15</td><td>+0.67</td><td>+3.82</td></tr>
  <tr><td>instructor-large</td><td>-0.70</td><td>-0.45</td><td>-1.18</td><td>-1.76</td><td>-0.41</td><td>-0.07</td><td>+0.00</td><td>+2.80</td><td>0.09</td><td>-0.79</td><td>-0.84</td><td>-2.57</td><td>-4.64</td><td>-2.12</td></tr>
  <tr><td>SFR-Embedding-Mistral</td><td>-0.62</td><td>-0.96</td><td>-0.76</td><td>-0.62</td><td>-1.37</td><td>-1.87</td><td>-0.64</td><td>-1.15</td><td>0.09</td><td>-0.68</td><td>-0.15</td><td>-0.76</td><td>-1.15</td><td>-2.31</td></tr>
  <tr><td>Mnemonics-v2</td><td>-0.73</td><td>-0.60</td><td>+0.04</td><td>+0.07</td><td>0.04</td><td>+0.14</td><td>+0.29</td><td>+0.30</td><td>0.07</td><td>+0.14</td><td>+0.10</td><td>+0.27</td><td>+0.17</td><td>+0.19</td></tr>
</table>

Table 13: Effect of perturbations on CheckThatTweet (MAP@5). First-Stage Retrieval shows the retrieval gap between unperturbed and perturbed inputs: Δ(MAP@5)retrieval. Reranking Recovery measures the Δrecovery gap for the perturbed input set before and after reranking top-50 candidates, with improvements highlighted in green and drops in red. Overall Pipeline shows the Δoverall gap for combined retrieval and reranking between unperturbed and perturbed inputs. Colors indicate positive or negative deltas compared to unperturbed. MAP@5 is used for all comparisons. Models finetuned on the task are indicated with the postfix ft.

<table>
  <tr>
    <th>Stage</th>
    <th>Model</th>
    <th colspan="2">Casing</th>
    <th colspan="2">Types</th>
    <th colspan="2">Negation</th>
    <th colspan="2">Entity Replacement</th>
    <th colspan="4">LLM Rewrite</th>
    <th>Dialect</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>TrueCase</th>
    <th>Upper</th>
    <th>Least</th>
    <th>Most</th>
    <th>Shallow</th>
    <th>Double</th>
    <th>Entity</th>
    <th>Replace.</th>
    <th>Add</th>
    <th>Leonard</th>
    <th>Most</th>
    <th>Med</th>
    <th>Add</th>
    <th>Jamaican</th>
    <th>Folke</th>
    <th>Simplfie</th>
  </tr>
  <tr><th rowspan="16">First-Stage Retrieval</th><td>BoS5</td><td>-0.19</td><td>+1.19</td><td>+25.63</td><td>+20.86</td><td>+6.51</td><td>+7.52</td><td>+13.25</td><td>+19.50</td><td>+0.14</td><td>+6.88</td><td>+14.89</td><td>+6.52</td><td>+14.55</td><td>+11.66</td></tr>
  <tr><td>all-distilroberta-v1</td><td>+2.11</td><td>+4.18</td><td>+27.63</td><td>+25.36</td><td>+14.38</td><td>+12.73</td><td>+22.37</td><td>+16.76</td><td>+1.65</td><td>+1.44</td><td>+2.90</td><td>+2.71</td><td>+1.21</td><td>+1.76</td><td>-18.33</td></tr>
  <tr><td>all-MiniLM-Lv2-v2</td><td>+1.99</td><td>+4.86</td><td>+28.66</td><td>+25.86</td><td>+10.81</td><td>+13.17</td><td>+26.48</td><td>+10.57</td><td>+3.52</td><td>+3.14</td><td>+7.25</td><td>+10.23</td><td>+4.55</td><td>+5.85</td><td>+1.31</td></tr>
  <tr><td>all-mnemonics-v2-ft</td><td>+1.99</td><td>+2.31</td><td>+28.28</td><td>+22.22</td><td>+13.17</td><td>+13.17</td><td>+22.00</td><td>+26.56</td><td>+1.65</td><td>+2.75</td><td>+2.39</td><td>+21.91</td><td>+2.50</td><td>+18.24</td></tr>
  <tr><td>all-mnemonics-v2-ft</td><td>+6.19</td><td>+6.87</td><td>+10.39</td><td>+17.12</td><td>+6.85</td><td>+5.41</td><td>+9.54</td><td>+12.02</td><td>+8.63</td><td>+6.95</td><td>+9.79</td><td>+8.26</td><td>+14.98</td><td>+10.55</td></tr>
  <tr><td>instructor-base</td><td>+15.04</td><td>+18.55</td><td>+19.87</td><td>+14.25</td><td>+16.24</td><td>+19.79</td><td>+15.60</td><td>+13.21</td><td>+15.96</td><td>+14.76</td><td>+2.55</td><td>+1.47</td><td>+6.16</td><td>+6.36</td><td>+6.87</td></tr>
  <tr><td>instructor-large-ft</td><td>+14.26</td><td>+20.68</td><td>+15.94</td><td>+14.18</td><td>+25.30</td><td>+18.77</td><td>+16.23</td><td>+21.47</td><td>+14.30</td><td>+13.88</td><td>+16.25</td><td>+17.22</td><td>+14.81</td><td>+14.26</td></tr>
  <tr><td>instructor-large-ft</td><td>+5.40</td><td>+4.68</td><td>+4.81</td><td>+4.81</td><td>+4.91</td><td>+5.41</td><td>+4.89</td><td>+4.89</td><td>+4.61</td><td>+4.11</td><td>+5.53</td><td>+6.69</td><td>+4.36</td><td>+4.16</td></tr>
  <tr><td>instructor-large</td><td>+2.12</td><td>+6.01</td><td>+4.11</td><td>+2.33</td><td>+3.35</td><td>+5.00</td><td>+1.93</td><td>+4.10</td><td>+0.31</td><td>+2.19</td><td>+4.18</td><td>+3.63</td><td>+5.51</td><td>+4.19</td></tr>
  <tr><td>SFR-Embedding-Mistral</td><td>+0.91</td><td>+2.12</td><td>+1.11</td><td>+0.45</td><td>+0.45</td><td>+0.97</td><td>+0.20</td><td>+2.45</td><td>-0.09</td><td>+0.20</td><td>+1.57</td><td>+0.17</td><td>-0.46</td><td>-0.50</td></tr>
  <tr><td>Mnemonics-v2</td><td>+0.22</td><td>-0.40</td><td>-3.47</td><td>-5.56</td><td>-3.16</td><td>-3.69</td><td>-3.02</td><td>-3.21</td><td>-0.08</td><td>-0.99</td><td>-5.88</td><td>-2.63</td><td>-2.26</td><td>-1.07</td></tr>
  <tr><th rowspan="16">Overall Pipeline</th><td>BoS5</td><td>-0.77</td><td>+2.37</td><td>-9.19</td><td>-7.08</td><td>+5.82</td><td>-2.33</td><td>-7.49</td><td>+7.04</td><td>+0.10</td><td>4.68</td><td>-5.69</td><td>-1.09</td><td>-0.09</td><td>-1.43</td></tr>
  <tr><td>all-distilroberta-v1</td><td>+3.54</td><td>+5.57</td><td>+14.82</td><td>+14.02</td><td>+4.95</td><td>+3.47</td><td>+8.33</td><td>+10.16</td><td>+1.84</td><td>+2.43</td><td>+6.43</td><td>+4.69</td><td>+4.19</td><td>+3.13</td></tr>
  <tr><td>all-MiniLM-Lv2-v2</td><td>-0.60</td><td>+4.12</td><td>-5.89</td><td>-3.66</td><td>-4.71</td><td>-6.70</td><td>-0.32</td><td>-1.30</td><td>-1.37</td><td>-1.76</td><td>-1.79</td><td>-2.44</td><td>-3.24</td><td>-2.35</td></tr>
  <tr><td>all-mnemonics-v2-ft</td><td>-0.60</td><td>+0.08</td><td>-1.88</td><td>-1.81</td><td>-6.14</td><td>-6.14</td><td>-0.21</td><td>-2.22</td><td>-0.25</td><td>-0.48</td><td>-0.99</td><td>-1.61</td><td>-1.08</td><td>-0.29</td></tr>
  <tr><td>instructor-base</td><td>-2.55</td><td>+15.99</td><td>-8.28</td><td>-7.17</td><td>-21.48</td><td>-12.49</td><td>-13.51</td><td>-5.46</td><td>-2.40</td><td>+1.93</td><td>-1.89</td><td>+5.55</td><td>-0.53</td></tr>
  <tr><td>instructor-large-ft</td><td>-1.00</td><td>-1.92</td><td>-3.01</td><td>-1.57</td><td>-6.14</td><td>-4.29</td><td>+1.32</td><td>+0.81</td><td>+1.07</td><td>+0.41</td><td>+0.42</td><td>+1.48</td><td>+0.74</td><td>+1.18</td></tr>
  <tr><td>instructor-large-ft</td><td>-0.99</td><td>-1.92</td><td>-1.99</td><td>-0.99</td><td>+2.73</td><td>+1.73</td><td>+1.73</td><td>+1.88</td><td>+1.17</td><td>+0.49</td><td>+4.52</td><td>+2.46</td><td>+3.58</td><td>+1.06</td></tr>
  <tr><td>instructor-large</td><td>-0.91</td><td>-0.97</td><td>+1.88</td><td>+1.79</td><td>+0.76</td><td>+0.29</td><td>+0.06</td><td>+0.92</td><td>-0.01</td><td>-0.57</td><td>+1.28</td><td>-1.40</td><td>+0.80</td><td>-1.05</td></tr>
  <tr><td>SFR-Embedding-Mistral</td><td>-0.63</td><td>-0.59</td><td>-0.68</td><td>-0.80</td><td>+0.32</td><td>-0.28</td><td>-0.93</td><td>-0.15</td><td>+0.09</td><td>+0.73</td><td>+1.95</td><td>+0.87</td><td>+1.62</td><td>+0.23</td></tr>
  <tr><td>Mnemonics-v2</td><td>+0.71</td><td>+0.22</td><td>-2.16</td><td>-4.79</td><td>-2.88</td><td>-4.66</td><td>+0.10</td><td>-1.34</td><td>+0.28</td><td>-0.97</td><td>-1.20</td><td>+0.25</td><td>+1.29</td><td>-1.15</td></tr>
</table>

Table 14: Effect of perturbations on CheckThat22 (MAP@10). First-Stage Retrieval shows the retrieval gap between unperturbed and perturbed inputs: Δ(MAP@10)retrieval. Reranking Recovery measures the Δrecovery gap for the perturbed input set before and after reranking top-50 candidates, with improvements highlighted in green and drops in red. Overall Pipeline shows the Δoverall gap for combined retrieval and reranking between unperturbed and perturbed inputs. Colors indicate positive or negative deltas compared to unperturbed. MAP@10 is used for all comparisons. Models finetuned on the task are indicated with the postfix ft.

evaluate the effect of claim normalization using the same setup as in our approach. The results for this experiment are shown in Figure 5.

Although the mitigation approaches are not directly comparable here due to the use of different embedding models (i.e., LASER vs.
<table>
  <tr>
    <th rowspan="2">Stage</th>
    <th rowspan="2">Model</th>
    <th colspan="4">Casing</th>
    <th colspan="4">Types</th>
    <th colspan="4">Negation</th>
    <th colspan="4">Entity Replacement</th>
    <th colspan="4">LLM Rewrite</th>
    <th colspan="4">Dialect</th>
  </tr>
  <tr>
    <th>TrueCase</th>
    <th>Upper</th>
    <th>Lower</th>
    <th>Most</th>
    <th>Shallower</th>
    <th>Deeper</th>
    <th>Double</th>
    <th>Alpha</th>
    <th>All</th>
    <th>LLM Rewrite Alpha</th>
    <th>All</th>
    <th>AEE</th>
    <th>Jamaican</th>
    <th>Poisen</th>
    <th>Simplif</th>
  </tr>
  <tr><td colspan="25">BOS</td></tr>
  <tr><td colspan="25">all-distilbert-base-v1</td></tr>
  <tr><td colspan="25">all-MiniLM-L12-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2-ft</td></tr>
  <tr><td colspan="25">sentence-15-base</td></tr>
  <tr><td colspan="25">sentence-15-large</td></tr>
  <tr><td colspan="25">instructor-base</td></tr>
  <tr><td colspan="25">instructor-large</td></tr>
  <tr><td colspan="25">SFB-Embedding-Mistral</td></tr>
  <tr><td colspan="25">SFB-Embedding-v2</td></tr>
  <tr><td colspan="25">Wt-Embed-v2</td></tr>
  <tr><td colspan="25">Re-ranking Recovery</td></tr>
  <tr><td colspan="25">all-distilbert-base-v1</td></tr>
  <tr><td colspan="25">all-MiniLM-L12-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2-ft</td></tr>
  <tr><td colspan="25">sentence-15-base</td></tr>
  <tr><td colspan="25">sentence-15-large</td></tr>
  <tr><td colspan="25">instructor-base</td></tr>
  <tr><td colspan="25">instructor-large</td></tr>
  <tr><td colspan="25">SFB-Embedding-Mistral</td></tr>
  <tr><td colspan="25">SFB-Embedding-v2</td></tr>
  <tr><td colspan="25">Wt-Embed-v2</td></tr>
  <tr><td colspan="25">Overall Pipeline</td></tr>
  <tr><td colspan="25">all-distilbert-base-v1</td></tr>
  <tr><td colspan="25">all-MiniLM-L12-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2-ft</td></tr>
  <tr><td colspan="25">sentence-15-base</td></tr>
  <tr><td colspan="25">sentence-15-large</td></tr>
  <tr><td colspan="25">instructor-base</td></tr>
  <tr><td colspan="25">instructor-large</td></tr>
  <tr><td colspan="25">SFB-Embedding-Mistral</td></tr>
  <tr><td colspan="25">SFB-Embedding-v2</td></tr>
  <tr><td colspan="25">Wt-Embed-v2</td></tr>
</table>

Table 15: Effect of perturbations on CheckThat22 (MAP@50). First-Stage Retrieval shows the *retrieval gap* between unperturbed and perturbed inputs: \( \Delta (\text{MAP}@5)_{\text{retrieval}} \). **Reranking Recovery** measures the *\( \Delta \)recovery gap* for the perturbed input set before and after reranking top-50 candidates, with improvements highlighted in green and drops in red. **Overall Pipeline** shows the *\( \Delta \)overall gap* for combined retrieval and reranking between unperturbed and perturbed inputs. Colors indicate positive or negative deltas compared to unperturbed. MAP@50 is used for all comparisons. Models finetuned on the task are indicated with the postfix *ft*.

<table>
  <tr>
    <th rowspan="2">Stage</th>
    <th rowspan="2">Model</th>
    <th colspan="4">Casing</th>
    <th colspan="4">Types</th>
    <th colspan="4">Negation</th>
    <th colspan="4">Entity Replacement</th>
    <th colspan="4">LLM Rewrite</th>
    <th colspan="4">Dialect</th>
  </tr>
  <tr>
    <th>TrueCase</th>
    <th>Upper</th>
    <th>Lower</th>
    <th>Most</th>
    <th>Shallower</th>
    <th>Deeper</th>
    <th>Double</th>
    <th>Alpha</th>
    <th>All</th>
    <th>LLM Rewrite Alpha</th>
    <th>All</th>
    <th>AEE</th>
    <th>Jamaican</th>
    <th>Poisen</th>
    <th>Simplif</th>
  </tr>
  <tr><td colspan="25">BOS</td></tr>
  <tr><td colspan="25">all-distilbert-base-v1</td></tr>
  <tr><td colspan="25">all-MiniLM-L12-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2-ft</td></tr>
  <tr><td colspan="25">sentence-15-base</td></tr>
  <tr><td colspan="25">sentence-15-large</td></tr>
  <tr><td colspan="25">instructor-base</td></tr>
  <tr><td colspan="25">instructor-large</td></tr>
  <tr><td colspan="25">SFB-Embedding-Mistral</td></tr>
  <tr><td colspan="25">SFB-Embedding-v2</td></tr>
  <tr><td colspan="25">Wt-Embed-v2</td></tr>
  <tr><td colspan="25">Re-ranking Recovery</td></tr>
  <tr><td colspan="25">all-distilbert-base-v1</td></tr>
  <tr><td colspan="25">all-MiniLM-L12-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2-ft</td></tr>
  <tr><td colspan="25">sentence-15-base</td></tr>
  <tr><td colspan="25">sentence-15-large</td></tr>
  <tr><td colspan="25">instructor-base</td></tr>
  <tr><td colspan="25">instructor-large</td></tr>
  <tr><td colspan="25">SFB-Embedding-Mistral</td></tr>
  <tr><td colspan="25">SFB-Embedding-v2</td></tr>
  <tr><td colspan="25">Wt-Embed-v2</td></tr>
  <tr><td colspan="25">Overall Pipeline</td></tr>
  <tr><td colspan="25">all-distilbert-base-v1</td></tr>
  <tr><td colspan="25">all-MiniLM-L12-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2</td></tr>
  <tr><td colspan="25">all-mpnet-base-v2-ft</td></tr>
  <tr><td colspan="25">sentence-15-base</td></tr>
  <tr><td colspan="25">sentence-15-large</td></tr>
  <tr><td colspan="25">instructor-base</td></tr>
  <tr><td colspan="25">instructor-large</td></tr>
  <tr><td colspan="25">SFB-Embedding-Mistral</td></tr>
  <tr><td colspan="25">SFB-Embedding-v2</td></tr>
  <tr><td colspan="25">Wt-Embed-v2</td></tr>
</table>

Table 16: Effect of perturbations on FactCheckTweet (MAP@5). First-Stage Retrieval shows the *retrieval gap* between unperturbed and perturbed inputs: \( \Delta (\text{MAP}@5)_{\text{retrieval}} \). **Reranking Recovery** measures the *\( \Delta \)recovery gap* for the perturbed input set before and after reranking top-50 candidates, with improvements highlighted in green and drops in red. **Overall Pipeline** shows the *\( \Delta \)overall gap* for combined retrieval and reranking between unperturbed and perturbed inputs. Colors indicate positive or negative deltas compared to unperturbed. MAP@5 is used for all comparisons. Models finetuned on the task are indicated with the postfix *ft*.

all-mpnet-base-v2), we observe that *claim normalization* is generally beneficial across all perturbation types. Additionally, RoLASER outperforms the standard LASER model on typos and shows marginal improvement on dialect perturbations. We hypothesize that this occurs because RoLASER is
Table 17: Effect of perturbations on FactCheckTweet (MAP@10). First-Stage Retrieval shows the retrieval gap between unperturbed and perturbed inputs: Δ(MAP@10)retrieval. Reranking Recovery measures the Δrecovery gap for the perturbed input set before and after reranking top-50 candidates, with improvements highlighted in green and drops in red. Overall Pipeline shows the Δoverall gap for combined retrieval and reranking between unperturbed and perturbed inputs. Colors indicate positive or negative deltas compared to unperturbed. MAP@10 is used for all comparisons. Models finetuned on the task are indicated with the postfix ft.

<table>
  <tr>
    <th rowspan="2">Stage</th>
    <th rowspan="2">Model</th>
    <th colspan="3">Casing</th>
    <th colspan="3">Typo</th>
    <th colspan="3">Negation</th>
    <th colspan="3">Entity Replacement</th>
    <th colspan="3">LLM Rewrite</th>
    <th colspan="3">Dialect</th>
  </tr>
  <tr>
    <th>TrueCase</th>
    <th>Upper</th>
    <th>Lower</th>
    <th>Most</th>
    <th>Shallow</th>
    <th>Dual</th>
    <th>Hit</th>
    <th>Miss</th>
    <th>False</th>
    <th>Hit</th>
    <th>Miss</th>
    <th>False</th>
    <th>Hit</th>
    <th>Miss</th>
    <th>False</th>
    <th>Hit</th>
    <th>Miss</th>
    <th>False</th>
  </tr>
  <tr>
    <td rowspan="8">First-Stage Retrieval</td>
    <td>BERTS</td>
    <td>+0.66</td><td>-0.09</td><td>-0.18</td><td>+0.49</td><td>+0.34</td><td>+0.49</td><td>+0.17</td><td>-0.19</td><td>+0.02</td><td>+0.63</td><td>+0.53</td><td>+0.75</td><td>+0.63</td><td>+0.75</td><td>+0.63</td><td>+0.75</td><td>+0.63</td><td>+0.75</td>
  </tr>
  <tr>
    <td>all-distilbertorta-v1</td>
    <td>-1.24</td><td>-2.82</td><td>-1.84</td><td>-1.44</td><td>-0.97</td><td>-1.44</td><td>-0.72</td><td>-1.10</td><td>-0.49</td><td>-0.43</td><td>-0.49</td><td>-0.49</td><td>-0.43</td><td>-0.49</td><td>-0.43</td><td>-0.49</td><td>-0.43</td><td>-0.49</td>
  </tr>
  <tr>
    <td>all-MiniLM-L12-v2</td>
    <td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td>
  </tr>
  <tr>
    <td>all-mpt-base-v2-ft</td>
    <td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td><td>+0.00</td>
  </tr>
  <tr>
    <td>sentence-t5-base</td>
    <td>+0.03</td><td>-0.18</td><td>-0.86</td><td>-0.53</td><td>-0.18</td><td>-0.53</td><td>-0.18</td><td>-0.53</td><td>-0.18</td><td>-0.53</td><td>-0.18</td><td>-0.53</td><td>-0.18</td><td>-0.53</td><td>-0.18</td><td>-0.53</td><td>-0.18</td><td>-0.53</td>
  </tr>
  <tr>
    <td>sentence-t5-large</td>
    <td>+0.53</td><td>-2.71</td><td>-1.71</td><td>-1.13</td><td>+1.27</td><td>-3.37</td><td>-2.22</td><td>-1.06</td><td>-1.96</td><td>-1.73</td><td>-0.32</td><td>-4.61</td><td>-1.44</td><td>-1.06</td><td>-1.96</td><td>-1.73</td><td>-0.32</td><td>-4.61</td>
  </tr>
  <tr>
    <td>instructor-large</td>
    <td>+0.53</td><td>-2.26</td><td>-1.81</td><td>-0.63</td><td>+1.19</td><td>-2.44</td><td>-1.96</td><td>+0.64</td><td>-1.38</td><td>-1.50</td><td>-0.19</td><td>-3.89</td><td>-0.97</td><td>+0.64</td><td>-1.38</td><td>-1.50</td><td>-0.19</td><td>-3.89</td>
  </tr>
  <tr>
    <td>SFB-Embedding-Mistral</td>
    <td>+0.53</td><td>-1.61</td><td>-0.88</td><td>-0.64</td><td>+1.47</td><td>-1.98</td><td>-1.19</td><td>+0.64</td><td>-1.28</td><td>-1.22</td><td>-0.53</td><td>-2.53</td><td>-0.86</td><td>+0.64</td><td>-1.28</td><td>-1.22</td><td>-0.53</td><td>-2.53</td>
  </tr>
  <tr>
    <td>tv-Gated-v1</td>
    <td>+0.53</td><td>-0.76</td><td>+0.68</td><td>-0.68</td><td>+0.42</td><td>+0.11</td><td>+0.52</td><td>+1.84</td><td>+1.07</td><td>+1.11</td><td>+0.51</td><td>+0.97</td><td>+0.51</td><td>+1.84</td><td>+1.07</td><td>+1.11</td><td>+0.51</td><td>+0.97</td>
  </tr>
  <tr>
    <td rowspan="8">Overall Pipeline</td>
    <td>BERTS</td>
    <td>+4.89</td><td>+5.88</td><td>+12.75</td><td>+11.88</td><td>+5.62</td><td>+6.12</td><td>+12.49</td><td>+15.79</td><td>+9.62</td><td>+12.61</td><td>+11.91</td><td>+16.11</td><td>+17.46</td><td>+16.60</td><td>+16.11</td><td>+17.46</td><td>+16.60</td><td>+16.11</td>
  </tr>
  <tr>
    <td>all-distilbertorta-v1</td>
    <td>+12.13</td><td>+16.30</td><td>+19.65</td><td>+14.46</td><td>+14.23</td><td>+19.92</td><td>+13.86</td><td>+13.24</td><td>+14.72</td><td>+15.61</td><td>+15.08</td><td>+23.09</td><td>+21.76</td><td>+18.75</td><td>+14.72</td><td>+15.61</td><td>+15.08</td><td>+23.09</td>
  </tr>
  <tr>
    <td>all-MiniLM-L12-v2</td>
    <td>+12.03</td><td>+12.58</td><td>+15.82</td><td>+18.64</td><td>+17.60</td><td>+17.70</td><td>+20.22</td><td>+15.75</td><td>+17.38</td><td>+23.39</td><td>+19.57</td><td>+18.72</td><td>+16.59</td><td>+17.59</td><td>+17.38</td><td>+23.39</td><td>+19.57</td><td>+18.72</td>
  </tr>
  <tr>
    <td>all-mpt-base-v2-ft</td>
    <td>+11.07</td><td>+11.62</td><td>+12.47</td><td>+12.22</td><td>+9.48</td><td>+15.23</td><td>+18.00</td><td>+13.66</td><td>+10.88</td><td>+14.14</td><td>+13.36</td><td>+15.57</td><td>+14.75</td><td>+13.61</td><td>+10.88</td><td>+14.14</td><td>+13.36</td><td>+15.57</td>
  </tr>
  <tr>
    <td>sentence-t5-base</td>
    <td>+14.44</td><td>+18.45</td><td>+21.40</td><td>+15.93</td><td>+23.44</td><td>+19.75</td><td>+14.68</td><td>+19.30</td><td>+18.90</td><td>+20.56</td><td>+19.72</td><td>+19.80</td><td>+19.35</td><td>+19.96</td><td>+18.90</td><td>+20.56</td><td>+19.72</td><td>+19.80</td>
  </tr>
  <tr>
    <td>sentence-t5-large</td>
    <td>+10.26</td><td>+12.12</td><td>+15.36</td><td>+13.26</td><td>+13.83</td><td>+14.62</td><td>+15.74</td><td>+13.33</td><td>+14.69</td><td>+14.12</td><td>+16.73</td><td>+19.09</td><td>+11.95</td><td>+13.88</td><td>+14.69</td><td>+14.12</td><td>+16.73</td><td>+19.09</td>
  </tr>
  <tr>
    <td>instructor-large</td>
    <td>+8.11</td><td>+8.71</td><td>+14.84</td><td>+10.39</td><td>+11.18</td><td>+13.79</td><td>+15.44</td><td>+16.18</td><td>+10.94</td><td>+14.34</td><td>+9.59</td><td>+12.21</td><td>+14.59</td><td>+15.73</td><td>+10.94</td><td>+14.34</td><td>+9.59</td><td>+12.21</td>
  </tr>
  <tr>
    <td>SFB-Embedding-Mistral</td>
    <td>+4.89</td><td>+6.46</td><td>+10.90</td><td>+8.75</td><td>+7.45</td><td>+10.82</td><td>+12.48</td><td>+12.53</td><td>+1.85</td><td>+2.24</td><td>+1.91</td><td>+1.84</td><td>+2.24</td><td>+1.91</td><td>+1.85</td><td>+2.24</td><td>+1.91</td><td>+1.84</td>
  </tr>
  <tr>
    <td>tv-Gated-v1</td>
    <td>+4.27</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td><td>+4.29</td>
  </tr>
</table>

Table 18: Effect of perturbations on FactCheckTweet (MAP@50). First-Stage Retrieval shows the retrieval gap between unperturbed and perturbed inputs: Δ(MAP@50)retrieval. Reranking Recovery measures the Δrecovery gap for the perturbed input set before and after reranking top-50 candidates, with improvements highlighted in green and drops in red. Overall Pipeline shows the Δoverall gap for combined retrieval and reranking between unperturbed and perturbed inputs. Colors indicate positive or negative deltas compared to unperturbed. MAP@50 is used for all comparisons. Models finetuned on the task are indicated with the postfix ft.

<table>
  <tr>
    <th rowspan="2">Stage</th>
    <th rowspan="2">Model</th>
    <th colspan="3">Casing</th>
    <th colspan="3">Typo</th>
    <th colspan="3">Negation</th>
    <th colspan="3">Entity Replacement</th>
    <th colspan="3">LLM Rewrite</th>
    <th colspan="3">Dialect</th>
  </tr>
  <tr>
    <th>TrueCase</th>
    <th>Upper</th>
    <th>Lower</th>
    <th>Most</th>
    <th>Shallow</th>
    <th>Dual</th>
    <th>Hit</th>
    <th>Miss</th>
    <th>False</th>
    <th>Hit</th>
    <th>Miss</th>
    <th>False</th>
    <th>Hit</th>
    <th>Miss</th>
    <th>False</th>
    <th>Hit</th>
    <th>Miss</th>
    <th>False</th>
  </tr>
  <tr>
    <td rowspan="8">First-Stage Retrieval</td>
    <td>BERTS</td>
    <td>+0.06</td><td>-0.09</td><td>-0.18</td><td>+0.49</td><td>+0.34</td><td>+0.49</td><td>+0.17</td><td>-0.19</td><td>+0.02</td><td>+0.63</td><td>+0.53</td><td>+0.75</td><td>+0.63</td><td>+0.75</td><td>+0.63</td><td>+0.75</td><td>+0.63</td><td>+0.75</td>
  </tr>
  <tr>
    <td>all-distilbertorta-v1</td>
    <td>-1.85</td><td>-20.33</td><td>-6.93</td><td>-16.88</td><td>+4.89</td><td>+4.92</td><td>-3.71</td><td>-18.23</td><td>+1.09</td><td>-1.22</td><td>-1.72</td><td>+2.91</td><td>-2.78</td><td>-0.89</td><td>+1.09</td><td>-1.22</td><td>-1.72</td><td>+2.91</td>
  </tr>
  <tr>
    <td>all-MiniLM-L12-v2</td>
    <td>+0.22</td><td>+0.62</td><td>-9.85</td><td>-14.28</td><td>-0.29</td><td>+3.17</td><td>-4.88</td><td>-1.91</td><td>+1.42</td><td>-1.14</td><td>-1.30</td><td>-9.88</td><td>-2.88</td><td>-2.88</td><td>+1.42</td><td>-1.14</td><td>-1.30</td><td>-9.88</td>
  </tr>
  <tr>
    <td>all-mpt-base-v2-ft</td>
    <td>+0.16</td><td>-0.39</td><td>-5.89</td><td>-7.59</td><td>+0.49</td><td>+4.64</td><td>-3.93</td><td>+0.96</td><td>+0.44</td><td>+3.55</td><td>-1.84</td><td>-1.41</td><td>-1.37</td><td>+0.44</td><td>+3.55</td><td>-1.84</td><td>-1.41</td><td>-1.37</td>
  </tr>
  <tr>
    <td>sentence-t5-base</td>
    <td>+0.20</td><td>+0.53</td><td>+2.77</td><td>+1.83</td><td>+2.77</td><td>+1.83</td><td>+0.53</td><td>+2.77</td><td>+1.83</td><td>+2.77</td><td>+1.83</td><td>+2.77</td><td>+1.83</td><td>+2.77</td><td>+1.83</td><td>+2.77</td><td>+1.83</td><td>+2.77</td>
  </tr>
  <tr>
    <td>sentence-t5-large</td>
    <td>+0.35</td><td>-2.23</td><td>-2.01</td><td>-4.69</td><td>+5.26</td><td>+4.73</td><td>+0.82</td><td>+1.85</td><td>+1.35</td><td>+1.83</td><td>+1.35</td><td>+1.83</td><td>+1.35</td><td>+1.83</td><td>+1.35</td><td>+1.83</td><td>+1.35</td><td>+1.83</td>
  </tr>
  <tr>
    <td>instructor-large</td>
    <td>+0.94</td><td>-3.52</td><td>-1.73</td><td>-2.23</td><td>+1.90</td><td>+1.82</td><td>-1.14</td><td>-1.78</td><td>+0.74</td><td>-0.18</td><td>-3.89</td><td>-1.61</td><td>-1.69</td><td>+0.74</td><td>-0.18</td><td>-3.89</td><td>-1.61</td><td>-1.69</td>
  </tr>
  <tr>
    <td>SFB-Embedding-Mistral</td>
    <td>-0.20</td><td>+0.23</td><td>+0.83</td><td>-3.97</td><td>+0.21</td><td>+0.57</td><td>+1.45</td><td>+1.02</td><td>+0.27</td><td>+0.57</td><td>+1.02</td><td>+0.27</td><td>+0.57</td><td>+1.02</td><td>+0.27</td><td>+0.57</td><td>+1.02</td><td>+0.27</td>
  </tr>
  <tr>
    <td>tv-Gated-v1</td>
    <td>+0.64</td><td>-0.59</td><td>-4.55</td><td>+0.64</td><td>+0.47</td><td>+0.80</td><td>+1.74</td><td>+1.59</td><td>+1.31</td><td>+0.88</td><td>+0.26</td><td>+1.19</td>
  </tr>
</table>

trained exclusively on user-generated content and not on other variations (e.g., dialect or entity replacements), which limits its generalizability to perturbation types not included in the augmentation set. Finally, we note that the LASER models generally perform poorly on this task compared to
the other embedding models we evaluate.

H  Scaling Laws and Performance Drop

We report an analysis of the performance gap as reported in Table 2, and the models’ size and embedding used to represent the sentences at each stage. Figures 6, 9 reports the correlation for the First-Stage Retrieval (embedding and model size); Figures 7 and 10 that for Reranking Recovery (embedding and model size); Figures 8 and 11 the Overall Pipeline (embedding and model size). We excluded BM25 from the analysis as there is no proper notion of embedding dimension for that technique. In general, we do not observe any significant correlation between the embedding dimension and the performance gap.
Figure 5: Effect of mitigation approaches on worst-case perturbations for typos, dialect, and entity replacement. For each embedding model, markers represent performance on the unperturbed input set, while the bars indicate performance on the perturbed set. RoLASER is a robust-LASER model adapted for user-generated content (Nishimwe et al., 2024), while models denoted with +CN incorporate claim normalization.

Figure 6: Correlation between a model embedding’s size and the performance gap for the First-Stage Retrieval as reported in Table 2.
Figure 7: Correlation between a model embedding’s size and the performance gap for the Reranking Recovery as reported in Table 2.

Figure 8: Correlation between a model embedding’s size and the performance gap for the Overall Performance as reported in Table 2.
Figure 9: Correlation between model size (millions of parameters) and the performance gap for the First-Stage Retrieval as reported in Table 2.

Figure 10: Correlation between model size (millions of parameters) and the performance gap for the Reranking Recovery as reported in Table 2.
Figure 11: Correlation between model size (millions of parameters) and the performance gap for the Overall Performance as reported in Table 2.
1 Prompts

We provide examples of prompts used in the perturbation generation framework, including the *LLM as a perturber* and *LLM as a verifier* prompts, as well as the *claim normalization* prompt used in the mitigation approaches. For brevity, we only present prompts for *Dialect* perturbation and verification; all other prompts are available in the repository. To encourage diversity in the generated perturbations, we set the generation temperature to 0.9, while for verification and claim normalization, we use a temperature of 0 to ensure reproducibility.

**Dialect Perturbation**

You are now a social media user tasked with rewriting a given tweet in different English dialects. You will receive two inputs:

1. A claim (original tweet)

2. A fact-check that supports or refutes the claim

Your task: Rewrite the given tweet in as many different ways as possible using the following dialects: African American Vernacular English, Nigerian Pidgin English, Singlish (Singapore English), Jamaican Patois
Ensure that:

• The fact-check remains applicable to the rewritten tweets.

• The rewritten tweets convey the same main claim as the original.

• The rewritten tweets read naturally and do not appear suspiciously altered.

Example:
Original tweet: “Biden signed an executive order today banning the term ’China virus’.”
Fact-check: “President Joe Biden issued an executive order in January 2021 banning the term ‘China virus.’ ”
Possible rewrites:
• Biden done signed an order today, no more sayin’ ‘China virus.’
• Biden don sign executive order today, e no go dey call am ‘China virus’ again.
• Wah, Biden just sign order today, cannot call ‘China virus’ liao.
• Biden sign one order today, fi stop di use a ‘China virus.’

Response Format:
• Rewritten Tweet 1: [Your first rewritten version]
• Rewritten Tweet 2: [Your second rewritten version]
• . . . (and so on)

Inputs:
Tweet: {claim}
Fact Check: {fact_check}
Generate your response in the specified string format.
Dialect Verification

You are now a fact checker tasked with verifying whether a fact-check is applicable to a list of rewritten tweets. You will receive three inputs:

1. Fact-check: A statement supporting or refuting a claim.

2. Original Tweet: The source tweet conveying the claim.

3. Rewritten Tweets: A list of tweets rewritten based on the original tweet in different English dialects.

Your task: For each rewritten tweet, evaluate:

• Does the fact-check apply to it (i.e., is the fact-check helpful in verifying the rewritten tweet)?

• Does it convey the same main claim as the original tweet?

• Does the rewritten tweet read naturally, as if written by a typical social media user?

Your output: For each rewritten tweet, provide a binary label indicating whether the constraints above are satisfied.

Example:

Fact-check: “President Joe Biden issued an executive order in January 2021 banning the term ‘China virus.’”
Original tweet: “Biden signed an executive order today banning the term ‘China virus’.”
Rewritten tweets:
• Biden done signed an order today, no more sayin’ ‘China virus.’
• Biden don sign executive order today, e no go dey call am ‘China virus’ again.
• Wah, Biden just sign order today, cannot call ‘China virus’ liao.
• Biden sign one order today, fi stop di use a ‘Bejing virus.’
Output:
{
    "labels": [1, 1, 1, 0]
}

Inputs:
Fact Check: {fact_check}
Original Tweet: {claim}
Rewritten Tweets: {rewrites}

Generate your response in the specified JSON format.
Claim Normalization
You will be provided with a noisy input claim from a social media post. The input claim may contain informal language, typos, abbreviations, double negations, and dialectal variations. Your task is to normalise the claim to a more formal and standardised version while preserving the original meaning.
Ensure that:

• The normalised claim conveys the same main claim as the original.

Let’s see an example:
Noisy Claim: “Wah, Biden just sign order today, cannot call ‘China virus’ liao”
Normalised Claim: “President Joe Biden issued an executive order today banning the term ‘China virus.’”
Noisy Claim: “Soros son sez he and dad pickd Harris 4 VP after pic interview!”
Normalised Claim: “George Soros son revealed that he and his father chose Kamala Harris as the Vice President after a picture interview.”
Noisy Claim: “It is not untrue that President-elect Joe Biden’s German shepherd, Major, is set to become the first shelter dog in the White House.”
Normalised Claim: “President-elect Joe Biden’s German shepherd, Major, is set to become the first shelter dog in the White House.”

Response Format:
Normalised Claim: [Your normalised claim]
Inputs:
Noisy Claim: {claim}
Generate your response in the specified string format.