Venue: NAACL 2025

FactEval: Evaluating the Robustness of Fact Verification Systems in the Era of Large Language Models

 Mamta and **Oana Cocarascu**

King's College London

{mamta.name, oana.cocarascu}@kcl.ac.uk

###### Abstract

Whilst large language models (LLMs) have made significant advances in every natural language processing task, studies have shown that these models are vulnerable to small perturbations in the inputs, raising concerns about their robustness in the real-world. Given the rise of misinformation online and its significant impact on society, fact verification is one area in which assessing the robustness of models developed for this task is crucial. However, the robustness of LLMs in fact verification remains largely unexplored. In this paper, we introduce FactEval, a novel large-scale benchmark for extensive evaluation of LLMs in the fact verification domain covering 16 realistic word-level and character-level perturbations and 4 types of subpopulations. We investigate the robustness of several LLMs in zero-shot, few-shot, and chain-of-thought prompting. Our analysis using FEVER, one of the largest and most widely-used datasets for fact verification, reveals that LLMs are brittle to small input changes and also exhibit performance variations across different subpopulations.

## 1 Introduction

Large language models (LLMs) have marked a significant milestone in the natural language processing (NLP) field. However, despite their impressive capabilities across a variety of tasks Cao et al. (2023); Zhang et al. (2024); Li et al. (2024a), it has been shown that these models exhibit vulnerabilities to adversarial samples - subtle alterations to the input that can be easily understood by humans Wang et al. (2023). This has raised concerns about the robustness of these models when deployed in the real world.

In the era where vast amounts of information are generated and shared rapidly, fact verification has become an increasingly important area of research in NLP as it is one of the key methods for detecting misleading and false claims Guo et al. (2022); Akhtar et al. (2023). Given the widespread influence of misinformation, which can have significant political, societal, and economic impact, assessing the robustness of fact verification systems and having a comprehensive understanding of these models' failures becomes crucial.

Numerous efforts have been made to evaluate the robustness of NLP models in a variety of tasks, including sentiment analysis and natural language inference Wang et al. (2022); Mamta and Ekbal (2022); Goyal et al. (2023). While there have been some studies examining adversarial attacks on fact verification models by manipulating either the claims Thorne et al. (2019); Hidey et al. (2020); Atanasova et al. (2020) or the evidence Abdelnabi and Fritz (2023), their scope remains limited to traditional task-specific fine-tuned models.

In this paper, we propose FactEval1, a novel benchmark for evaluating the robustness of LLMs for fact verification. To the best of our knowledge, we are the first to study LLM-based fact verification models; prior studies have been limited to traditional BERT style models.

Footnote 1: Code and data available at [https://github.com/TRAI-group/FactEval](https://github.com/TRAI-group/FactEval)

FactEval evaluates the robustness of fact verification models through _perturbations_ and _subpopulations_ applied to the claims, leaving the evidence unchanged. This allows us to focus on evaluating the model's ability to handle variations in the claims without adding noise to the evidence.

We introduce 16 word-level and character-level perturbations used to create adversarial datasets. We incorporate visual perturbations Boucher et al. (2022); Eger et al. (2019), expanding the scope of attacks on fact verification models. Recognizing that regional accents and the influence of native languages impact the writing style of non-native English speakers in informal conversations Le et al. (2022), we integrate phonetic perturbations for a comprehensive evaluation.

 

Furthermore, we designed subpopulations based on demographic factors, temporal information, sentiment, and named entities. These subpopulations consist of distinct subgroups of data that may differ in specific properties. Subpopulation analysis helps ensure that the model is not only accurate on the overall dataset but also performs consistently across all subgroups. This is not only important for fairness but also crucial for enhancing the overall reliability and effectiveness of the systems.

With FactEval, we examine the robustness of LLM models for fact verification under zero-shot, few-shot, and chain-of-thought prompting. We evaluate three LLMs: Llama3 8B Dubey et al. (2024), Mistral 7B Jiang et al. (2023), and Gemma 7B Mesnard et al. (2024). For comparison, we also investigate the robustness of transformer-based pre-trained language models (PLMs) such as BERT Devlin et al. (2019).

Our experiments on the FEVER dataset, one of the largest and most widely-used benchmarks for fact verification Thorne et al. (2018), reveal that PLMs and LLMs based fact verification systems severely lack robustness against even simple perturbations, raising concerns about the trustworthiness of these systems and emphasizing the need for more robust methodologies. Addressing these challenges is essential for ensuring the reliability and effectiveness of fact verification models in real-world applications.

To summarize, our **contributions** are:

1. We propose FactEval, a novel large-scale benchmark for comprehensive evaluation of the robustness of fact verification systems, covering 16 realistic word-level and character-level perturbations and 4 types of subpopulations.
2. We evaluate the performance of PLMs and LLMs, specifically Llama, Gemma, and Mistral, in zero-shot, few-shot, and chain-of-thought prompting.
3. Our comprehensive evaluation shows that PLMs and LLMs based fact verification systems are not resilient to minor perturbations.

Many perturbations in FactEval can be applied to tasks other than fact verification, making it easy to incorporate in evaluations beyond just reporting standard metrics on held-out data.

## 2 Related Work

### Robustness of NLP models

Despite having achieved great progress on standard benchmarks, NLP models still struggle in the presence of small changes to the input. Several works have shown that high-performing transformer-based models are brittle to adversarial attacks and small perturbations to the inputs Alzantot et al. (2018); Lin et al. (2021); Neerudu et al. (2023). Recent studies on robustness of LLMs investigated out-of-distribution datasets Yuan et al. (2023); Gupta et al. (2024) as well as challenge test sets, behavioral testing, contrast sets, and adversarial inputs Gupta et al. (2024).

Various tasks have been explored in studies on NLP robustness, including sentiment analysis Jin et al. (2020); Kiela et al. (2021); Yuan et al. (2023); Mamta et al. (2023), toxic content detection Li et al. (2019); Yuan et al. (2023), argument mining Sofi et al. (2022); Mayer et al. (2020), machine translation Sai et al. (2021); Wang et al. (2021); Morris et al. (2020), question answering Goel et al. (2021); Moradi and Samwald (2021); Kiela et al. (2021); Yuan et al. (2023); Gupta et al. (2024), natural language inference Wu et al. (2021); Morris et al. (2020); Li et al. (2021); Yuan et al. (2023), and dialogue generation Sai et al. (2020); Li et al. (2023). Perturbations are applied at character-level, word-level, or sentence-level Wang et al. (2022).

Few works created test sets by constructing subpopulations based on various input characteristics Mille et al. (2021); Goel et al. (2021). These sub-populations were defined by features such as topic, entity names, input length, presence of pronouns for a particular gender. allowing for a more granular analysis of model performance across different subsets of data.

### Robustness of Fact Verification Systems

There have been few attempts to assess the robustness of fact verification systems against adversarial attacks. The majority of these works manipulate the claims to create adversarial examples. Niewinski et al. (2019) introduced a generative enhanced model (GEM), a modified and fine-tuned GPT-2 language model used to generate adversarial claims. Other works Thorne et al. (2019); Hidey et al. (2020) evaluated the robustness of the BERT model Devlin et al. (2019). Thorne et al. (2019) evaluated three adversarial attacks by altering the claim and without modifying the evidence. The attacks in 

clude rules-based meaning-preserving transformations to induce classification errors, hand-crafted rules exploiting common patterns and constructions in the FEVER claims and a paraphrase model to generate new instances. Hidey et al. (2020) created an adversarial dataset of 1000 examples that targeted models' weaknesses in (i) multi-hop reasoning by augmenting existing claims with conjunctions or relative clauses, (ii) temporal reasoning by manipulating claims using date-alteration rules, and (iii) named entity ambiguity and lexical variation using the using the lexical substitution method of Alzantot et al. (2018).

Atanasova et al. (2020) evaluated the robustness of the RoBERTa model Liu et al. (2019) using gold evidence from the dataset. They constructed universal adversarial triggers which are n-grams that, when appended to the actual claim, can change the model's prediction from the source to a target class. These trigger tokens were generated using the HotFlip algorithm Ebrahimi et al. (2018) which updates the embeddings of the trigger tokens to minimize the loss for the target class.

Some works Du et al. (2022); Abdelnabi and Fritz (2023) modified the evidence repository to mislead the retrieval system by adding, removing, or altering evidence, and evaluated their attacks on BERT and RoBERTa-based models.

Prior studies on robustness for fact verification have focused on traditional task-specific fine-tuned models (e.g. BERT style models). There is a noticeable gap in the literature regarding the investigation of the robustness of LLMs within the context of fact verification, which we address in this paper.

### Fact Verification

Recent efforts have leveraged LLMs to solve the fact verification task Vykopal et al. (2024). Lee et al. (2021) used GPT-2 Radford et al. (2019) to assess the factuality of the claim based on the perplexity of evidence-conditioned claim generation. Tang et al. (2024) used GPT models to create synthetic training data to enhance the performance of LLMs in fact verification tasks, whereas Li et al. (2024b) proposed a self-sufficient approach to claim verification using prompting instructions across multiple language models. Zeng and Gao (2023) leveraged a consistency mechanism to improve the performance of fact verification models by constructing 3 variants of the original prompt based on three logical relations (confirmation, negation, uncertainty) and fine-tuned the model on these variants. Yao et al. (2023); Ni et al. (2024) leveraged chain-of-thought prompting to effectively verifying complex claims using reasoning steps.

## 3 FactEval

We present FactEval, a framework for evaluating the robustness of fact verification models. Our framework incorporates word-level and character-level perturbations, as well as visual and phonetic perturbations. Furthermore, we design subpopulations of the data based on demographic factors, temporal information, sentiment, and named entities. FactEval contains 16 realistic perturbations and 4 types of subpopulations.

### Task Definition

Given a claim and some evidence, the aim of a fact verification model is to determine whether the evidence supports, refutes, or does not provide enough information to reach a verdict.

To verify the robustness of a fact verification model, we assume no black-box or white-box access to the model. Hence, we modify the input by applying perturbations before passing it to the model. In particular, for a claim \(C\) consisting of \(n\) tokens \(c_{1},c_{2},c_{3},...,c_{n}\), \(m\) pieces of evidence \(E=E_{1},E_{2},..E_{m}\) with label \(y\), the objective is to apply _label-preserving_ perturbations to \(C\) and test whether these mislead the target model \(FV\), i.e., \(FV(C^{\prime},E)\neq y\). Perturbations are applied to the claims, while the evidence remains unchanged in order to evaluate how well the model can handle variations in the claims without introducing noise into the evidence.

### Perturbations

FactEval contains several adversarial datasets obtained using word-level and character-level perturbations. Examples are presented in Table 1.

#### 3.2.1 Word-Level Perturbations

Contractions/Expansions Contractions are words obtained by shortening and combining two words, often using an apostrophe (e.g. _do not \(\rightarrow\) don't_). Expansions are the opposite, where a contraction is written out in its full form (e.g. _don't \(\rightarrow\) do not_). These perturbations help evaluate how well models handle linguistic variations and maintain meaning across different writing styles, such as formal and informal language. For this pertur 

bation, we use a dictionary containing all possible expansions and contractions (Sai et al., 2021).

**Jumbling Word Order** We perturb the claim by randomly changing the order of its words. This perturbation tests how well models handle variations in syntax and word arrangement.

**Numbers to Words** In real-world data, numbers can appear as numerals or words. To evaluate the versatility of a model, we convert all numbers (e.g. "2") to their word equivalents (e.g. "two").

**Repeat Phrases** To test how robust a model is to repetitive patterns, we perturb a sentence by adding the first quarter of the claim to the end of the original claim.

**Subject Verb Disagreement** As grammar errors occur frequently in real-world data, we evaluate the robustness of models in understanding grammatical structure. We follow Sai et al. (2021) and create syntactically incorrect sentences based on subject-verb disagreement (i.e. singular vs plural).

**Typos** We introduce a typographical error (typo) into a claim by swapping adjacent characters. This perturbation simulates realistic typing errors, which is essential for creating more resilient applications capable of handling noisy or imperfect data.

**Word Repetition** We randomly select a word from a sentence and insert it immediately after the selected word.

**Synonym Adjectives** We select the adjectives in a claim and replace them with their synonyms from WordNet (Fellbaum, 1998). The goal is to create semantically equivalent but lexically varied versions of the claims.

**Addition of Tautology** To create this adversarial dataset, we append _and true is true_ three times at the end of the claim.

**Phonetic Perturbations** Non-native English speakers may pronounce words differently as they apply the speech rules of their first language. This may also affect their writing style. Using a dictionary of human-written phonetic perturbations (Le et al., 2022), we introduce phonetic perturbations using 25% and 50% budget values (\(x\%\) budget value means perturbing \(x\%\) words of the claim).

#### 3.2.2 Character-Level Perturbations

**Character Swapping** For this perturbation, we randomly swap adjacent characters within a word.

**Character Repetition** We randomly select a character (except the first/last characters) from a random word in the claim and insert it directly after the selected character. This mimics common typographical errors where a key is accidentally pressed twice in quick succession. This perturbation introduces subtle noise into the text while preserving the readability of the word.

**Character Insertion** We select a random character (except the first/last characters) from a randomly chosen word of at least three characters in the claim. We then insert the caracter into a randomly chosen position within the word. The insertion occurs at any position except the first and last, mimicking typographical errors such as unintentional key presses during typing.

**Character Deletion** We randomly select a character (except the first/last characters) from a random word in the claim and delete it. This simulates common typing errors where characters are accidentally omitted. This perturbation preserves the

\begin{table}
\begin{tabular}{l l p{146.7pt}}  & **Perturbed Claim** \\ Contractions & Oscar Isaac **did not** act in X-Men & Oscar Isaac **didn’t act in X-Men \\ Expansions & Henry Cavillid **didn't** play Superman. & Oscar Casvil **did not** play Superman. \\ Jumbling & Oscar Isaac **did not act in X-Men** & Oscar Isaac **not did act in X-Men \\ Num to Words & Southern Hospitality felt to number **23** on the Top 40. & Southern Hospitality fell to number **twenty-three** on the Top 40. \\ Repeat Phrases & South Island is sometimes referred to as the “main- & South Island is sometimes referred to as the ” main- \\  & land” of New Zealand. & land” of New Zealand. **South Island is sometimes** \\ Subject verb Disagreement & One Dance **wers** by Drake. & One Dance **were** by Drake. \\ Typos & Richard Kuklikski is **a innocent man. & Richard Kuklikski is **aimonect man.** \\ Word Repetition & Trouble with the Curve is a cat & Trouble with the Curve **Curve** is a cat \\ Synonyms & A **good Day** to Die Hard stars Bruce Willis s John McClane & A **nice love** by Ole Hadar Astrose Willis s as John McClane \\ Tautology & Benjamin Franklin was a person. & Benjamin Franklin was a person. **and true is true.** \\  & and **true is true.** & **and true is true.** \\ Character Swap & LinkedIn **is based in Russia.** & Likechin **is based in russa.** \\ Character Repetition & The Burj Kabilita contains zero escalators & The Burj Kabilita contains zero escalators \\ Character Insertion & Trouble with the Curve is a cat . & Trouble with the Curve is a cat . \\ Character Deletion & The Catcher in the Rey is a young adult **novel.** & The Catcher in the Rey is a young adult **novel.** \\ Phonetic & **Spider-Man ’s was written by Donald Trump.** & **Spideman 2**was written by Donald Trump.** \\ Homoglyph & **Annie is the title of a work.** & **Añite is the title of a work.** \\ LEET & **Scandal is a Mexican band.** & **Scandal is a M3kican band.** \\ \hline \end{tabular}
\end{table}
Table 1: Perturbation examples.

 

general structure of the word and sentence, introducing noise in a controlled way.

**Homoglyph Perturbations** Homoglyphs are characters that look similar or identical to other characters (e.g. \(n\) and \(\bar{n}\)). We perturb characters based on the dictionary from Unicode Security.2 We experiment with various perturbation budgets (i.e. k = 25% and 50%) to evaluate the robustness of models against these changes.

Footnote 2: www.unicode.org/Public/security/latest/confusables.txt

**LEET Perturbations** LEET perturbations amount to replacing letters with their visually similar counterparts (i.e. numbers, special characters, or other symbols) and are often used as a distinct writing style online. For example, 'A' can be replaced with '4', 'E' with '3', and 'I' with '1'. We use a pre-defined dictionary Eger et al. (2019); Eger and Benz (2020) to perturb claims and experiment with character perturbation ratios of 25% and 50%.

### Subpopulations

Subpopulations represent distinct subgroups of data. We design four types of subpopulations to ensure that models perform robustly and fairly across diverse groups of data.

**Demographic** We create subpopulations based on _nationality_, _ethnicity_, and _gender_ to analyze a model's behavior across different demographic groups. We identify person entities and retrieve their ethnicity, gender, and nationality from their corresponding Wikipedia pages, if available. If a claim contains multiple nationalities, we include it in all relevant subpopulations.

**Temporal Information** We design temporal subpopulations based on the presence or absence of date entities (i.e. calendar dates, years, months, days, and general time expressions such as "the 90s"). This is useful to assess how well the model handles claims that rely on accurate temporal understanding.

**Sentiment** Similarly to Mamta et al. (2020) where sentences are annotated based on the event described, we create subpopulations related to the claim's sentiment (i.e. _positive_, _negative_, or _neutral_) using VADER Hutto and Gilbert (2014). This allows us to evaluate whether the model correctly classifies claims with positive sentiments more effectively than those with negative or neutral sentiments.

**Named Entities** We design subpopulations related to _person_, _location_, and _organization_ entities to evaluate model performance on claims mentioning persons compared to those mentioning organizations or locations.

## 4 Experiments

### Dataset

For our experiments, we use FEVER, one of the largest and most widely-used benchmarks for fact verification Thorne et al. (2018a, b). As one of the first large-scale datasets for fact verification Guo et al. (2022), FEVER contains 185,000 annotated claims, each accompanied by evidence from Wikipedia. The claims are classified as _supported_, _refuted_ or _not enough information (NEI)_. FEVER provides gold evidence for _supported_ and _refuted_ classes only. We follow Zeng and Gao (2023b) to provide evidence for samples in the _NEI_ class.

Table 2 shows the data distribution in FEVER. We further split the training data into train (80%) and development (20%) sets to fine-tune pretrained language models.

### Models

We experiment with pre-trained language models which can be directly fine-tuned, as well as large language models.

Pre-trained Language Models.We perform task-specific fine-tuning of BERT Devlin et al. (2019) by adding an output layer on top of these models. The input sequence is constructed by separating the claim \(C\) and the evidences \(E\) using a separator token. This input sequence is passed to the model and the final sentence representation is fed into an output layer for the classification task.

Large Language Models.We evaluate Llama3 8B (base model) Dubey et al. (2024), Mistral 7B Instruct Jiang et al. (2023), and Gemma 7B Instruct Mesnard et al. (2024) models using zero-shot, few-shot, and chain-of-thought prompting.

Zero-Shot Prompting Zero-shot prompting relies solely on the pre-trained knowledge and generalization abilities of LLMs. We provide task instructions and class definitions to ensure that LLMs

\begin{table}
\begin{tabular}{l r r r} \hline  & Supported & Refuted & NEI \\ \hline Train & 80,035 & 29,775 & 35,639 \\ Test & 3,333 & 3,333 & 3,333 \\ \hline \end{tabular}
\end{table}
Table 2: Data distribution in FEVER.

 

understand the task at hand. The instructions clarify the expected input (the claim and supporting evidence) and the required answer (classification into one of the predefined classes).

**Few-Shot Prompting** In the few-shot setting, the model is provided with a small set of labeled examples to guide its response, leveraging in-context learning [25]. The model observes a few labeled instances from the dataset and uses these examples to infer the class at inference time. We provide task instructions and randomly select two examples from each class to include in the prompt.3

Footnote 3: The prompts are provided in the Appendix.

Chain-of-Thought PromptingWe use Chain-of-Thought (CoT) along with the task definition in zero-shot setting. We follow Wei et al. (2022) for CoT and add _Think step by step_ at the end of each zero-shot prompt.

## 5 Results and Discussion

To evaluate the robustness of LLMs and PLMs in different attack settings, we compute accuracy, \(F_{1}\), and Attack Success Rate (ASR). Accuracy and \(F_{1}\) are calculated on the FEVER test set as well as the adversarial test sets. \(ASR\) is calculated as the percentage of adversarial examples that can successfully attack the target model.

### Adversarial Robustness

Which model is more/least robust among LLMs?Table 3 shows the results for zero-shot and few-shot learning across Llama, Mistral, and Gemma models. In the zero-shot setting, Gemma outperforms both Llama and Mistral on the FEVER test set, indicating that Gemma has a stronger ability to generalize in the absence of labeled examples. However, all models, despite their good performance on the FEVER test set, are vulnerable to small perturbations in the input data, as indicated by their ASR.

Among the models, Llama is the most vulnerable (i.e. having a higher ASR when exposed to adversarial perturbations). It is interesting to note that even character swapping in the input text significantly affects the performance of all models. While few-shot models perform notably better than zero-shot models in terms of accuracy, they still remain vulnerable to adversarial examples.

In the few-shot setting, Llama has a higher ASR than both Mistral and Gemma in most perturbations. Gemma, on the other hand, is the most robust model across the majority of perturbations. For example, in homoglyph and LEET perturbations, Gemma demonstrates a lower ASR than both Llama and Mistral, showing its resilience against these types of adversarial attacks.

We also investigate whether CoT makes the best two models, Mistral and Gemma, more robust to perturbations. The results in Table 4 indicate that both models remain vulnerable to perturbation in the CoT setting. Gemma has a higher ASR than Mistral for the majority of perturbations, suggesting that Mistral is more resilient to perturbations compared to Gemma in this setting. However, Gemma has lower ASR for LEET perturbations compared to Mistral.

How do LLMs compare to PLMs in terms of robustness?Table 5 shows that BERT performs better compared to LLMs, highlighting the importance of task-specific fine-tuning. However, despite its good performance, BERT remains vulnerable to all types of adversarial perturbations. While BERT shows greater robustness than the Llama zero-shot and few-shot (base model) variants, the instruction-tuned models, Mistral and Gemma, have even higher resilience compared to BERT.

Which perturbations are more challenging?The LLama zero-shot model is highly vulnerable to almost all types of perturbations, with ASR ranging from 13% to 37%. Homoglyph and LEET perturbations have the highest ASR across all zero-shot and few-shot models, indicating that these models are particularly susceptible to such attacks. Additionally, all models show increased vulnerability to perturbations involving typos, tautology, phonetic variations, and character swapping. Zero-shot models, in particular, are more affected by phonetic perturbations than their few-shot models. Interestingly, the few-shot LLama model has a higher ASR in the presenece of tautology perturbations compared to its zero-shot variant. This also indicates that the presence of tautology in in-context learning (ICL) can confuse the Llama model about the patterns it has learned from the few-shot samples, leading to a higher ASR.

### Subpopulations

Table 6 shows the results for different subpopulations in zero-shot, few-shot, and CoT setting for Llama, Mistral, and Gemma. We present the data distributions for subpopulations in Section C.

How do models behave across demographic sub 

populations?** The models exhibit varying performance across different demographic groups, suggesting that models do not treat each group equally, as certain subgroups achieve higher accuracy compared to others. For instance, the zero-shot Llama model has the lowest classification performance on the _Mexican_ subgroup, whereas the few-shot Llama model underperforms on the _Spanish_ subgroup.

**How does temporal information affect the performance of the models?** All models exhibit superior performance when claims do not contain any mention of temporal information. This indicates that the models struggle to interpret and reason about temporal information.

**How does sentiment affect the performance of the models?** The models' performance is significantly influenced by the sentiment of the claims. Claims with positive sentiments achieve higher accuracy compared to those with negative or neutral sentiments across all the models.

**How do models perform on named entities subpopulations?** The results indicate that the models can effectively handle various types of named entities mentioned in claims. Both the zero-shot and few-shot Llama models perform significantly better on claims that include location information compared to those mentioning person and organization entities.

### Qualitative Analysis

Figure 1 shows a few examples for the zero-shot setting where models fail to correctly classify adversarial samples. When the input text is perturbed,

\begin{table}
\begin{tabular}{c c r|r r|r r|r r|r r|r r|r r} \hline  & \multicolumn{2}{l}{**Perturbation**} & \multicolumn{3}{l}{**Llama**} & \multicolumn{4}{l}{**Marked**} & \multicolumn{4}{l}{**Gamma**} \\  & \multicolumn{2}{l}{**Acc**} & \multicolumn{2}{l}{**F1**} & \multicolumn{2}{l}{**ASR**} & \multicolumn{2}{l}{**Acc**} & \multicolumn{2}{l}{**F1**} & \multicolumn{2}{l}{**ASR**} & \multicolumn{2}{l}{**Acc**} & \multicolumn{2}{l}{**F1**} & \multicolumn{2}{l}{**AssR**} & \multicolumn{2}{l}{**Acc**} & \multicolumn{2}{l}{**F1**} & \multicolumn{2}{l}{**AssR**} & \multicolumn{2}{l}{**Acc**} & \multicolumn{2}{l}{**F1**} & \multicolumn{2}{l}{**AssR**} \\ \cline{2-19}  & None & 57.43 & 54.88 & – & 72.73 & 69.08 & — & 77.44 & 47.82 & — & 66.63 & 60.69 & — & 83.82 & 83.18 & - & 61.25 & 950.03 & – \\ \hline Conformations & 47.55 & 42.57 & 24.71 & 70.32 & 67.18 & 63.78 & 57.54 & 72.61 & 2.19 & 62.53 & 55.59 & 4.87 & 81.87 & 81.42 & 2.13 & 49.84 & 65.25 & 7.11 \\ Elepanions & 47.23 & 42.29 & 17.64 & 70.38 & 67.29 & 2.84 & 78.39 & 22.85 & 6.24 & 6.42 & 48.92 & 84.12 & 81.92 & 2.07 & 49.14 & 64.93 & 7.36 \\ Inhaling & 35.71 & 26.60 & **35.78** & 63.13 & 54.77 & **16.50** & 57.17 & **14.66** & 38.60 & **36.84** & **26.81** & 59.83 & 56.1 & **28.69** & 40.05 & **15.88** \\ Number to Westly & 47.21 & 22.27 & 78.22 & 67.08 & 77.20 & 77.47 & 15.99 & 32.53 & 58.51 & 5.59 & 5.41 & 8.18 & 81.07 & 2.49 & 46.14 & 7.5 \\ Repeat Presses & 42.12 & 36.52 & 26.66 & 68.63 & 65.23 & 57.47 & 7.48 & 71.85 & 32.31 & 58.55 & 50.07 & 11.77 & 80.70 & 79.26 & 4.46 & 88.64 & 5.41 & 8.73 \\ SVM & 43.88 & 39.87 & 29.22 & 68.16 & 65.47 & 77.41 & 70.42 & 70.45 & 64.13 & 53.55 & 5.23 & 80.39 & 79.49 & 40.78 & 44.46 & 40.22 \\ Type & 44.22 & 38.86 & 29.69 & 68.37 & 77.53 & 7.21 & 68.16 & 67.22 & 67.03 & 53.59 & 15.90 & 78.38 & 77.22 & 6.53 & 4.89 & **45.91** & **81.2** \\ Word Repetition & 42.21 & 36.96 & 21.91 & 68.13 & 65.33 & 7.26 & 66.27 & 5.99 & 53.34 & 57.96 & 78.18 & 77.63 & 6.53 & 47.97 & 49.66 & **8.03** \\ Synonyms & 46.82 & 47.78 & 14.88 & 68.72 & 66.72 & 66.5 & 37.66 & 74.26 & 71.16 & 36.48 & 61.19 & 56.43 & 4.66 & 81.22 & 50.46 & 2.91 & 4.99 & 45.91 & 8.12 \\ Tanators & 47.73 & 40.00 & 22.11 & 68.11 & 69.13 & 67.14 & 70.43 & 74.08 & 44.13 & 53.48 & **31.75** & 77.51 & 76.23 & 35.08 & 66.56 & 55.98 & 7.48 \\ Proteinic (0.25) & 42.61 & 36.59 & 21.66 & 68.23 & 7.83 & 7.81 & 74.06 & 69.99 & 72.85 & 61.34 & 55.65 & 56.90 & 79.13 & 4.93 & 57.69 & 5.52 & 5.16 \\ Promotion (0.50) & 39.64 & 32.45 & **30.96** & 64.05 & 64.03 & 73.08 & 64.59 & **30.99** & 60.08 & 63.53 & 53.77 & 7.80 & 7.33 & 50.95 & 55.86 & 53.41 & 6.99 \\ Client Swing & 39.05 & 31.57 & 31.97 & 68.68 & 61.05 & 79.02 & 64.02 & 67.59 & 47.97 & 38.77 & 82.86 & 74.33 & **72.97** & **67.37** & 35.01 & 7.96 & 35.01 & 7.96 \\ ChrRepetition & 42.84 & 37.18 & 25.46 & 62.46 & 64.55 & 61.32 & 67.12 & 69.42 & 55.90 & 52.54 & 52.10 & 1.09 & 76.92 & 78.92 & 4.82 & 57.98 & 55.41 & 5.33 \\ Char Insertion & 42.27 & 36.44 & 39.69 & 68.64 & 62.74 & 67.32 & 69.02 & 55.91 & 53.71 & 51.53 & 41.58 & 77.47 & 67.77 & 68.67 & 5.72 & 54.87 & 5.75 \\ Chart Deletion & 42.67 & 36.91 & 25.76 & 63.91 & 52.76 & 63.13 & 77.28 & 69.82 & 58.73 & 50.53 & 12.35 & 78.21 & 77.44 & 69.67 & 51.84 & 54.37 & 6.63 \\ Bottomfly (0.25) & 49.75 & 45.76 & 13.37 & 6.68 & 61.11 & 70.12 & 71.12 & 71.06 & 70.17 & 53.56 & 49.69 & 16.62 & 76.22 & 50.80 & 9.06 & 36.53 & 53.77 & 8.8 \\ Randomfly (0.50) & 47.53 & 47.73 & 17.24 & 68.58 & 57.72 & **13.66** & 66.06 & **14.01** & 51.88 & 41.79 & 23.18 & 69.65 & 67.68 & **16.96** & 54.24 & 51.69 & **11.44** \\ LEET (0.25) & 41.85 & 36.07 & 27.13 & 65.88 & 41.58 & **35.94** & 53.59 & **34.28** & 39.82 & 28.82 & 40.26 & 31.7 

the models often misclassify supported/refuted classes as NEI (first 3 rows), suggesting that the models struggle to maintain their understanding of the original input when we apply subtle changes. Gemma and Mistral are more robust compared to Llama - example 5 illustrates a case where Gemma and Mistral models show resilience to homoglyph perturbations as opposed to Llama.

Figure 2 shows a few examples of character swapping perturbations for few-shot setting. We can see that (i) Llama and Mistral are fooled by character swaps, while Gemma is robust (examples 1 and 5); (ii) all models are misled by characterswap perturbations (examples 2 and 4); (iii) only Mistral is robust (example 3).

### Adversarial Training

To investigate the impact of adversarial training on PLMs and LLMs, we conduct several experiments on BERT and Mistral using the perturbations with a high success rate. For BERT model, we generate adversarial data by applying perturbations to the training data. The adversarial data is then combined with the original training data, and the model is re-trained on the augmented dataset. Table 7 shows the results for BERT. We observe that adversarial training improves the robustness of the BERT model. For example, in the case of LEET, phonetic, and homoglyph perturbations, there is a significant improvement compared to other perturbations.

Similarly, for the Mistral model, we incorporate adversarial examples for each selected perturbation in the prompts, along with their corresponding examples. A total of 12 adversarial examples were added to the few-shot prompt, consisting of 6 original examples and 6 adversarial examples. Table 8 shows the results for Mistral. It can be seen that the addition of adversarial prompts increases the robustness of the Mistral model against only a few of the perturbations (highlighted). There is an increase in ASR for the remaining perturbations, which can be attributed to the added noise in the prompts.

## 6 Conclusion

We introduced FactEval, a novel large-scale benchmark designed to evaluate the robustness of large language models in the fact verification do 

main. We developed 16 word-level and character-level perturbations, along with 4 types of subpopulations. Our comprehensive evaluation covered zero-shot, few-shot, and chain of thought prompting methods across Llama, Mistral, and Gemma models. Our analysis revealed that LLMs are not robust to these perturbations, with the Llama model being the least robust compared to Mistral and Gemma. Furthermore, our experiments show that these models show varying performance on subpopulations such as demographic groups, sentiment, entity types, and temporal information. This indicates that more work needs to be done to improve the robustness of the models.

In future work, we plan to extend our framework to other monolingual and code-mixed languages, allowing for a more comprehensive evaluation of robustness across diverse linguistic contexts. We also aim to explore providing explanations for model failures through model explainability techniques, which can offer valuable insights into the model's behavior and help identify areas for improvement.

### Limitations

Like many studies, this research has certain limitations that future investigations could address. Specifically, our current work is focused exclusively on the English language and textual inputs. Multi-lingual and multi-modal data ought to be also investigated. In addition, we have assumed no white-box or black-box access to the model, and thus, adversarial samples were generated by randomly selecting words in the claim. This approach may have perturbed words that are not critical to the classification task. In future work, this limitation can be addressed by assuming black-box access and applying perturbations only to words which are crucial for the model's decision-making. Moreover, all subpopulation subsets are derived from a single, existing dataset, limiting the range of subpopulations we could analyze. Some sub-populations (demographic) with a small number of samples were excluded from this analysis.

## Ethics Statement

We utilize publicly available datasets for our experiments. These datasets are used solely for academic purposes and in full compliance with their licensing agreements.

## Acknowledgement

This research was supported by EPSRC (grant number EP/X04162X/1).

## References

* A. Abdelnabi and M. Fritz (2023)Fact-subdeeurs: a taxonomy of evidence manipulation attacks against fact-verification systems. In 32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA, August 9-11, 2023, pp. 6719-6736. External Links: Link Cited by: SS1.
* M. Akhtar, M. Schlichtkrull, Z. Guo, O. Cocarascu, E. Simperl, and A. Vlachos (2023)Multimodal automated fact-checking: a survey. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, pp. 5430-5448. External Links: Link Cited by: SS1.
* M. Alzantot, Y. Sharma, A. Elgohary, B. Ho, M. Srivastava, and K. Chang (2018)Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 2890-2896. External Links: Link Cited by: SS1.

\begin{table}
\begin{tabular}{c l c r r r r} \hline  & **Acc** & **F1** & **ASR** & **w/o Adv. Acc. TS** \\ \hline  & Phonetic (0.25) & 78.99 & 78.76 & **1.589** & 22.11 & 93.93 \\  & Phonetic (0.50) & 78.56 & 78.23 & **16.35** & 28.48 & 93.12 \\ \cline{2-7}  & Char swapping & 71.85 & 70.64 & 23.56 & 23.47 & 93.41 \\  & Charlie Repetition & 76.32 & 75.93 & **18.74** & 19.63 & 93.56 \\  & Char Insertion & 76.09 & 75.71 & **18.99** & 19.77 & 93.48 \\  & Char Deletion & 76.07 & 75.22 & **19.01** & 19.73 & 93.89 \\  & Homoglyph (0.25) & 78.65 & 78.33 & **16.04** & 19.6 & 93.69 \\  & Homoglyph (0.50) & 75.93 & 75.92 & **18.95** & 24.61 & 93.7 \\  & LEFT (0.25) & 78.96 & 78.68 & **15.1** & 45.03 & 93.27 \\  & LEET (0.50) & 77.12 & 76.76 & **17.08** & 48.24 & 93.01 \\ \hline \end{tabular}
\end{table}
Table 7: Results on adversarial perturbations using BERT after adversarial training on augmented data. Here w/o Adv is ASR without adversarial training, TS: accuracy on actual test set.

\begin{table}
\begin{tabular}{c l r r r r r} \hline
**Perturbation** & & \multicolumn{3}{c}{**Mistral**} & \multicolumn{3}{c}{**w/o Adv. Acc. TS**} \\
**Acc** & **F1** & **ASR** & **ASR** & & \\ \hline Contractions & 80.71 & 80.11 & 3.71 & **2.13** & 87.4 \\ Expansions & 80.67 & 80 & 3.76 & **2.07** & 87.46 \\ Jurnbling & 69.01 & 67.34 & **17.67** & 28.59 & 85.21 \\ Number to Words & 80.45 & 79.84 & 4.02 & **2.21** & 87.35 \\ Repeat Phrases & 79.5 & 79.01 & 5.14 & **4.46** & 87.44 \\ SVD & 80.22 & 79.63 & 4.3 & **4.01** & 87.73 \\ Types + & 77.98 & 77.43 & 6.95 & **6.53** & 87.03 \\ Word Rejection & 78.18 & 77.82 & 6.72 & **5.01** & 86.92 \\ Synonyms & 79.89 & 79.28 & 4.68 & **2.91** & 87.37 \\ Trandology & 74.55 & 73.24 & 10.66 & **7.33** & 77.41 \\ Phonetic (0.25) & 77 & 76.5 & 8.14 & **4.96** & 82.55 \\ Phonetics (0.50) & 76.93 & 75.76 & 8.19 & **5.93** & 82.67 \\ \hline Chair Swapping & 76.34 & 77.56 & 8.92 & **10.77** & 82.88 \\ CharRepetition & 76.46 & 76.17 & 8.78 & **4.82** & 85.1 \\ Char Insertion & 74.25 & 74.11 & 11.42 & **6.18** & 83.18 \\ Char Deletion & 74.64 & 74.42 & 10.95 & **6.49** & 82.95 \\ Homeoglyph (0.25) & 76.82 & 75.79 & **8.35** & 9.06 & 86.64 \\ Honoglyph (0.50) & 70.88 & 69.2 & **15.43** & 16.9 & 85.55 \\ LEET (0.25) & 64.18 & 63.44 & **23.42** & 38.29 & 86.6 \\ LEET (0.50) & 51.08 & 51.08 & **39.05** & 48.72 & 85.74 \\ \hline \end{tabular}
\end{table}
Table 8: Results on adversarial perturbations for Mistral after adding adversarial prompts.

 

Pepa Atanasova, Dustin Wright, and Isabelle Augenstein. 2020. Generating label cohesive and well-formed adversarial claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 3168-3177. External Links: Link Cited by: SS4.
* N. Boucher, I. Shumailov, R. Anderson, and N. Papernot (2022)Bad characters: imperceptible nlp attacks. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1987-2004. External Links: ISBN 978-1-5035-381X, Document, ISSN 1573-0070 Cited by: SS1.
* H. Cao, L. Wei, M. Chen, W. Zhou, and S. Hu (2023)Are large language models good fact checkers: a preliminary study. arXiv preprint arXiv:2311.17355. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS1.
* March 1, 2022, pp. 10581-10589. External Links: Link Cited by: SS1.
* A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Savankumar, A. Korenev, A. Hinsevark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Brion, B. Tang, B. Chern, C. Caucheteux, C. Nayak, Chloe Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. Canton Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, Danny Livshits, D. Esibou, D. Croudblary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hughes, E. Lakomkin, E. AlgBadawy, E. Lobanova, E. Dinan, E. Michael Smith, F. Radenovic, F. Zhang, G. Synnaeve, G. Lee, G. Lewis Anderson, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Youro, I. Zarov, I. Artieta Ibarra, I. M. Klumann, I. Misra, I. Evtimov, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bilton, J. Spisak, J. Park, J. Rosca, J. Johnstun, J. Saxe, J. Jia, K. Alwala, K. Upasani, K. Plawiak, K. Li, K. Stone, and et al. (2024)The llama 3 herd of models. CoRRabs/2407.21783. External Links: Link Cited by: SS1.
* J. Ebrahimi, A. Rao, D. Lowd, and D. Dou (2018)HotFlip: white-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia, pp. 31-36. External Links: Link Cited by: SS1.
* S. Eger and Y. Benz (2020)From hero to zeroe: a benchmark of low-level adversarial attacks. In Proceedings of the 1st conference of the Asia-Pacific chapter of the association for computational linguistics and the 10th international joint conference on natural language processing, pp. 786-803. External Links: Link Cited by: SS1.
* S. Eger, G. Gul Sahin, A. Ruckle, J. Lee, C. Schulz, M. Mesgar, K. Swarnkar, Edwin Simpson, and I. Gurevych (2019)Text processing like humans do: visually attacking and shielding NLP systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 1634-1647. External Links: Link Cited by: SS1.
* C. Fellbaum (1998)Wordnet: an electronic lexical database. MIT Press google schola2, pp. 678-686. External Links: Link Cited by: SS1.
* K. Goel, N. Fatema Rajani, J. Vig, Z. Taschdijian, M. Bansal, and C. Re (2021)Robustness gym: unifying the NLP evaluation landscape. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, Online, pp. 42-55. External Links: Link Cited by: SS1.
* S. Goyal, S. Dodolapaneni, M. M. Khapra, and B. Ravindran (2023)A survey of adversarial defenses and robustness in NLP. ACM Comput. Surv.55 (14s), pp. 332:1-332:39. External Links: Link, Document Cited by: SS1.
* Z. Guo, M. Schlichtkrull, and A. Vlachos (2022)A survey on automated fact-checking. Transactions of the Association for Computational Linguistics10, pp. 178-206. External Links: Link, Document Cited by: SS1.
* A. Gupta, R. Rajendharn, N. Stringham, V. Srikumar, and A. Marasovic (2024)Whispers of doubt amidst echoes of triumbnl in NLP robustness. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico City, Mexico, pp. 5533-5590. External Links: Link, Document Cited by: SS1.

 

Christopher Hidey, Tuhin Chakrabarty, Tariq Alhindi, Siddharth Varia, Kriste Krstovski, Mona Diab, and Smaranda Muresan. 2020. DeSePtion: Dual sequence prediction and adversarial examples for improved fact-checking. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8593-8606, Online. Association for Computational Linguistics.
* Hutto and Gilbert (2014) Clayton Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In _Proceedings of the international AAAI conference on web and social media_, volume 8, pages 216-225.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lelio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavill, Thomas Wang, Timothe Lacroix, and William El Sayed. 2023. Mustal 7b. _CoRR_, abs/2310.06825.
* Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In _Proceedings of the AAAI conference on artificial intelligence_, volume 34, pages 8018-8025.
* Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4110-4124, Online. Association for Computational Linguistics.
* Le et al. (2022) Thai Le, Joyoung Lee, Kevin Yen, Yifan Hu, and Dongwon Lee. 2022. Perturbations in the wild: Leveraging human-written text perturbations for realistic adversarial attack and defense. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 2953-2965, Dublin, Ireland. Association for Computational Linguistics.
* Lee et al. (2021) Nayeon Lee, Yejin Bang, Andrea Madotto, and Pascale Fung. 2021. Towards few-shot fact-checking via perplexity. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1971-1981, Online. Association for Computational Linguistics.
* Li et al. (2024a) Chen Li, Meishan Zhang, Xuebo Liu, Zhaocong Li, Derek Wong, and Min Zhang. 2024a. Towards demonstration-aware large language models for machine translation. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 13868-13881.
* Li et al. (2021) Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. 2021. Contextualized perturbation for textual adversarial attack. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5053-5069, Online. Association for Computational Linguistics.
* Li et al. (2019) Jingfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2019. Textbigger: Generating adversarial text against real-world applications. In _26th Annual Network and Distributed System Security Symposium, NDSS 2019, San Diego, California, USA, February 24-27, 2019_. The Internet Society.
* Li et al. (2024b) Miaoran Li, Baolin Peng, Michel Galley, Jianfeng Gao, and Zhu Zhang. 2024b. Self-checker: Plug-and-play modules for fact-checking with large language models. In _Findings of the Association for Computational Linguistics: NAACL 2024_, pages 163-181, Mexico City, Mexico. Association for Computational Linguistics.
* Li et al. (2023) Yufei Li, Zexin Li, Yingfan Gao, and Cong Liu. 2023. White-box multi-objective adversarial attack on dialogue generation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1778-1792, Toronto, Canada. Association for Computational Linguistics.
* Lin et al. (2021) Bill Yuchen Lin, Wenyang Gao, Jun Yan, Ryan Moreno, and Xiang Ren. 2021. RockNER: A simple method to create adversarial examples for evaluating the robustness of named entity recognition models. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 3728-3737, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _CoRR_, abs/1907.11692.
* Mamta and Ekbal (2022) Mamta and Asif Ekbal. 2022. Adversarial sample generation for aspect based sentiment classification. In _Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022_, pages 478-492, Online only. Association for Computational Linguistics.
* Mamta et al. (2020) Mamta, Asif Ekbal, Pushpak Bhattacharyya, Shikha Srivastava, Alka Kumar, and Tista Saha. 2020. Multidomain tweet corpora for sentiment analysis: Resource creation and evaluation. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 5046-5054, Marseille, France. European Language Resources Association.
* Mamta et al. (2023) Mamta Mamta, Zishan Ahmad, and Asif Ekbal. 2023. Elevating code-mixed text handling through auditory information of words. In _Proceedings of the 2023 

Conference on Empirical Methods in Natural Language Processing_, pages 15918-15932, Singapore. Association for Computational Linguistics.
* Mayer et al. (2020) Tobias Mayer, Santiago Marro, Elena Cabrio, and Serena Villata. 2020. Generating adversarial examples for topic-dependent argument classification 1. In _Computational Models of Argument_, pages 33-44. IOS Press.
* Mesnard et al. (2020) Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Riviere, Mihir Sanjay Kale, Juliette Love, Pouyo Tafti, Leonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amelie Heliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clement Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Cristian Muraru, Grigory Rozhodestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishenenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, and et al. 2024. Gemma: Open models based on gemini research and technology. _CoRR_, abs/2403.08295.
* Mille et al. (2021) Simon Mille, Kaustubh D. Dhole, Saad Mahmood, Laura Perez-Beltrachini, Varun Gangal, Mihir Sanjay Kale, Emiel van Miltenburg, and Sebastian Gehrmann. 2021. Automatic construction of evaluation suites for natural language generation datasets. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks_.
* Moradi and Samwald (2021) Milad Moradi and Matthias Samwald. 2021. Evaluating the robustness of neural language models to input perturbations. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 1558-1570, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Morris et al. (2020) John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsysb, Di Jin, and Yanyjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 119-126.
* Neerudu et al. (2023) Pavan Kadyan Reddy Neerudu, Subba Oota, Mounika Marreddy, Venkateswara Kagita, and Manish Gupta. 2023. On robustness of finetuned transformer-based NLP models. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 7180-7195, Singapore. Association for Computational Linguistics.
* Ni et al. (2024) Jingwei Ni, Minjing Shi, Dominik Stammbach, Mrinmaya Sachan, Elliott Ash, and Markus Leippold. 2024. AFaCTA: Assisting the annotation of factual claim detection with reliable LLM annotators. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1890-1912, Bangok, Thailand. Association for Computational Linguistics.
* Niewinski et al. (2019) Piotr Niewinski, Maria Pszona, and Maria Janicka. 2019. GEM: Generative enhanced model for adversarial attacks. In _Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)_, pages 20-26, Hong Kong, China. Association for Computational Linguistics.
* Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, D. Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners.
* Bai et al. (2021a) Ananya B Bai, Tanay Dixit, Dev Yashpal Sheth, Breyes Mohan, and Mitesh M Khapra. 2021a. Perturbation checklists for evaluating nlg evaluation metrics. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7219-7234, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Sai et al. (2020) Ananya B Sai, Akash Kumar Mohankumar, Siddhartha Moran, and Mitesh M Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. _Transactions of the Association for Computational Linguistics_, 8:810-827.
* Sofi et al. (2022) Mehmet Sofi, Matteo Fortier, and Oana Cocarascu. 2022. A robustness evaluation framework for argument mining. In _Proceedings of the 9th Workshop on Argument Mining_, pages 171-180.
* Tang et al. (2024) Liyan Tang, Philippe Laban, and Greg Durrett. 2024. Minicheck: Efficient fact-checking of llms on grounding documents. _ArXiv_, abs/2404.10774.
* Thorne et al. (2018a) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a. FEVER: a large-scale dataset for fact extraction and VERification. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
* Thorne et al. (2019) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Evaluating adversarial attacks against multiple fact verification systems. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing 

(EMNLP-IJCNLP)_, pages 2944-2953, Hong Kong, China. Association for Computational Linguistics.
* Thorne et al. (2018b) James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b. The fact extraction and VERification (FEVER) shared task. In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_, pages 1-9, Brussels, Belgium. Association for Computational Linguistics.
* Vykopal et al. (2024) Ivan Vykopal, Matus Pikuliak, Simon Ostermann, and Marian Simko. 2024. Generative large language models in automated fact-checking: A survey. _arXiv preprint arXiv:2407.02351_.
* Wang et al. (2023) Jiongxiao Wang, Zichen Liu, Keun Hee Park, Muhao Chen, and Chaowei Xiao. 2023. Adversarial demonstration attacks on large language models. _CoRR_, abs/2305.14950.
* Wang et al. (2021) Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, et al. 2021. Textflint: Unified multilingual robustness evaluation toolkit for natural language processing. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_, pages 347-355.
* Wang et al. (2022) Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022. Measure and improve robustness in NLP models: A survey. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4569-4586, Seattle, United States. Association for Computational Linguistics.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Wolf (2019) T Wolf. 2019. Huggingface's transformers: State-of-the-art natural language processing. _arXiv preprint arXiv:1910.03771_.
* Wu et al. (2021) Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. 2021. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6707-6723.
* Xu et al. (2017) Guangxu Xu, Xiaowei Jia, Vishrawas Gopalakrishnan, and Aidong Zhang. 2017. A survey on context learning. _IEEE Trans. Knowl. Data Eng._, 29(1):38-56.
* Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
* 16, 2023_.
* Zeng and Gao (2023a) Fengzhu Zeng and Wei Gao. 2023a. Prompt to be consistent is better than self-consistent? few-shot and zero-shot fact verification with pre-trained language models. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 4555-4569, Toronto, Canada. Association for Computational Linguistics.
* Zeng and Gao (2023b) Fengzhu Zeng and Wei Gao. 2023b. Prompt to be consistent is better than self-consistent? few-shot and zero-shot fact verification with pre-trained language models. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 4555-4569.
* Zhang et al. (2024) Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2024. Benchmarking large language models for news summarization. _Transactions of the Association for Computational Linguistics_, 12:39-57.

## Appendix A Implementation Details

All models were implemented using PyTorch and HuggingFace's [2] for Llama, Mistral, Gehma, and BERT models. The BERT-base model has 12 transformer layers, a hidden size of 768, and 12 self-attention heads, with a total of 110 million trainable parameters. We optimized the BERT model using the Adam optimizer, with weight updates computed based on categorical cross-entropy loss. All computations were performed on an NVIDIA A100-SXM4 GPU with 40 GB of memory. Named entities are extracted using spaSCy Named Entity Recognition.4
Footnote 4: [https://spacy.io/api/entityrecognizer](https://spacy.io/api/entityrecognizer)

## Appendix B Prompts

The zero-shot and few-shot prompts are presented in Figure 3.

## Appendix C Subpopulations

We include the dataset distribution across each subpopulation in Table 9. Our findings from Section 

5.2 indicate that the performance of LLMs models on various subpopulations is influenced by the pre-training dataset used for each model. Since we are not fine-tuning these models on specific fact-checking datasets such as FEVER, their performance is impacted by the biases and composition of their pre-trained corpora, which can vary across models. For example, the sensitivity to sentiment or demographic factors in these models may be more pronounced due to the representation (or lack thereof) of certain sentiments or demographic groups in their pre-training data.

[TABLE:S4.T9][ENDTABLE:S10.T8][ENDTABLE:S30.T7][ENDTABLE:S40.T6][ENDTABLE:S50.T9][ENDTABLE:S50.T10][ENDTABLE:S50.T11][ENDTABLE:S50.T12][ENDTABLE:S50.T13][ENDTABLE:S50.T14][ENDTABLE:S50.T15][ENDTABLE:S50.T16][ENDTABLE:S50.T17][ENDTABLE:S50.T18][ENDTABLE:S50.T19][ENDTABLE:S50.