Venue: NAACL 2025

# Is your benchmark _truly_ adversarial?

AdvScore: Evaluating Human-Grounded Adversarialness

Yoo Yeon Sung\({}^{1}\), Maharshi Gor\({}^{1}\), Eve Fleisig\({}^{2}\), Ishani Mondal\({}^{1}\), Jordan Boyd-Graber\({}^{1}\)

###### Abstract

Adversarial datasets should validate AI robustness by providing samples on which humans perform well, but models do not. However, as models evolve, datasets can become obsolete. Measuring whether a dataset remains adversarial is hindered by the lack of a standardized metric for measuring adversarialness. We propose AdvScore, a human-grounded evaluation metric that assesses a dataset's adversarialness by capturing models' and humans' varying abilities, while also identifying poor examples. We then use AdvScore to motivate a new dataset creation pipeline for realistic and high-quality adversarial samples, enabling us to collect an adversarial question answering (OA) dataset, AdvQA. We apply AdvScore using 9,347 human responses and ten language models' predictions to track model improvement over five years (2020-2024). AdvScore thus provides guidance for achieving robustness comparable with human capabilities. Furthermore, it helps determine to what extent adversarial datasets continue to pose challenges, ensuring that, rather than reflecting outdated or overly artificial difficulties, they effectively test model capabilities.1

Footnote 1: Code and data available here: github.com/yysung/Advscore

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 623–642

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction: Evaluating Adversarial Datasets Requires Human Answers

As language models attain near-perfect performance on existing benchmarks, there is an increasing demand for unexpected and challenging tasks to evaluate them. _Adversarial datasets_ contain examples that cause models to generate harmful Perez et al. (2022), unsafe Quaye et al. (2024), or incorrect Goodfellow et al. (2015) responses. An ideal adversarial example should be much easier for a human to answer correctly than for a model on realistic tasks Ilyas et al. (2019); Tsipras et al. (2019); Engstrom et al. (2020); Biggio et al. (2012). However, as models improve, these adversarial datasets can become outdated Kiela et al. (2021)--what was hard for a model in 2020 can become trivial in five years--requiring periodic updates Recht et al. (2019); Bowman and Dahl (2021). On the other hand, it is difficult to recognize at what point have these adversarial datasets outlived their usefulness systematically, nor is there an established metric to measure which datasets best captures the gap between human and model ability.

To fill this gap, we formulate AdvScore (SS 3). This metric measures two critical aspects: **(i) adversarialness**, which captures the performance gap between models and humans, while penalizing "illposed" examples (i.e., ambiguity), and **(ii) discriminability**--how effectively can a dataset rank models by their abilities.

Measuring whether a dataset is truly adversarial requires human answers; thus, AdvScore builds on item response theory Lalor et al. (2016), irt), a framework widely used in psychometrics and educational testing. It captures the diversity of human and model abilities and identifies poor examples (SS 2). AdvScore is the first metric that evaluates an example's "adversarialness" grounded in human abilities: it can measure whether the dataset's adversarial challenge becomes weaker or stronger as language models improve.

We apply AdvScore to motivate authors to contribute to a new human-in-the-loop hitl benchmark of adversarial questions, AdvQA. AdvQA's creation pipeline (Figure 1) produces _high-quality_ and _realistic_ questions that are adversarial. Moreover, AdvScore helps make AdvQA discriminative, ensuring that the captured adversarialness reflects the varying skills of humans and models.

AdvQA exhibits the least decline in adversarialness over recent years compared to other adversarial benchmarks (SS 4). This minimal, but meaningful decline in AdvQA reveals that current models (e.g., GPT4) continue to struggle with tasks requiring _commonsense reasoning_ and _multistep reason

ing_ and on topics such as _Lifestyle_ (SS 6), which are likely tied to real-world challenges.

We conclude with an analysis of how model have improved improve over the years since researchers began releasing adversarial datasets and how that can inform the development of future adversarial datasets (SS 4).

## 2 Preliminaries of AdvScore: irt

Prior metrics for evaluating adversarial question generation strategies, such as attack success rate [15], distributional similarity [30], and proximity measurement [9] assess algorithmic adversarialness without human validation. In contrast, we identify adversarial examples that pose realistic challenges aligned with _human_ skills, not just pathological cases that break models. This requires evaluating how well the examples align with varying levels of human performance, particularly where models fall short, while ensuring that the examples are unambiguous. To capture this, we adopt item response theory (irt), which models the interactions between subjects' skills--in the QA setting, the subject answering the question could be either a human or a model--and example difficulty. This framework, widely used in psychometrics and educational testing [8], provides insights beyond accuracy: it can diagnose question quality as well as skilled subjects.

2pl-irtIn question answering (qa) tasks, irt models the probability that a subject correctly answers a question based on their skill and question difficulties. 2pl-irt (Eq. 1) models the probability of getting a question correct as a function of subject _skill_\(\beta_{i}\) and question _difficulty_\(\theta_{j}\):

\[p(r_{ij}=1\,|\,\beta_{i},\theta_{j},\gamma_{j})=\sigma(\gamma_{ j}(\underbrace{\beta_{i}-\theta_{j}}_{\text{skill gap}}), \tag{1}\]

where \(\sigma\) is the sigmoid function [7]. The skill gap, \((\beta_{i}-\theta_{j})\), is the difference between the subject \(i\)'s skill and question \(j\). When a subject's skill is _equal_ to the question's difficulty \((\beta_{i}=\theta_{j})\), they have a 50% probability of answering it correctly. Thus, an agent with skill equal to or greater than the question's difficulty level has at least a 50% chance of answering correctly.

The final latent variable is the question _discriminability_\(\gamma_{j}\) which models how sensitive this probability is to changes in skill gap.2 This encodes how strongly the question rewards the skill being higher or lower than the difficulty level. The objective of irt is to estimate the parameters that maximize the correctness probability \(p(r_{ij})\).3

Footnote 2: Perfect discriminability means that any subjects with a positive skill gap will answer the question correctly [10] but negative skill gap will never answer the question correctly.

Advantages of irt over question success rate While question success rate (qsr)--the percentage of subjects answering a question correctly--may seem like a reliable measure of difficulty, it can be misleading. A good yet difficult question and an easy yet poorly written question could yield the same qsr, obscuring the true measure of difficulty.

In contrast, irt evaluates subject responses. Not only does irt consider the number of humans who answer a question correctly, but it also accounts for

Figure 1: AdvScore diagnoses when a question is adversarial (top) and difficult for computers to answer for other reasons (bottom). After collecting candidate questions, we ask humans and computers to answer the questions. The top question (from AdvQA) has a higher AdvScore because it is specific, adversarial, discriminative, high-quality, and realistic. In contrast, the bottom question is ambiguous (e.g., none of humans or models correctly answered due to its ambiguity), which is confirmed by its low AdvScore.

 

_who_ answer _which questions_. If the probability of answering a question correctly increases with subject skill, this relationship will naturally correlate with skill \(\beta_{i}\) and question discriminability \(\gamma_{j}\). The model can confidently assign higher probabilities for these questions, while questions that are answered correctly by luck--rather than skill--will have estimated probabilities closer to 0.5, reflecting their lower discriminability.

Consider three questions: \(q_{\text{ambig}}\) (ambiguous question: "What is a capital of Georgia?" Answer: [Atlanta or Tbilisil]), \(q_{\text{hard}}\) (hard but well-formed question: "Who founded Tbilisi?"), and \(q_{\text{easy}}\) (easy question: "What U.S. state has Atlanta as its capital?"). Comparable qsr values may suggest \(q_{\text{ambig}}\) and \(q_{\text{hard}}\) have the same difficulty. However, irt distinguishes them: \(q_{\text{ambig}}\) has low discriminability (\(\gamma_{j}\approx 0\)), resulting in a low \(p(r_{ij})\) close to 0.5 regardless of the subject skill, while \(q_{\text{hard}}\) and \(q_{\text{easy}}\) are likely to have high discriminability (\(\gamma_{j}\approx 1\)) and reverse difficulty (\(\theta_{j}\)) values. irt thus provides a more nuanced evaluation of question adversarialness, capturing its appropriate challenge levels for humans and models while accounting for its "well-posedness" (SS 3.1).4

Footnote 4: Feasibility, another latent variable in irt, also reflects poor-quality questions when a large proportion of participants answer incorrectly Rodriguez et al. (2021). However, our approach explicitly accounts for disagreement among highly skilled human subjects (§ 3.1). We leave feasibility analysis to future work.

## 3 AdvScore

This section introduces AdvScore, a metric that evaluates how _adversarial_ and _discriminative_ a dataset is. We measure these two key criteria: **(i) adversarialness**, how much more challenging a question is for ai models compared to humans while being well-posed; and **(ii) discriminability**, how informative is the question in effectively distinguishing between different skill levels.

### Quantifying Adversarialness

A question is adversarial if _skilled_ humans consistently answer a question correctly but computers do not. We measure this gap by fitting irt parameters and then computing the probabilities predicted by the trained 2pl-irtt model (§ 2). During margin computation, we conduct synthetic groups for both human and computer subjects with representative skill levels. Then, we compute the probability of each group correctly answering the question, as estimated by the irt model, which accounts for question quality. A question is considered adversarial if the human representative has a higher probability of answering correctly than the computer representative.

Skilled Groups.We first define what constitutes a _skilled_ group \(g\), and further define its _representative skill_\(\beta^{g}_{*}\), which we use in subsequent equations (3,5). For a set of randomly sampled subjects \(S\), skilled group \(S_{(k)}\) is the subset of subjects with skill at least \(k\) standard deviations above the mean--\(\beta_{i}>\mu^{S}_{\beta}+kr^{S}_{\beta}\)--where \(\mu^{S}_{\beta}\) and \(\tau^{S}_{\beta}\) are the mean and standard deviation of subject skills over the set \(S\), and \(k\) indicates the degree of expertise. We define the _representative skill_\(\beta^{g}_{*}\) for the chosen group \(g\) as the expected skill level of the subjects within that group:

\[\beta^{g}_{*}=\operatorname*{\mathbb{E}}_{\beta_{i}\sim g}[\beta_{i}]. \tag{2}\]

Margin Computation.For question \(j\) in a dataset \(D\), the performance-margin \(\mu_{j}\) is the difference between the probabilities of _skilled_ humans \(H_{(0)}\) and _skilled_ models \(M_{(0)}\) correctly answering the question, using their respective representative skills \(\beta^{H_{(0)}}\) and \(\beta^{M_{(0)}}\). We set \(k=0\) and designate _skilled_ humans (\(H_{(0)}\)) and models (\(M_{(0)}\)) as the skilled subsets of subjects. These subjects have skills above the average level of their respective subject pools:

\[\mu_{j}=\underbrace{\sigma_{\text{2pl}}\left(\beta^{H_{(0)}}_{*},\theta_{j}, \gamma_{j}\right)}_{\text{Skilled human rep. prob.}}-\underbrace{\sigma_{\text{2 pl}}\left(\beta^{M_{(0)}}_{*},\theta_{j},\gamma_{j}\right)}_{\text{ Skilled model rep. prob.}}, \tag{3}\]

where \(\sigma_{\text{2pl}}\left(\beta,\theta,\gamma\right)\) is the logistic function for our 2pl-irt (Eq. 1, SS 2), that uses \(\beta^{g}_{*}\) as the representative skill for subject group \(g\in\{H_{(0)},M_{(0)}\}\), and \(\theta_{j}\) and \(\gamma_{j}\) are the difficulty and discriminability parameters of the question \(j\).

A **positive value for the margin \(\mu_{j}\) implies that the question \(j\) is _adversarial_** (examples in A.4), while a negative value implies the opposite, and the magnitude indicates the extent of adversarialness.

Accounting for Question Ambiguity.While the margin (\(\mu_{j}\)) captures the core of adversarialness, it does not ensure if the questions are genuinely well-posed; ambiguous, or poorly formulated questions could inflate this score without being _truly_ adversarial. To address this issue, we introduce a discount term (Eq. 4) that relies on the disagreement level among _highly-skilled_ (or expert) human subjects (\(H_{(1)}\)) for each questio 

\[\mu^{\prime}_{j}=\frac{\mu_{j}}{1+\delta_{j}}, \tag{4}\]

where \(\mu^{\prime}_{j}\) is the adjusted adversarialness score, \(\mu_{j}\) is the original adversarialness score, and \(\delta_{j}\) is a measure of disagreement among highly skilled human subjects \(H_{(1)}\) for question \(j\).5 To keep this measure of disagreement standardized, \(\delta_{j}\) is the mean deviation (MD) of the probabilities of \(H_{(1)}\) answering question \(j\) correctly:

Footnote 5: We use this approach for crowdsourced human subjects. For manually identified expert human subjects, we directly use their responses without the need for skill-based filtering.

\[\delta_{j}=\underset{i\sim H_{(1)}}{\text{MD}}\left[\sigma_{2\text{pl}}\left( \beta_{i}^{H_{(1)}},\theta_{j},\gamma_{j}\right)\right)\right]. \tag{5}\]

This discount term ensures that questions with high disagreement among expert humans (potentially ambiguous or ill-posed questions) are penalized, even if they show large human-model performance gaps. This approach leverages the value of human judgment for _true_ adversarial quality assessment.

### Measuring Discriminability

The best questions distinguish between subjects' varying skill levels--they are _informative_ and showcase high _discriminability_. We measure this by leveraging Fisher information over our zl-irt's response prediction function, also called Item Information Function (Lord et al., 1968, iif); it is a function that measures an item's contribution to the measurement precision of \(P(\theta)\) across the skill range (\(\theta\)). With \(P(\theta)\) as the zl-irt's response prediction function \(\sigma_{2\text{pl}}\left(\beta,\theta,\gamma\right)\), we get the item information function (iif\({}_{j}(\theta)\)) that quantifies how much statistical information a question \(j\) provides about a subject's skill level \(\theta\):

\[\textsc{if}_{j}(\theta) =\gamma_{j}^{2}\cdot p_{j}(\theta)\cdot(1-p_{j}(\theta)),\text{ where} \tag{6}\] \[p_{j}(\theta) =\sigma_{2\text{pl}}\left(\theta,\theta_{j},\gamma_{j}\right). \tag{7}\]

Here, the questions with **high discrimination** (large \(\gamma_{j}^{2}\)) and moderate difficulty (resulting in \(P(r_{ij})\approx 0.5\)) provide the most information.

Finally, we define the total item information (tif\({}_{j}\)) provided by question \(j\) as the area under the iiff\({}_{j}(\theta)\) curve, and scale it by exponential normalization to obtain a standardized, calibrated measure of discriminability \(\kappa_{j}\) for question \(j\):

\[\textsc{tif}_{j} =\int_{-\infty}^{\infty}\textsc{if}_{j}(\theta)d\theta, \tag{8}\] \[\kappa_{j} =1-\exp\left(-\textsc{tif}_{j}\right). \tag{9}\]

### Combining into AdvScore

To recap, an ideal adversarial question should (i) have a high margin of human and model performance gap, while being well-posed (low experthumans disagreement), and (ii) be discriminative (informative of the subject's skill). Thus, first combine the adversarialness (\(\mu^{\prime}_{j}\)) and discriminability (\(\kappa_{j}\)) to get a single metric:

\[\textsc{AdvScore}_{\mathrm{j}}=\frac{\mu_{j}}{1+\delta_{j}}\cdot(1+\kappa_{j}) \tag{10}\]

To have human-model probability margin (\(\mu_{j}\)) as a key factor in AdvScore, we treat \(\kappa_{j}\) as a multiplicative bonus to \(\mu_{j}\). This prevents questions with high discriminiability (\(\kappa_{j}\)) from contributing to AdvScore if their \(\mu_{j}\) values are low.

**A positive AdvScore indicates a truly adversarial dataset, with higher values suggesting more discriminative and adversarial questions.** We use AdvScore to evaluate existing datasets (SS 4) and to reward authors in our AdvQA dataset creation process (SS 5.1). We define the AdvScore of a dataset \(D\) as the average AdvScore of its questions. An effective adversarial dataset should contain numerous questions with high AdvScore.

## 4 Adversarial Benchmark Evaluation

We compare adversarial benchmarks across different domains using AdvScore. Our evaluation includes AdvQA, a new QA dataset developed through a human-in-the-loop (hitl) process to align adversarial data with human capabilities. This section, analyzes AdvScore as a metric, while SS 5 details the creation of AdvQA, and SS 6 examines what makes AdvQA questions adversarial.

**Adversarial datasets with human responses.** For AdvQA, we gathered human responses through a live, in-person QA competition involving 8 human teams, as well as through online crowd-sourcing with 165 participants. In total, we collected 1,839 human responses from 172 individuals. To compare the adversarialness of these datasets using AdvScore, which relies on both human and model response data, we are limited to comparing AdvScore with datasets with human annotations. Thus, we select trickme(Wallace et al., 2019b) and fm2(Eisenschlos et al., 2021). While trickme challenges models with qa pairs, fm2 uses entailment pairs for fact-checking.6 Additionally, we included bamboogle(Press et al., 2022), 

which consists of general knowledge questions designed to be adversarial, similar to AdvQA. As bamboo lacked human responses, we gathered 10,391 responses from 165 crowdworkers.

We also collected model responses for each dataset from ten models, including Dense Passage Retrieval (dpr) Karpukhin et al. (2020), GPT-3-Instruct Ouyang et al. (2022), gpt-3.5-turbo OpenAI (2023), mistral-vo.1-instruct Jiang et al. (2023), gpt-4 Achiam et al. (2023), llama-2-chat models in sizes of 7b and 70b, and llama-3-instruct models in sizes of 8b and 70b Touvron et al. (2023). After collecting human and model responses, we apply 2pl-irt to extract the learned subject and item parameters and compute AdvScore.

Comparison of adversarial benchmarks.We compute AdvScore\({}_{D}\) and its components (\(\mu_{D}\), \(\kappa_{D}\), and \(\delta_{D}\)) for each dataset, presenting results in Table 1. Figure 2 walks through the computation of AdvScore by illustrating (i) the skill density of _skilled_ humans \(H_{(0)}\) (blue) and models \(M_{(0)}\) (red), (ii) the response correctness probability (\(\sigma_{\text{2pl}}\), purple), and (iii) the _item information function_, ifg (green, E.q. 6), over skill \(\theta\).

Both AdvQA and trickme show a clear separation between human and model skill levels (first row), resulting in positive, high margins (\(\mu\)) of 0.17 and 0.13, correspondingly (yellow in second row). However, AdvQA has a higher overlap of iff with regions where human skill exceeds model skill (dark green area in third row), compared to TrickMe, which has a flatter and less informative life. These lead to lower \(\kappa_{D}\) (0.56 vs 0.93), suggesting that trickme questions are less discriminative (less useful in assessing subject skills).

In contrast, bamboogle has an informative lif, but the skill of the model tends to exceed humans, resulting in a negative \(\mu_{D}\) (Table 1). This suggests that babboogle questions are inversely adversarial, containing questions where models outperform humans, and therefore fail to serve as an effective adversarial benchmark. Similarly, fm2 has a negative \(\mu_{D}\) and low \(\kappa_{D}\), indicating that the dataset is neither adversarial nor discriminative. Our analysis establishes AdvQA questions as most adversarial, as indicated by its highest AdvScore\({}_{D}\) of 0.31; thus demonstrating that the unique components of AdvScore effectively support the evaluation of adversarial benchmarks.

Figure 2: **Visualization of key AdvScore components across datasets. For each dataset, we plot: (1) Skill density of skilled humans (\(H_{(0)}\)) and skilled models (\(M_{(0)}\)), (2) response correctness probability, \(\sigma_{\text{2pl}}(\theta)\) (Eq. 1, § 2) averaged over dataset examples, and (3) Item information function (if\((\theta)\)(Eq. 6, § 3.2). Vertical dashed lines show representative (average) skill levels for humans and models. The gap between human and model probabilities (shaded region between the horizontal lines) indicates adversarialness (\(\mu_{D}\)). lif peaks show three questions are most informative, with area under curve signaling total informativeness (discriminability, \(\kappa_{D}\)). Key insights: bamboogle has high informativeness but favors models (negative \(\mu_{D}\)). TrickMe separates humans and models but has lower discriminability (positive \(\mu_{D}\)). AdvQA is the best of all, effectively discriminating between humans and models while maintaining high informativeness throughout, resulting in the highest AdvScore of 0.31.

 



**Chronological evaluation of adversarialness** Adversarial datasets inevitably become obsolete as models improve, either by training on these datasets or overcoming previously identified vulnerabilities. Using AdvScore, we assess model improvements over the last five years by identifying which datasets have become less adversarial, incorporating new models into the AdvScore computation.7 Figure 3 shows the AdvScore for each dataset over the years, confirming that AdvVQA holds the highest AdvScore (2024) with the smallest decline over the last five years. In contrast, trickme, which was initially the most highly adversarial (2020), saw a sharp decline over the following four years, indicating that the models improved on the tasks that they previously struggled with. bamboogle and fm2 are no longer adversarial, showing negative AdvScore values since 2022. bamboogle's reliance on a 2-hop tactic and simple questions (e.g., _"What is the capital of the second largest state in the US by area"_) likely explains its decline since 2021. fm2's drop suggests llms have improved at fact-checking or benefitted from similar questions in training. Although pinpointing the exact factors behind model improvement may be challenging, it is crucial to determine whether these models have become more resilient or remain vulnerable as new models emerge. AdvScore facilitates this by quantifying how much a dataset has lost its adversarialness, offering a concrete measure of how well the model withstands adversarial challenges over time.

Footnote 6: [https://github.com/journals/cai/fam2022/](https://github.com/journals/cai/fam2022/)

Qualitative Examples with AdvScoreWe examine the human-model margin probability (\(\mu_{j}\)) and each subject's answers to the example question for each dataset. In Table 6, AdvQA and trickme questions show a positive \(\mu_{j}\) value, indicating adversarial, correspondent to the human's correct answer to ("Putin") and GPT4's wrong answer ("Russia"). On the other hand, bamboo and fm2's negative adversarialness value suggests that the question is easier for models compared to humans, as reflected in the higher correctness from models versus humans.

**Comparision of AdvScore and QSR** Moreover, we conducted a comparative analysis of model and human success rates (QSR) and AdvScores (SS 2). While QSR may suggest that humans outperform models, the questions can consistently yield negative AdvScores, due to their low or negative \(\mu\) (margin) or high \(\delta\) (ambiguity). Examples and analyses in Appendix A.5). This highlights that QSR alone is insufficient to determine question adversarialness, whereas each parameter in AdvScore offers a more reliable measure.

## 5AdvQA creation pipeline

In the previous sections, we showed that AdvQA is more adversarial and discriminative than other datasets, suggesting its creation process contributed to these qualities. Here, we discuss the AdvQA collection process as a case study to guide future high-quality adversarial datasets.

### Collecting questions and answer pairs through adversarial competitions

To obtain human-written question-answer pairs, we hold two adversarial model-human qa competitions. First, in the writing competition, we col

Figure 3: We report AdvScore for each dataset over the years, confirming that AdvQA holds the highest AdvScore with the smallest decline over the last five years, proving its adversarial robustness.



lect 399 adversarial questions through the interface (SS5.2), which are then edited and filtered by an expert editor. Second, in the answering competition, we invited eight expert human groups (composed of three to four trivia experts) to run eight human vs. model qa tournaments to obtain 780 human responses. Each tournament initially consisted of 30 questions, which are then filtered based on experts' comments (E.g., _"This question is ill-posed"_). After this filtering process, AdvQA results in 182 questions.8 After the competitions, we incentivize the writers with the highest AdvScore and players with the highest skill.9

Footnote 8: Larger than other irt-analysed test sets (e.g., 139 for RTE, 20 for CommitmentBank, 50 for COPA) [46]. Also, additional 1,839 human responses collected from 172 individuals (165 crowdsource workers). Dataset value includes both questions and response volume.

### Skilled writers use adversarial interface

We provide an adversarial writing interface as a human-AI collaborative tool for the adversarial writing competition, motivated by You and Lowd (2022)'s finding that human-AI collaboration strengthens adversarial attacks. We supply the writers with real-time model interpretations, inspired by Wallace et al. (2019); they could continuously counteract the model response and make edits.

**Eliciting incorrect model predictions** The center of the interface (Figure 5 in Appendix A.8) provides the Wikipedia page for the target answer, which they use to write the question. While the author is writing, the retrieval widget and qa models widgets are updated [10]. Motivated by Feng et al. (2018), we embed the input perturbation inside the question writing widget to highlight which words trigger the model predictions. For example, changing "company" to a different token would be most likely to change the prediction except the answer "Apple."

Retrieval systemsUsers receive real-time feedback on qa systems' performance on their questions via the interface's fine-tuned retrieval and reader model components (the retrieval system outputs: contexts that elicit qa system predictions). If the target answer appears at the top of the retrieval widget, which means the author failed to fool the retriever and the reader, authors can rephrase questions to avoid retrieving information that makes qa systems answer correctly. We use lightweight sparse and neural retrieval models for writer feedback: a tf-IDF baseline and dpr. To ensure that dpr predictions are diverse and up-to-date, we create a database that indexes each sentence in a set of Wikipedia pages (see Appendix A.8). We then use the RoBerta-based FarmReader, which is fine-tuned on squad(Rajpurkar et al., 2016), to read and sort the retrieved sentences from the two retrieval models by their relevance.

lm-based qa systemsWe enrich the model guidance using extractive and generative model answer predictions. For extractive qa, we use DistilBert (fine-tuned on squad), since its promptness and lightness facilitate rapid human-AI interaction. We also use T510[12] to answer the questions in a closed-book setting.

Footnote 10: The writing competition was held in Spring 2023, when distillert and T5 were considered comparatively strong.

## 6 Discussion and Analysis on AdvQa

In this section, we show how AdvScore can help identify factors that encourage high-quality adversarial datasets. Effective strategies in AdvQA may guide the creation of more adversarial questions, and we analyze how the dataset's realistic aspect can help incorporate human variability during model evaluation.

**Ensuring high-quality adversarial questions** The questions should be adversarial for reasons that

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Dataset** & **Question** & **Answer** & **Margin (\(\mu_{j}\))** & **Human Response** & **tf-4** \\ \hline AdvQA & Who is the president of the country repesented by the second letter in the acronym BRICS [...] & Vladimir & 0.19 & Putin & Russia \\ \hline fm2 & Azam Khachaturian had Russian roots. & False & -0.01 & “False” & True \\ \hline trickME & In a novel by this author, a detective wrang his arm & Durremental & 0.12 & “Daremenmat” & Franz Kafka \\  & to survive a dog attack [...] & & & & \\ \hline BAMBOGLE & Who directed the highest groossing film? & James & -0.02 & “No idea” & James \\  & & Cameroon & & & Cameron \\ \hline \hline \end{tabular}
\end{table}
Table 2: AdvQA demonstrates the most balanced properties of challenging the model and distinguishing between skills, as indicated by a positive \(\mu_{j}\) value, which aligns with humans outperforming the models.

 

identify model weaknesses, such as the inability to compose clues or exclude redundant clues (Min et al., 2020, 2022) not because of trivial errors (e.g., grammar mistakes). If the question meets this criteria, we consider it high-quality. We base our criteria on the taxonomy of adversarial categories in Wallace et al. (2019b). To understand what yielded AdvQA's _high-quality_ adversarial questions, manually annotate the adversarial tactics and topics for AdvQA questions (Appendix B.2).

With the identified question characteristics, we run a logistic regression model to learn how much each adversarial tactic or topic contributed to AdvScore.11 Since all questions in AdvQA yielded a positive AdvScore, the coefficients in Figure 4 reflect how much specific features contributed to adversarialness, highlighting areas where models need improvement. For instance, the tactic involving _commonsense knowledge_ on the topic of _lifestyle_ exposed a model weakness (e.g., _"Take away four from a group including Barnard and Smith, and you get what play?"_), which had a notably high AdvScore of 0.27.12

Footnote 11: Focusing on assessing adversarialness through irt, we provide only a basic analysis using pre-assigned features. Applying advanced irt models is encouraged for a richer analysis of adversarial factors (Gor et al., 2024).

**Leveraging human feedback for _realisticness_ Realism is crucial for an adversarial dataset as it creates challenges that closely resemble real-world scenarios, effectively testing model robustness against plausible but diverse situations. This approach enhances the reliability of performance evaluation as it reflects high variance in collective human ability. For example, not only should the questions be adversarial, but they should mimic diverse reasoning and problem-solving strategies of different people. Our preliminary results revealed that crowdworkers often produced ambiguous or poorly-formed questions.13 Although AdvScore could identify these issues, many examples were ineffective for assessing model performance. We thus recruit expert trivia writers and guide them in writing adversarial questions. Then, other trivia editors scrutinize the human-authored questions' poor quality (see Appendix B.1). Finally, our human vs. model competition provides an additional quality check, as human subjects flag potential issues while answering questions. If the subject or the editor considers a question unnatural or ambiguous, we exclude it from our final dataset (Appendix A.1).

Footnote 13: E.g., “_Who led the final siege of Constantinople?”_ carries ambiguity depending on historical framing (_Mehmed II for the 1453 siegre_ or _other leaders in prior sigees_).

We emphasize that human responses are especially useful in adversarial evaluation contexts, as they ensure that adversarial examples are genuinely challenging and realistic. Moreover, these responses are provided by each individual's intuition, creativity, and understanding. Thus, capturing variability is crucial to evaluate the benchmarks that are meant to assess evolving models aiming for human alignment. Such aspects are what traditional model-generated adversarial attacks cannot replicate. Ultimately, incorporating human responses adds depth and reliability to adversarial benchmarks, making them essential in evaluating models' true progress toward human-level understanding and their performance.

## 7 Related Work

Adversarial samples expose and evaluate model capabilities (Melis et al., 2017; Biggio et al., 2013). Recently, the Natural Language Processing (NLP) community has questioned whether models trained on benchmarks learn to solve tasks in robust and generalizable ways (Ribeiro et al., 2020; Bartolo et al., 2021; Nie et al., 2018; Gururangan et al., 2018; Kaushik et al., 2021). Thus, evaluation of adversarial samples has been active in areas of reading comprehension (Jia and Liang, 2017) and neural

Figure 4: The overall distribution of LR coefficients suggests that _lifestyle_ and _commonsense knowledge_ contribute more to adversarialness than other features. This implies that models still struggle with commonsense knowledge, highlighting an area where they remain vulnerable compared to human understanding.

 

translation tasks (Belinkov and Bisk, 2018; Wallace et al., 2019). Tedeschi et al. (2023) postulates that the abilities of many "superhuman" models may be overestimated due to poorly annotated datasets and biases embedded in the evaluation process (e.g., fixed test sets).

An alternative is to provide more challenging benchmarks that require a stronger form of generalization and diversity (Rychalska et al., 2019; Bowman, 2023; Yuan et al., 2023); hitl adversarial generation framework enables humans create examples while interacting with the model (Ma et al., 2021). For qa tasks, it is crucial to validate the model's ability to correctly answer easy and natural questions that are likely to be expressed by humans. For hitl adversarial generation for qa, Bartolo et al. (2021) and Kiela et al. (2021) uses a synthetic generation method to amplify small set of human-authored adversaries. Sheng et al. (2021) introduces a benchmark in which the humans interact with a visual qa model, and write an adversarial question for each of a set of images. Wallace et al. (2019) and Eisenschlos et al. (2021) both use hitl incentive mechanisms to create adversarial questions. For evaluation of these adversarial datasets, Lalor et al. (2019) introduces an irt-based ranking method to remedy the issue that current evaluation treats each model independently rather than considering relative differences. Rodriguez et al. (2021) also redesigns the leaderboard framework with a Bayesian approach where latent subject skill and item difficulty predict correct responses. Our AdvScore can systematically probe models to understand their capabilities, and provide a measure to understand which also contribute in hitl adversarial dataset framework to help to create the next generation of data.

## 8 Conclusion

Adversarial datasets offer practical benefits for evaluating models to improve robustness and performance. Grounded in human feedback, AdvScore ensures that evaluations of adversarial benchmarks align with human capabilities by post-hoc assessment of adversarial robustness and model improvements. Thus, applying AdvScore in real-time benchmark construction can aid in evaluating the robustness of the models, and integrating AdvScore into model training can improve their adaptability to real-world applications.

## 9 Limitations and Future Works

One limitation of AdvScore is its reliance on expert-level human annotations that makes it challenging to implement. However, human feedback ensures that adversarial questions are not only technically challenging but also meaningful and reflective of real-world scenarios. To mitigate this, semi-supervised or active learning approaches could be explored to minimize manual annotations, where models assist in identifying adversarial examples based on human feedback.

Another limitation is that AdvScore does not account for model confidence, which may overlook reliability aspects. We recommend incorporating a calibration assessment to determine if predicted probabilities align with accuracy, encouraging more reliable adversarial benchmarks and thereby preventing overconfident models.

Furthermore, as the core of AdvScore aims to assess how well models match human ability in real-life tasks, it is valuable to evaluate adversarial datasets in real-world applications, such as machine translation and chatbot evaluation across different modalities. We encourage using AdvScore to develop adversarial datasets across diverse NLP tasks and contribute to robust system developments.

## 10 Ethical Considerations

We address ethical considerations for dataset papers, given that our work contains a new dataset AdvQA and collecting human responses in our user study. We reply to the relevant questions posed in the ACL 2022 Ethics FAQ.14

Footnote 14: [https://www.acm.org/code-of-ethics](https://www.acm.org/code-of-ethics)

When collecting human responses and questions, our study was pre-monitored by an official IRB review board to protect the participants' privacy rights. Moreover, the identity characteristics of the participants were self-identified by the workers by answering the survey questions.

Before distributing the survey, we collected consent forms for the workers to agree that their answers would be used for academic purposes. The trivia experts were awarded a total $\(1100\) worth of online gift cards after the competitions. The prizes were awarded to the first, second, and third winners, depending on each group's AdvScore. The crowdworkers were compensated over \(10\) usd an hour (a rate higher than the us national minimum wage of \(7.50\) usd ).

 



## 11 Acknowledgements

We thank all the CLIP members who reviewed the idea of improving adversarial benchmark evaluation. We also thank the players who participated in the tournament: Munir Siddiqui, Aaron Lichtig, J.R. Parsons, Ethan Medwetsky, Matt Weiner, and Alex Schmidt. Their valuable contributions greatly impacted the progress of this work. This project was awarded the MetaAI Dynabench Grant "A Leaderboard and Competition for Human-computer Adversarial Question Answering". Additionally, this research was partially supported by an NSF GRFP grant. Sung and Boyd-Graber are supported by NSF Grant IIS2403436. Opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the views of the sponsors.

## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Baker and Kim (2004) Frank B Baker and Seock-Ho Kim. 2004. _Item response theory: Parameter estimation techniques_. CRC press.
* Bartolo et al. (2021) Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. 2021. Improving question answering model robustness with synthetic adversarial data generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 8830-8848, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Belinkov and Bisk (2018) Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In _International Conference on Learning Representations_.
* Biggio et al. (2013) Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Srndic, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. 2013. Evasion attacks against machine learning at test time. In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13_, pages 387-402. Springer.
* Biggio et al. (2012) Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012. Poisoning attacks against support vector machines. In _Proceedings of the 29th International Coference on International Conference on Machine Learning_, pages 1467-1474.
* Bowman (2023) Samuel R Bowman. 2023. Eight things to know about large language models. _arXiv e-prints_, pages arXiv-2304.
* Bowman and Dahl (2021) Samuel R. Bowman and George Dahl. 2021. What will it take to fix benchmarking in natural language understanding? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4843-4855, Online. Association for Computational Linguistics.
* Dathathri et al. (2019) Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. _arXiv preprint arXiv:1912.02164_.
* Eisenschlos et al. (2021) Julian Eisenschlos, Bhuwan Dingra, Jannis Bulian, Benjamin Borschinger, and Jordan Boyd-Graber. 2021. Fool me twice: Entailment from Wikipedia gamification. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 352-365, Online. Association for Computational Linguistics.
* Engstrom et al. (2020) Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. 2020. Adversarial robustness as a prior for learned representations. _arXiv preprint arXiv:1906.00945_.
* Feng et al. (2018) Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Ivyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3719-3728, Brussels, Belgium. Association for Computational Linguistics.
* Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In _International Conference on Learning Representations (ICLR)_.
* Gor et al. (2024) Maharshi Gor, Hal Daume III, Tianyi Zhou, and Jordan Boyd-Graber. 2024. Do great minds think alike? investigating human-ai complementarity in question answering with caimira. _arXiv preprint arXiv:2410.06524_.
* Gurungan et al. (2018) Suchin Gurungan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
* Ilyas et al. (2019) Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. 2019. Adversarial examples are not bugs, they are

features. _Advances in neural information processing systems_, 32.
* Jennings (2007) Ken Jennings. 2007. _Brainiac: adventures in the curious, competitive, compulsive world of trivia buffs_. Villard.
* Jia and Liang (2017) Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2021-2031.
* Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. _arXiv preprint arXiv:2310.06825_.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781, Online. Association for Computational Linguistics.
* Kaushik et al. (2021) Divvansh Kaushik, Douwe Kiela, Zachary C. Lipton, and Wen-tau Yih. 2021. On the efficacy of adversarial data collection for question answering: Results from a large-scale randomized study. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6618-6633, Online. Association for Computational Linguistics.
* Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divvansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4110-4124, Online. Association for Computational Linguistics.
* Lalor et al. (2016) John P. Lalor, Hao Wu, and Hong Yu. 2016. Building an evaluation scale using item response theory. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 648-657, Austin, Texas. Association for Computational Linguistics.
* Lalor et al. (2019) John P. Lalor, Hao Wu, and Hong Yu. 2019. Learning latent parameters without human response patterns: Item response theory with artificial crowds. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4249-4259.
* Lelkes et al. (2021) Adam D Lelkes, Vinh Q Tran, and Cong Yu. 2021. Quizstyle question generation for news stories. In _Proceedings of the Web Conference 2021_, pages 2501-2511.
* Lord et al. (1968) Frederic M Lord, Meivin R Novick, and Allan Birnbaum. 1968. Statistical theories of mental test scores. 1968. _Reading: Addison-Wesley_.
* Ma et al. (2021) Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Yu Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021. Dynaboard: An evaluation-as-a-service platform for holistic next-generation benchmarking. In _Neural Information Processing Systems_.
* Martinez-Plumed et al. (2019) Fernando Martinez-Plumed, Ricardo BC Prudencio, Adolfo Martinez-Uso, and Jose Hernandez-Orallo. 2019. Item response theory in ai: Analysing machine learning classifiers at the instance level. _Artificial intelligence_, 271:18-42.
* Melis et al. (2017) Marco Melis, Ambra Demontis, Battista Biggio, Gavin Brown, Giorgio Fumera, and Fabio Roii. 2017. Is deep learning safe for robot vision? adversarial examples against the icub humanoid. In _Proceedings of the IEEE international conference on computer vision workshops_, pages 751-759.
* Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5783-5797, Online. Association for Computational Linguistics.
* Min et al. (2022) Sewon Min, Luke Zettlemoyer, Hannaneh Hajishirzi, et al. 2022. Crepe: Open-domain question answering with false presuppositions. _arXiv e-prints_, pages arXiv-2211.
* Nie et al. (2018) Yixin Nie, Yicheng Wang, and Mohit Bansal. 2018. Analyzing compositionality-sensitivity of nli models. _ArXiv_, abs/1811.07033.
* OpenAI (2023) OpenAI. 2023. Chatgpt (mar 14 version). Large language model.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhnii Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in neural information processing systems_, 35:27730-27744.
* Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3419-3448.
* Pollard (2006) John K Pollard. 2006. Student reflection using a web-based quiz. In _2006 7th International Conference on Information Technology Based Higher Education and Training_, pages 871-874. IEEE 

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. _arXiv preprint arXiv:2210.03350_.
* Quaye et al. (2024) Jessica Quaye, Alicia Parrish, Oana Inel, Charvi Rastogi, Hannah Rose Kirk, Minsuk Kahng, Erin Van Liemt, Max Bartolo, Jesus Tsang, Justin White, et al. 2024. Adversarial nibbler: An open red-teaming method for identifying diverse harms in text-to-image generation. In _The 2024 ACM Conference on Fairness, Accountability, and Transparency_, pages 388-406.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _J. Mach. Learn. Res._, 21(1).
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do ImageNet classifiers generalize to ImageNet? In _Proceedings of the 36th International Conference on Machine Learning_, volume 97 of _Proceedings of Machine Learning Research_, pages 5389-5400. PMLR.
* Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4902-4912, Online. Association for Computational Linguistics.
* Rodriguez et al. (2021) Pedro Rodriguez, Joe Barrow, Alexander Miseris Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber. 2021. Evaluation examples are not equally informative: How should that change NLP leaderboards? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 4486-4503, Online. Association for Computational Linguistics.
* Rodriguez et al. (2019) Pedro Rodriguez, Shi Feng, Mohit Ilyer, He He, and Jordan L. Boyd-Graber. 2019. Quizbowl: The case for incremental question answering. _CoRR_, abs/1904.04792.
* Rogers et al. (2023) Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension. _ACM Comput. Surv._, 55(10).
* Ross et al. (2021) Alexis Ross, Ana Marasovic, and Matthew E Peters. 2021. Explaining nlp models via minimal contrastive editing (mice). In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 3840-3852.
* Rychalska et al. (2019) Barbara Rychalska, Dominika Basaj, Alicja Gosiewska, and Przemyslaw Biecek. 2019. Models in the wild: On corruption robustness of neural nlp systems. In _Neural Information Processing: 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part III_, page 235-247, Berlin, Heidelberg. Springer-Verlag.
* Sheng et al. (2021) Sasha Sheng, Amampreet Singh, Vedanuj Goswami, Jose Alberto Lopez Magana, Wojciech Galuba, Devi Parikh, and Douwe Kiela. 2021. Human-adversarial visual question answering. _CoRR_, abs/2106.02280.
* Si et al. (2023) Chenglei Si, Navita Goyal, Sherry Tongshuang Wu, Chen Zhao, Shi Feng, Hal Daume III, and Jordan Boyd-Graber. 2023. Large language models help humans verify truthfulness-except when they are convincingly wrong. _arXiv preprint arXiv:2310.12558_.
* Tedeschi et al. (2023) Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajic, Daniel Hershcovich, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, and Roberto Navigli. 2023. What's the meaning of superhuman performance in today's NLU? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12471-12491, Toronto, Canada. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. LLama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Tsipras et al. (2019) Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Robustness may be at odds with accuracy. In _International Conference on Learning Representations_.
* Uesato et al. (2018) Jonathan Uesato, Brendan O'donoghue, Pushmeet Kohli, and Aaron Oord. 2018. Adversarial risk and the dangers of evaluating against weak attacks. In _International conference on machine learning_, pages 5025-5034. PMLR.
* Vania et al. (2021) Clara Vania, Phu Mon Hutt, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, and Samuel R. Bowman. 2021. Comparing test sets with item response theory. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1141-1158, Online. Association for Computational Linguistics.
* Wallace et al. (2019a) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019a. Universal adversarial triggers for attacking and analyzing NLP. In _Proceedings of the 2019 Conference on Empirical Methods 

in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2153-2162, Hong Kong, China. Association for Computational Linguistics.
* Wallace et al. (2019b) Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019b. Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering. _Transactions of the Association for Computational Linguistics_, 7:387-401.
* You and Lowd (2022) Wencong You and Daniel Lowd. 2022. Towards stronger adversarial baselines through human-AI collaboration. In _Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP_, pages 11-21, Dublin, Ireland. Association for Computational Linguistics.
* Yu et al. (2023) Xinyan Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. CREPE: Open-domain question answering with false presuppositions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 10457-10480, Toronto, Canada. Association for Computational Linguistics.
* Yuan et al. (2023) Quan Yuan, Mehran Kazemi, Xin Xu, Isaac Noble, Vaiva Imbrasaite, and Deepak Ramachandran. 2023. Tasklama: Probing the complex task understanding of language models. _arXiv preprint arXiv:2308.15299_.

## Appendix A Details on Dataset Creation

### Recruitment for Dynamic QA Generation

When tasking human authors with adversarial writing of questions, Wallace et al. (2019b) emphasizes the importance of "who" the authors should be: _talented and eager_ question writers with _specific goals_; they should aim to generate questions that stump computers but seem normal enough for humans to answer. To make this work, they recruit members of the quizbowl community, who have deep trivia knowledge and craft question for quizbowl tournaments Jennings (2007). However, their challenge was to convey what is "normal" to authors and stimulate examples that can elucidate the weaknesses of QA models.

### Merging Trivia Question Generation and Dynamic Adversarial Generation Process

Many QA datasets are now too easy for modern models as models have become more powerful Rogers et al. (2023). However, even these easy QA datasets have serious data flaws Min et al. (2020); Yu et al. (2023), which suggests that creating question-answer pairs is a very challenging task. This is also a norm for questions written for human players, where more than 100,000 questions are produced annually. To create effective and challenging enough questions, the professional experts (e.g., writing staff) take a rigorous editing pass on the questions to decide whether they are adequate enough to guarantee players a fair game Lelkes et al. (2021); Pollard (2006). They follow strict guidelines to be selected to be used in the quiz matches. We propose to merge the above pipelines to help improve data creation for robust QA models by adding an editing step to ensure that grammatical errors and nonfactual questions (following the norms of Trivia questions) do not exist in the pool. In Table 3, we list the problematic question types that we ask the editors or subjects to flag.

### Details on errors in using raw scores in question answering competition

We infer that the human accuracy does not necessarily translate to answering ability or question difficulty measurement, which obscures the measuring the the question's adversarial-ness. While the most skillful human team answered all three questions correctly, the estimated probability of the human teams answering the question correctly when compared to their ability was low (50%) 



### Qualitative Examples of each dataset with AdvScore

We examine the adversarial properties of each question (\(\mu_{j}\) and \(\kappa_{j}\)) with qualitative examples and each subject's example responses from four datasets [Table 6].

#### Comparison Analysis of AdvScore and QSR

We show that QSR alone is insufficient to determine question adversarialness, obscuring the real challenge, whereas each parameter in AdvScore offers a more nuanced measurement.

For questions like _What was the founding date of the university in which Plutonium was discovered?_ and _Who is the father of the father of observational astronomy?_, humans significantly outperformed models, but their negative AdvScores (\(-0.365\) and \(-0.340\)) indicate that these questions remain non-adversarial. This demonstrates that QSR alone is insufficient to identify question adversarialness. AdvScore, by incorporating both margin and discriminative power, provides a more nuanced and reliable measure, and reflects the adversarial nature of questions.

In AdvQA, AdvScore highlights contrasts that QSR may fail to capture. For instance, the question _Name the color of the sky in Aivazovsky's "The Ninth Wave"_ exhibits a significant QSR gap between humans (0.667) and models (0.083), yet its positive AdvScore\({}_{j}=0.188\) remains low, due to high \(\delta\) (indicating) compared to other examples. The question implies a single color, but the 'The Ninth Wave" painting contains multiple hues. It also lacks specificity about which part of the sky is being referenced.

Other examples in Table 5 show a similar trend of having a high qsr gap, suggesting that humans significantly exceed model performance, but this

\begin{table}
\begin{tabular}{l l l l} \hline Question & Gold Answer & Human & Probability \\  & & Answer & \(\sigma(\beta_{i}-\theta_{j})\) \\ \hline What phrase is common to the title of novel featuring a fictional Nat King Cole recording, a Gene Aturty film and song, and an I-95 attraction between the Carolinas? & South of the Border & Correct & 0.57 \\ In which novel, written by an author who was originally a botanist and born in Cuba, features a fictitious conversation between a merchant who travelled a road that was known by a smooth natural material and an emperor who loved to write Chinese poetry, both of which are actual people in history? & Unwit: Cities & Correct & 0.55 \\ What is the name of the first mosque in the world that was built by Prophet Muhammed (s.a.w) during his hijrah from Mecca to Medina? & Quba Masjid & Correct & 0.56 \\ \hline \end{tabular}
\end{table}
Table 4: While the most skillful human team answered all three questions correctly, the estimated probability of the human teams answering the question correctly when compared to their ability was low (50%).

\begin{table}
\begin{tabular}{p{28.5pt} p{113.8pt} p{113.8pt}} \hline \hline
**Question Type** & **Description** & **Examples** \\ \hline Lacks Factuality & Requires information is factual & “Trump, the first woman president of the United States, is charged against federal laws” is non factual as the gender of Trump is male \\ Lacks Specificity (False Presupposition) & Requires move information to be answered with clarity & “What is the color of Flannigony’s feathers?” is ambiguous as Pink and White could be two possible answers depending on when they are born \\ Subjectivity & Contains clues that are highly subjective & “What’s the name of Christopher Columbus’s most famous ship?” Possible answers could help either Santa Maria, La Nima, Santa Clara Also, as “Most famous” can mean many different things, the revised question could be “Which of Columbia’s ships was stripped or it’s unlikely to aid a feel called La Neriudadi in northern Hail!” \\ Ambiguity \& Multiple acceptable answers & Can be answered with multiple answers & Nikolas Alexandrovich Romanov; Nicolas II, Nikolai Il Alexandrovich Romanov: all of these are acceptable as answers. \\ \hline \hline \end{tabular}
\end{table}
Table 3: We list the problematic question types that we ask to annotate. The four types are illustrated with descriptions and examples to help them better understand each question, and help determine whether each question has good quality.



is contradicted by the corresponding AdvScore. For example, the question _What country shares a language with its more populous northern neighbor but in its written form omits a letter that looks like a Greek beta, writing the sound instead by doubling another letter?_ shows low discriminability (\(\kappa_{j}=0.626\)) and a low AdvScore\({}_{j}=0.081\). The question _A German admiral sailing for Russia named what islands for an English captain and not for the librettist of the HMS Pinofore nor for the announcer of Jeopardy!_ represents a low discriminability (\(\kappa_{j}=0.504\)) and the lowest AdvScore\({}_{j}=0.051\) among the dataset. Although it is adversarial (\(\mu_{j}=0.233\)), it fails to significantly differentiate between human and model abilities. Similarly, for bamboogle's questions which were mostly _reversely_ adversarial, while qsr suggested that the question is easier for humans compared to models.

### User Study

We conducted two user studies for this paper. We recruited 1) human writers to write on the interface and 2) human respondents to answer collected AdvQA questions and bambooquestions that did not have existing human responses.

#### User Study to collect questions

We recruited the writing team via online advertisement three months ahead of the human vs. computer question-answering competition. We collected 399 questions from five expert human writers (members of trivia community). We first display our consent form and instructions before question writers encounter the interface. They were dismissed from the study immediately if they did not pay their consent. We then inform them how their questions and prizes will be assessed; AdvScore accurately estimates assigned criteria (e.g., adver

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline  & \multicolumn{7}{c}{**AdvQA Dataset**} \\ \hline
**Question** & **Answer** & **Human QSR** & **Model QSR** & \(\mu_{j}\) & \(\delta_{j}\) & \(\kappa_{j}\) & **ADvScore\({}_{j}\)** \\ \hline Name the color of the sky in Aivazovsky’s “The Ninth Wave” & Orange & 0.667 & 0.083 & 0.583 & 0.106 & 0.963 & 0.188 \\ \cline{2-7} \multirow{-2}{*}{The title of this book shares a word with the title of a song of which the author, who acted in the 2002 film, 8 Mile, addressed to his daughter and nice nice} & To Kill a Mokingbird & 0.333 & 0.000 & 0.323 & 0.102 & 0.983 & 0.179 \\ \cline{2-7} \multirow{-2}{*}{What country shares a language with its more populous northern right in its written form omits a letter that looks like a Greek beta, writing the sound instead by dough another letter? That character appears in that language’s words for foot, bigoutside, and street} & & & & & & \\ \cline{2-7} \multirow{-2}{*}{A German admiral sailing for Russian named what islands for an English captain and not for the librettist of the HMS Pinofore nor for the announceor of Jeopardy!} & Gilbert Islands & 0.333 & 0.100 & 0.233 & 0.034 & 0.504 & 0.051 \\ \hline \hline \end{tabular}
\end{table}
Table 5: A substantial gap in QSR may suggest human superiority over models, indicating an adversarial question. However, it can still yield negative AdvScores due to low or negative \(\mu\) or relatively high \(\delta\). In both AdvQA and Bambooleg, even when human QSR surpasses model QSR, this is not always reflected in AdvScore, given the distinct criteria of each parameter. For instance, the first question in AdvQA, _Name the color of the sky in Aivazovsky’s ”The Ninth Wave” exhibits a significant QSR gap between humans (0.667) and models (0.083), yet its positive AdvScore\({}_{j}=0.188\) remains low, due to high \(\delta\) (indicating question ambiguity) compared to other examples. The question implies a single color, but the “The Ninth Wave” painting contains multiple hues. It also lacks specificity about which part of the sky is being referenced.

 

sarialness and discriminability). To make the question writing process more interesting and fun, we gamify the writing process by applying a reward system. After submitting their question sets, we calculate the AdvScore for each writer's question set; then, we reward $500 for those who won the first place, $250 for second place, and $100 for third place.

### Interface details

Interface ScreenshotWe provide an adversarial writing interface (Figure 5) as a human-AI collaborative tool for the adversarial writing competition, motivated by You and Lowd (2022)'s finding that human-AI collaboration strengthens adversarial attacks. We focus on supplying the skilled-human with the real-time model interpretations, inspired by Wallace et al. (2019b), so that they could continuously counteract the model response and make better edits.

Retrieval System DetailsTo ensure that the retrieval results help in obtaining up-to-date information for the writers, we created the database for Wikipedia pages and DPR training data. DPR retrieves the most relevant sentence from a database that consists of the Top 1000 popular Wikipedia pages16 from 2021 to 2022. DPR is finetuned with the 2018 and 2021 QANTA datasets Rodriguez et al. (2019). For training, we used the questions and gold evidence as positive samples, and sentences from pages that are two hops away (pages linked by randomly selected hyperlinks in the summary section) from the question page as negative samples.

Footnote 16: Inspired by Wallace et al. (2019b), we add more tactics such as Location Misalignment.

## Appendix B Adversarial Tactics and Question Categories

### Question Category Annotation

We report the statistics of topic categories and adversarial tactics present in AdvQA.

We ask the question writers to tag their questions with the categories below. On specific categories and examples, we encourage them to be as creative and diverse as possible when authoring the questions. In the interface, they can monitor how many questions they wrote per category. They are required to submit question sets in each of ten categories: Art, Literature, Geography, History, Science, TV and Film, Music, Lifestyle, and Sports, Miscellaneous (Appendix B.1).

### Adversarial Tactic Annotation

In Table 9, we list adversarial tactics used in AdvQA questions. We provide descriptions and examples to annotate questions with adversarial tactics (Table 9).17

Footnote 17: Inspired by Wallace et al. (2019b), we add more tactics such as Location Misalignment.

### Annotation Examples

Table 10 shows question examples that are annotated with question and adversarial tactics. The highlights in the question correspond to either adversarial tactics or question categories that are highlighted with the same color.

### IRT Model Details

We use a neural approach to train our 2PL IRT model, leveraging the flexibility and scalability of

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**Dataset** & **Question** & **Answer** & \(\mu_{j}\) & \(\kappa_{j}\) & **Human** & **gpt-4** \\ \hline AdvQA & Who is the president of the country represented by the second letter in the acronym BRICS [...] & Vladimir Putin & 0.16 & 0.80 & Putin & Russia \\ \hline fm2 & Henry I got married and took the throne in 1100. & True & 0.02 & 0.01 & “True” & False \\ \hline trickme & In a novel by this author, a detective wraps his arm to survive a dog attack [...] & Durrenmatt 0.19 & 0.16 & “Durrenmatt” & Franz Kafka \\ \hline bamboogle Who directed the highest grossing film? & James Cameron & -0.02 & 0.10 & “No idea” & James Cameron \\ \hline \hline \end{tabular}
\end{table}
Table 6: AdvQA demonstrates the most balanced properties of challenging the model and distinguishing between skills, as indicated by a positive value, which aligns with humans outperforming the models.

 

neural networks while maintaining the interpretability of the IRT framework. The model parameters are learned through backpropagation, with the network architecture designed to mimic the 2PL IRT structure.

Model ArchitectureThe neural 2PL IRT model consists of three main components:

1. An item embedding layer representing item difficulties (\(\beta_{i}\)) and discriminations (\(\gamma_{i}\))
2. A person embedding layer representing person abilities (\(\theta_{j}\))
3. A sigmoid output layer computing the probability of a correct response

The total number of parameters in our model is \(2N+M\), where \(N\) is the number of items and \(M\) is the number of subjects. This count includes \(N\) difficulty parameters, \(N\) discrimination parameters, and \(M\) ability parameters.

Prior DistributionsWe incorporate prior distributions on the model parameters to enhance regularization and interpretability:

* Item difficulties (\(\beta_{i}\)) and person abilities (\(\theta_{j}\)): Gaussian priors with mean 0 and variance 1
* Item discriminations (\(\gamma_{i}\)): Gamma prior with shape \(k\) and scale \(\theta\)

The use of a Gamma prior for discriminations ensures positivity and allows for fine-tuning the model's sensitivity to item discrimination.

Training Procedure
1. Initialize network weights randomly, sampling from the respective prior distributions

Figure 5: As the target answer to the question should be “Apple Inc,” the interface is updated with answers from retrieval models with the most relevant sentence and from Lms (e.g., Distilbert, T5). Also, the highlights are updated by the input perturbation technique.

 

2. For each training epoch: 1. Forward pass: Compute predicted probabilities for each person-item interaction 2. Calculate the negative log-likelihood loss 3. Add regularization terms based on prior distributions 4. Backpropagate the gradients and update model parameters
3. Monitor validation performance and use early stopping to prevent overfitting We use the Adam optimizer for parameter updates due to its efficiency in treating sparse gradients and its ability to adapt the learning rate for each parameter.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Question** & **Answer** \\ \hline Art & Questions about works: Mona Lisa, Raft of the Medussa, B) Questions about forms: color, contour, texture, C) Questions about artists: Picasso, Monet, Leonardo da Vinci, D) Questions about context: Renaissance, post-modernism, expressionism, surrealism \\ \hline Literature Movement & A) Questions about works: novels (1984), plays (The Lion and the Jewel), poems (Rubaiyat), criticism (Poetics), B) Questions about major characters or events in literature: The Death of Anna Karenina, Noboru Wataya, the Marriage of Hippophyta and Theseus \\ \hline Literary Movement & A) Cross-cutting questions (appearances of Overcoats in novels), B) Common link questions (the literary output of a country/region) \\ \hline Geography & A) Questions about location: names of capital, state, river, B) Questions about the place: temperature, wind flow, humidity \\ \hline History & A) When: When did the First World war start?, B) Who: who is called Napoleon of Iran?, C) Where: Where was the first Summer Olympics held?, D) Which: which is the oldest civilization in the world? \\ \hline Science & Questions about terminology: The concept of gravity was discovered by which famous physicist?, Questions about the experiment, Questions about theory: The social action theory believes that individuals are influenced by this theory. \\ \hline TV and Film & Quotes: What are the dying words of Charles Foster Kane in Citizen Kane?, Title: What 1927 musical was the first “talkie”?, Plot: In The Matrix, does Neo take the blue pill or the red pill? \\ \hline Music & Singer: What singer has had a Billboard No. 1 hit in each of the last four decades?, Band: Before Bleachers and fun., Jack Antonoff fronted what band?; Title: What was Madonna’s first top 10 hit? \\ \hline Lifestyle & Clothes: What clothing company, founded by a tennis player, has an alligator logo?, Decoration: What was the first perfume sold by Coco Chan!? \\ \hline Sports & Known facts: What sport is best known as the “king of sports”? \\  & Nationality: What is the national sport of Canada? \\  & Sport player: The classic 1980 movie called Raging Bull is about which real-life boxer? \\  & Country: What country has competed the most times in the Summer Olympics yet has not won any kind of medal? \\ \hline \hline \end{tabular}
\end{table}
Table 8: We list categories of questions along with the subcategories and corresponding examples.

 

* [25] M. C. Bader, A. D. Kessler, and R. J. Sorensen. The design of the model: a new approach to modeling models in multi-agent systems. In _Proceedings of the IEEE Conference on Decision and Control_, pages 1378-1406, 2009.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l l p{142.3pt} p{56.9pt} p{78.0pt}}
**Question** & **Answer** & **Adversarial Type** & **Question Type** & **Grounding** \\ What is a fourth of the 5th Bell number, often seen as an unlucky number? & 13/Thirteen & Logic & Subjectivity “Unlucky” is a subjective term. & The meme can be referred to _many_ titles: "Jordan’s Cigar", "Jordan’s Meme", "Laughing Jordan”, and “Crying Jordan” \\ What substance can cause burns in its gaseous form, lead to vomiting and sweating in high doses, and is the main component by weight in acid rain? & Magic & Specificity & _Many substances_ could cause these effects in the novel portion. & \\ Name the title character of the 2024 Best Picture nominee about a fictional conductor who Leonard Bernstein mentored. & Lydia Tar & Temporal Mis- alignment & Factuality & 2024 Best Picture Nominee _cannot be factually identified_ yet \\ The easternmost state in the US has more than triple its population in lakes and it is known to have good salmon, which state is it? & Multihop Reasoning & Subjectivity, _Good salmon_ is subjective, and _easternmost is misleading and it requires relative position_ of the author, hence non-specific. & _Good salmon_ is subjective, and _easternmost is misleading and it requires relative position_ of the author, hence non-specific. \\  & & & \\ \end{tabular}
\end{table}
Table 10: We annotated whether each question falls into which adversarial and question type. While being adversarial; some questions lack specificity and factuality. Other questions contained subjectivity and specificity.