SlowGuess/NAACL_2022_OCR huggingface dataset:
Max Bartolo†‡ Tristan Thrush‡ Sebastian Riedel†‡
Pontus Stenetorp† Robin Jia§‡ Douwe Kiela‡
†UCL §USC ‡Facebook AI Research
m.bartolo@cs.ucl.ac.uk
In Dynamic Adversarial Data Collection (DADC), human annotators are tasked with finding examples that models struggle to predict correctly. Models trained on DADC-collected training data have been shown to be more robust in adversarial and out-of-domain settings, and are considerably harder for humans to fool. However, DADC is more time-consuming than traditional data collection and thus more costly per annotated example. In this work, we examine whether we can maintain the advantages of DADC, without incurring the additional cost. To that end, we introduce Generative Annotation Assistants (GAAs), generator-in-the-loop models that provide real-time suggestions that annotators can either approve, modify, or reject entirely. We collect training datasets in twenty experimental settings and perform a detailed analysis of this approach for the task of extractive question answering (QA) for both standard and adversarial data collection. We demonstrate that GAAs provide significant efficiency benefits with over a  $30\%$  annotation speed-up, while leading to over a 5x improvement in model fooling rates. In addition, we find that using GAA-assisted training data leads to higher downstream model performance on a variety of question answering tasks over adversarial data collection.
Natural language processing has become increasingly reliant on large datasets obtained using crowd sourcing. However, crowdsourcing as an unconstrained annotation approach is known to result in machine-exploitable annotator artefacts (Jia and Liang, 2017; Schwartz et al., 2017; Gururangan et al., 2018; Geva et al., 2019), leading to poor out-of-distribution generalisation (Chen et al., 2016; Weissenborn et al., 2017; Yogatama et al., 2019; McCoy et al., 2019). Dynamic Adversarial Data Collection (DADC) aims to address these issues
After a slow start to the 2008-09 season, the Bruins won 17 of their next 20 games, leading many to see them as a revival of the "Big Bad Bruins" from the 1970s and 1980s. During the 2009 All-Star Weekend's Skills Competition, captain Zdeno Chara fired the NHL's then-fastest measured "hardest shot" ever, with a clocked in speed of  $105.4\mathrm{mph}$  (169.7 km/h) velocity. (Chara has since broken his own record three times, two of those on the same night.) The number of injured players in the season...
A:105.4 mph
Figure 1: Example interaction between an annotator and the models in the loop. The annotator selects an answer from the passage, for which the Generative Annotation Assistant (GAA) prompts a question. The annotator can then freely modify the question and/or answer, or generate another prompt. In the adversarial data collection setting, a model-in-the-loop provides predictions with the aim of encouraging annotators to find model-fooling examples. In the answer prompting setting, an answer suggestion is prompted by the assistive model instead of being selected by the annotator.
A:Zdeno Chara
by introducing state-of-the-art models into the data collection loop and asking human annotators to produce examples that these models find challenging (Kiela et al., 2021). The intuition behind this approach is that it leads human annotators to better explore the space of possible examples. Previous work has found that DADC leads to improved model robustness on adversarial datasets (Nie et al., 2020; Bartolo et al., 2020), increased sample diversity (Bartolo et al., 2020; Wallace et al., 2021), better training data (Wallace et al., 2021) and better domain generalisation (Bartolo et al., 2021).
Despite these advantages, a downside to DADC is that it increases the human effort necessary to annotate a single example and thus the overall annotation cost. In fact, to date, only a limited number of
Human Language Technologies, pages 3754 - 3767
large-scale training datasets have been produced using DADC and its application has been primarily restricted to producing challenge sets or as additional training data to improve the performance of models already trained on non-DADC curated datasets. To make better use of DADC data, Bartolo et al. (2021) propose generating synthetic adversarial training sets to further improve model robustness. However, this approach inevitably limits example diversity as it relies on examples ultimately generated by a model with no additional human input, and provides no guarantees that useful synthetic examples would transfer across target adversary models of varying capabilities or across annotation rounds.
In this work, we propose assisting annotators by having generative models aid human annotators in the data collection loop. Concretely, we utilise a Generative Annotation Assistant (GAA) model that provides prompt suggestions to crowd-workers, while allowing full flexibility for edits and rewrites to support example generation while still allowing for human creativity as shown in Figure 1. We explore GAAs in a broad range of experimental settings, including standard and adversarial data collection approaches, training on various source datasets, and employing sampling methodologies based on likelihood, adversarial feedback, and uncertainty. We showcase the value of this approach on the task of extractive question answering (QA), and find that GAAs can help improve both the standard and adversarial data collection paradigms. We find considerable efficiency gains, with over  $30\%$  observed annotation speed-ups, as well as improved data effectiveness with up to a  $6.1\mathrm{F}_1$  improvement in downstream performance over adversarial data collection.
2.1 Dynamic Adversarial Data Collection (DADC)
There is a rich body of recent work showing the benefits of dynamic adversarial data collection in model evaluation (Yang et al., 2017; Dua et al., 2019; Dinan et al., 2019; Nie et al., 2020; Bartolo et al., 2020; Kiela et al., 2021; Wallace et al., 2021), although the approach has been challenged for not necessarily leading to better generalisation on non-adversarial test sets (Kaushik et al., 2021) and being sensitive to the choice of model that was used in the loop (Bowman and Dahl, 2021; Phang et al., 2021). This work builds on previ
ous work in adversarial data collection methods for QA (Bartolo et al., 2020), and work investigating the use of question generation models to create synthetic adversarial data to improve QA model robustness (Bartolo et al., 2021).
2.2 Generative Model Annotation Support
A long line of prior work has trained generative models for question answering (Du et al., 2017; Du and Cardie, 2018; Zhao et al., 2018; Lewis and Fan, 2019; Alberti et al., 2019; Puri et al., 2020; Yang et al., 2020; Bartolo et al., 2021; Lewis et al., 2021). In many cases, these approaches filter out questions that an external QA model gets wrong, in order to ensure correctness of the generated questions; our filtering strategies instead focus on generated questions that QA models get wrong as we hypothesise that these would serve as more useful initial prompts to human annotators.
Generative models have also been used to aid experts with writing contrast sets (Wu et al., 2021; Ross et al., 2021), but to the best of our knowledge, this is the first work to investigate the use of generative annotation assistants for crowdworkers directly in the annotation loop for NLP. Recent work on supporting crowdworkers for textual entailment in a non-adversarial setting shows no improvements on downstream transfer performance over baseline, albeit with reductions in previously observed issues with annotation artefacts (Bowman et al., 2020). Subsequent work highlights the need for further data collection efforts focusing on improving writing-based annotation processes (Vania et al., 2021), which we aim to investigate in this work. Separately, Ettinger et al. (2017) provide breakers with the ability to minimally edit original data to identify the boundaries of system capabilities, while Potts et al. (2021) analyse the use of prompts to assist crowdworkers in beating a model in the loop for sentiment analysis. In both cases, prompts are sourced from existing datasets and are not generated on the fly.
2.3 Active Learning and Weak Supervision
Active learning approaches have been used to accelerate annotation (Tsuruoka et al., 2008), although this typically assumes access to a pool or stream of unlabelled data for which the learning algorithm can query labels (Settles, 2009). In our setting, no unlabelled questions are provided, necessitating the use of a generative model to suggest questions instead. Moreover, our annotators are free to edit
Instructions (Click to expand) Can you ask Questions that fool the AI into giving the wrong answer?
Figure 2: The Annotation Interface used for data collection. This example shows a question generated using a generative assistant trained on the AdversarialQA data and selected an adversarial sampler, which successfully allowed the annotator to beat the QA model in the loop.
Remember, the goal is to find an example that the AI gets wrong but that another person would get right. Load time may be slow, please be patient.
and browse generated questions, whereas annotators in active learning typically only provide labels and have no choice in what to label. Some of our sampling and filtering strategies based on entropy are inspired by uncertainty sampling, a standard active learning algorithm (Lewis and Gale, 1994).
Our study focuses on the effects of incorporating generative annotation assistants, and understanding their interactions with annotators and discriminative models-in-the-loop in a DADC context for QA. We provide crowdworkers with a short passage from Wikipedia and ask them to write five questions and highlight the span in the passage that best answers the question for each (see Figure 2). We pay workers equally across experiment modes to avoid creating an incentive imbalance and pay out an additional bonus for each question that successfully beats the discriminative QA model i.e., for each question that the model fails to answer correctly. Finally, we validate all collected examples using a separate worker pool that also undergoes rigorous onboarding and validation. We ask three of these additional workers to report on the validity of each annotated example.
Selected Passages We select passages from KILT (Petroni et al., 2021) to allow for the possibility of future investigation into cross-domain and task transfer. We restrict KILT passages to
those with between 100 and 600 tokens that are used by at least 5 of the KILT tasks. Furthermore, we filter out any passages with any 8-gram overlap (after normalisation) to the SQuAD1.1 training or development sets, seeking to ensure that all passages used in our study are novel and previously unseen by the discriminative QA models in the loop. This leaves a total of 10,109 passages from 421 Wikipedia pages. We retain and supply all passage-relevant KILT metadata (such as IDs and provenances) with our collected datasets to facilitate future work.
Model-in-the-Loop The discriminative QA model in the loop is ELECTRALarge (Clark et al., 2020) trained on SQuAD1.1 and AdversarialQA, and enhanced using SynQA to improve adversarial robustness as investigated by Bartolo et al. (2021). This model represents the best-performing model on the Dynabench (Kiela et al., 2021) leaderboard at the time of conducting this study, obtaining a word-overlap  $\mathbf{F}_1$  score of  $94.5\%$  on the SQuAD1.1 dev set, and represents the state-of-the-art on AdversarialQA achieving  $77.6\%$  on the  $\mathcal{D}_{\mathrm{BiDAF}}$  subset,  $71.5\%$  on  $\mathcal{D}_{\mathrm{BERT}}$ , and  $63.2\%$  on  $\mathcal{D}_{\mathrm{RoBERTa}}$ .
Generator-in-the-Loop For our generative models, we use the fairseq (Ott et al., 2019) implementation
<sup>1</sup>You can interact with this model at https://dynabench.org/models/109.
tation of  $\mathrm{BART}_{\mathrm{Large}}$  (Lewis et al., 2020), and fine-tune the decoder to generate questions conditioned on the passage and the answer highlighted by the annotator. To provide annotators with a diverse set of questions, we decode using nucleus sampling with  $top_{p} = 0.75$ , as decoding using standard beam search results in questions which are more similar to each other and therefore likely to be less useful as question prompts to annotators. To speed up inference and model-annotator interaction, we preemptively identify answer candidates for each passage and generate questions to build up a large cache from which we serve questions during annotation. Once there are no questions remaining in the cache for a particular answer, or if the annotator selects an answer that is not in the cache, we fall back to querying the generative model in real-time. In this work, we investigate generative assistants trained on three different sources of questions: SQuAD1.1, AdversarialQA, and the combination of both SQuAD and AdversarialQA.
Question Sampling We investigate three different selection strategies for presenting the generated questions as prompts to annotators: i) generator likelihood samples candidates in the order prescribed by the generative model's associated likelihood values; ii) adversarial sampling selects generated questions in order of the least word-overlap  $\mathrm{F}_1$  scores when queried against the discriminative QA model; and iii) uncertainty sampling is inspired by active learning and selects generated questions in order of the least span selection confidence when queried against the QA model. The latter two provide an interesting trade-off for exploration as we would expect the quality of the generated questions to be worse than if sampled based on likelihood. However, we hope that such prompts could serve to inspire annotators and provide a "starting point" beyond the answering capabilities of the QA model, irrespective of correctness. We hypothesise that modifying such examples might be a more effective process for annotators to undertake than when starting from higher quality but less model-confusing prompts, and investigate this question thoroughly.
Answer Prompts We also investigate the effects of abstracting away the answer selection task from the annotator. To identify potential candidate answers, we use Self-Attention Labelling (SAL) (Bartolo et al., 2021) and investigate providing annotators with both answer prompts as well as the
corresponding generated questions.
Experimental Settings In total, there are twenty different experimental settings involving combinations of the above-mentioned pipeline components. We collect 2,000 validated training examples for each of these settings, for a total of 40,000 examples. For downstream evaluation we train ELECTRALarge QA models on the training datasets collected for each setting, and perform identical model selection and hyper-parameter tuning.
Annotation Interface We use an adaptation of the Dynabench (Kiela et al., 2021) QA interface that allows annotators to interact with the models in the loop, and further allows them to edit and modify generated questions and answers as required. The same base interface is used across experimental settings and only varied minimally depending on the current setting, for example by changing the title and instructions in the adversarial annotation setting, or by adding a "Generate Question" button when the setting involves GAAs. In the GAA settings, annotators are not informed what generative model they are interacting with, or what sampling mechanism is being used.
Crowdsourcing Protocol We use Amazon Mechanical Turk to recruit workers for this study and run all experiments using Mephisto.2 To ensure proficiency in English, crowdworkers are required to be based in Canada, the UK, or the US. They are also required to have a Human Intelligence Task (HIT) Approval Rate greater than  $98\%$ , have previously completed at least 1,000 HITs, and undergo a dedicated onboarding process. Workers were randomly assigned to one of the possible experiment modes and were all presented with passages sampled from the same set, for which they were tasked with writing and answering five questions. All collected questions were than validated for correctness by a separate group of crowdworkers. We collect three validations per question and use this information, along with manual verification of a subset of the annotated examples, to maintain a high level of quality and remove examples from workers with less than an  $80\%$  validity rate. We calculate the reliability of agreement between invalidators using Fleiss' kappa at 0.46. Workers were provided an additional bonus for each example validated as having successfully fooled the model in the adversarial data collection settings. In total,
$^{2}$ github.com/facebookresearch/Mephisto
Adversary-in-the-loop?	t (s)	vMER (%)	t/vMFE (s)	SQuADdev	DBiDAF	DBERT	DRoBERTa	MRQA
X	57.223.9	0.63	11,537	85.7	43.5	28.3	21.1	52.0
✓	61.527.1	1.86	4,863	85.0	53.0	34.2	26.9	58.8
Table 1: Baseline results comparing standard and adversarial data collection.  $t$  shows the median time taken per example in seconds and median absolute deviation (subscript).  $vMER$  is the validated model error rate.  $t / vMFE$  is the time per validated model-fooling example. Lower is better for the time-dependent metrics. Downstream evaluation is measured by training an ELECTRALarge QA model on the collected datasets and evaluating  $F_1$  scores on the SQuAD1.1 dev set, the AdversarialQA test sets, and the MRQA dev sets for domain generalisation.
1,931 workers participated in the study, with 1,559 contributing to the final datasets. We also continuously validate both annotators and validators based on signals such as repetitiveness, agreement, and manual checks.
Evaluation We evaluate the outcomes in each of the experimental settings by a selection of metrics:
i. median time per example as a measure of annotation efficiency and where a lower time taken is better;
ii. validated Model Error Rate (vMER) (Bartolo et al., 2021) which evaluates the effectiveness of annotators at generating valid question-answer pairs that the QA model in the loop fails to answer correctly;
iii. median time per validated model-fooling example which serves as a single metric incorporating both method efficiency and effectiveness and thus provides a convenient metric for comparison across the various experimental settings; and
iv. downstream effectiveness in which we evaluate the performance (by word-overlap  $\mathrm{F_1}$  score) of a QA model trained on the data collected in each of the experimental modes on the standard SQuAD1.1 benchmark, on the AdversarialQA benchmark, and in terms of domain generalisation ability on the MRQA (Fisch et al., 2019) dev sets.
Lower values are better for the time-dependent metrics, however, from the perspective of training data we consider a higher vMER to be better guided by the performance benefits observed for adversarial over standard data collection. This is corroborated by comparison with downstream results.
Our study allows us to perform a thorough investigation into both the efficiency and effectiveness of
the different data annotation methodologies. It also allows us to build on work investigating the various differences between standard and adversarial data collection (Kaushik et al., 2021).
4.1 Standard versus Adversarial Data Collection
The standard and adversarial data collection settings we use as baselines do not make use of GAAs, and are designed to replicate the SQuAD1.1 (Rajpurkar et al., 2016) and AdversarialQA (Bartolo et al., 2020) annotation setups as closely as possible. However, in contrast to AdversarialQA, our setting only provides annotators with a financial incentive to try to beat the model in the loop through the use of a bonus, and does not restrict annotators to only submitting model-fooling examples.
The results, shown in Table 1, highlight the differences between the two annotation approaches. As expected, standard data collection is more efficient in terms of the time taken per example, as there is no requirement for annotators to make any effort to try to beat a model. However, the efficiency differences are not as large as seen in settings where annotators have to submit model-fooling examples (Bartolo et al., 2020).
We also find considerable benefits from adversarial data collection in terms of the validated model error rate and subsequent downstream performance. As observed by Bartolo et al. (2020), adversarial data collection is more effective on adversarial test sets and aids domain generalisation, with slight performance degradation in the standard evaluation setting, which may be mitigated by increasing the amount of training data or combining with non-adversarial training data (for detailed results, refer to Appendix B). The combined performance across evaluation settings is considerably higher for adversarial data collection.
We note that the training data sizes in both these experimental settings are relatively small, and the benefits of adversarial data collection have been
Sampling Strategy	t (s)	vMER (%)	t/vMFE (s)	SQuADdev	DBiDAF	DBERT	DRoBERTa	MRQA
Likelihood	37.521.4	0.62	8,708	85.5	41.8	25.3	20.3	53.6
Adversarial	55.620.9	4.02	1,760	84.7	45.5	26.1	20.0	54.3
Uncertainty	63.128.6	2.77	3,018	83.2	45.5	28.2	21.9	53.2
Table 2: Results for the investigation into supporting standard data collection using GAAs. Since this setting assumes no access to adversarially-sourced data, we use a generative model trained only on questions from SQuAD1.1. There is no adversarial QA model in the loop in this setting.
shown to be more pronounced in the low data regime, likely due to increased example diversity. Furthermore, while the passages used in this study are sourced from Wikipedia, there may exist characteristic differences between these and the passages used in SQuAD.
We also observe considerably lower (i.e., better) adversarial human evaluation vMER scores achieved for our synthetically-augmented  $\mathrm{ELECTRA}_{\mathrm{Large}}$  model-in-the-loop compared to the  $8.8\%$  reported for RoBERTa by Bartolo et al. (2021). We hypothesise that this is primarily due to two factors: the improved robustness of ELECTRA in comparison to RoBERTa, and more tightly-controlled example validation. For further evidence of the improved adversarial robustness of ELECTRA, refer to Appendix C.
4.2 Improving Standard Data Collection
We now investigate whether it might be possible to improve standard data collection practices using generative assistants - can we achieve similar performance to adversarial data collection without access to any adversarial data?
We therefore use a GAA trained on SQuAD1.1, and investigate the three sampling techniques namely: likelihood, adversarial, and uncertainty sampling. Results are shown in Table 2. We find that using a GAA with likelihood sampling considerably improves the efficiency of the annotation process in comparison to the standard data collection baseline in Table 1. It also gives comparable vMER results and downstream QA performance.
Furthermore, both the adversarial and uncertainty sampling strategies prove effective. While the reduction in time taken per example is not as substantial as for standard likelihood sampling, and is comparable to the standard data collection baseline, the vMER - an indicator of the diversity of the collected training data - is substantially improved and outperforms the adversarial data collection baseline. The downstream results are also promising, providing slight improvements over the
standard data collection setting, particularly with regards to domain generalisation. They start to make progress towards the values obtained for the adversarial data collection baseline although, despite the improved vMER, overall downstream performance is considerably higher in the adversarial data collection setting.