Progress in AI has been driven by, among other things, the development of challenging large-scale benchmarks like ImageNet BIBREF0 in computer vision, and SNLI BIBREF1, SQuAD BIBREF2, and others in natural language processing (NLP). Recently, for natural language understanding (NLU) in particular, the focus has shifted to combined benchmarks like SentEval BIBREF3 and GLUE BIBREF4, which track model performance on multiple tasks and provide a unified platform for analysis.With the rapid pace of advancement in AI, however, NLU benchmarks struggle to keep up with model improvement. Whereas it took around 15 years to achieve “near-human performance” on MNIST BIBREF5, BIBREF6, BIBREF7 and approximately 7 years to surpass humans on ImageNet BIBREF8, BIBREF0, BIBREF9, the GLUE benchmark did not last as long as we would have hoped after the advent of BERT BIBREF10, and rapidly had to be extended into SuperGLUE BIBREF11. This raises an important question: Can we collect a large benchmark dataset that can last longer?The speed with which benchmarks become obsolete raises another important question: are current NLU models genuinely as good as their high performance on benchmarks suggests? A growing body of evidence shows that state-of-the-art models learn to exploit spurious statistical patterns in datasets BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17, instead of learning meaning in the flexible and generalizable way that humans do. Given this, human annotators—be they seasoned NLP researchers or non-experts—might easily be able to construct examples that expose model brittleness.We propose an iterative, adversarial human-and-model-in-the-loop solution for NLU dataset collection that addresses both benchmark longevity and robustness issues. In the first stage, human annotators devise examples that our current best models cannot determine the correct label for. These resulting hard examples—which should expose additional model weaknesses—can be added to the training set and used to train a stronger model. We then subject the strengthened model to human interference and collect more weaknesses over several rounds. After each round, we both train a new model, and set aside a new test set. The process can be iteratively repeated in a never-ending learning BIBREF18 setting, with the model getting stronger and the test set getting harder in each new round.This process yields a “moving post” dynamic target for NLU systems, rather than a static benchmark that will eventually saturate.Our approach draws inspiration from recent efforts that gamify collaborative training of machine learning agents over multiple rounds BIBREF19 and pit “builders” against “breakers” to learn better models BIBREF20. Recently, Dinan2019build showed that a similar approach can be used to make dialogue safety classifiers more robust. Here, we focus on natural language inference (NLI), arguably the most canonical task in NLU. We collected three rounds of data, and call our new dataset Adversarial NLI (ANLI).Our contributions are as follows: 1) We introduce a novel human-and-model-in-the-loop dataset, currently consisting of three rounds that progressively increase in difficulty and complexity, that includes annotator-provided explanations. 2) We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks. 3) We provide a detailed analysis of the collected data that sheds light on the shortcomings of current models, categorizes the data by inference type to examine weaknesses, and demonstrates good performance on NLI stress tests. The ANLI dataset is available at github.com/facebookresearch/anli/. A demo of the annotation procedure can be viewed at adversarialnli.com.The primary aim of this work is to create a new large-scale NLI benchmark on which current state-of-the-art models fail. This constitutes a new target for the field to work towards, and can elucidate model capabilities and limitations. As noted, however, static benchmarks do not last very long these days. If continuously deployed, the data collection procedure we introduce here can pose a dynamic challenge that allows for never-ending learning.To paraphrase the great bard BIBREF21, there is something rotten in the state of the art. We propose Human-And-Model-in-the-Loop Entailment Training (HAMLET), a training procedure to automatically mitigate problems with current dataset collection procedures (see Figure FIGREF1).In our setup, our starting point is a base model, trained on NLI data. Rather than employing automated adversarial methods, here the model's “adversary” is a human annotator. Given a context (also often called a “premise” in NLI), and a desired target label, we ask the human writer to provide a hypothesis that fools the model into misclassifying the label. One can think of the writer as a “white hat” hacker, trying to identify vulnerabilities in the system. For each human-generated example that is misclassified, we also ask the writer to provide a reason why they believe it was misclassified.For examples that the model misclassified, it is necessary to verify that they are actually correct —i.e., that the given context-hypothesis pairs genuinely have their specified target label. The best way to do this is to have them checked by another human. Hence, we provide the example to human verifiers. If two human verifiers agree with the writer, the example is considered a good example. If they disagree, we ask a third human verifier to break the tie. If there is still disagreement between the writer and the verifiers, the example is discarded. Occasionally, verifiers will overrule the original label of the writer.Once data collection for the current round is finished, we construct a new training set from the collected data, with accompanying development and test sets. While the training set includes correctly classified examples, the development and tests sets are built solely from them. The test set was further restricted so as to: 1) include pairs from “exclusive” annotators that are never included in the training data; and 2) be balanced by label classes (and genres, where applicable). We subsequently train a new model on this and other existing data, and repeat the procedure three times.We employed crowdsourced workers from Mechanical Turk with qualifications. We collected hypotheses via the ParlAI framework. Annotators are presented with a context and a target label—either `entailment', `contradiction', or `neutral'—and asked to write a hypothesis that corresponds to the label. We phrase the label classes as “definitely correct”, “definitely incorrect”, or “neither definitely correct nor definitely incorrect” given the context, to make the task easier to grasp. Submitted hypotheses are given to the model to make a prediction for the context-hypothesis pair. The probability of each label is returned to the worker as feedback. If the model predicts the label incorrectly, the job is complete. If not, the worker continues to write hypotheses for the given (context, target-label) pair until the model predicts the label incorrectly or the number of tries exceeds a threshold (5 tries in the first round, 10 tries thereafter). To encourage workers, payments increased as rounds became harder. For hypotheses that the model predicted the incorrect label for, but were verified by other humans, we paid an additional bonus on top of the standard rate.For the first round, we used a BERT-Large model BIBREF10 trained on a concatenation of SNLI BIBREF1 and MNLI BIBREF22, and selected the best-performing model we could train as the starting point for our dataset collection procedure. For Round 1 contexts, we randomly sampled short multi-sentence passages from Wikipedia (of 250-600 characters) from the manually curated HotpotQA training set BIBREF23. Contexts are either ground-truth contexts from that dataset, or they are Wikipedia passages retrieved using TF-IDF BIBREF24 based on a HotpotQA question.For the second round, we used a more powerful RoBERTa model BIBREF25 trained on SNLI, MNLI, an NLI-version of FEVER BIBREF26, and the training data from the previous round (A1). After a hyperparameter search, we selected the model with the best performance on the A1 development set. Then, using the hyperparameters selected from this search, we created a final set of models by training several models with different random seeds. During annotation, we constructed an ensemble by randomly picking a model from the model set as the adversary each turn. This helps us avoid annotators exploiting vulnerabilities in one single model. A new non-overlapping set of contexts was again constructed from Wikipedia via HotpotQA using the same method as Round 1.For the third round, we selected a more diverse set of contexts, in order to explore robustness under domain transfer. In addition to contexts from Wikipedia for Round 3, we also included contexts from the following domains: News (extracted from Common Crawl), fiction (extracted from BIBREF27, and BIBREF28), formal spoken text (excerpted from court and presidential debate transcripts in the Manually Annotated Sub-Corpus (MASC) of the Open American National Corpus), and causal or procedural text, which describes sequences of events or actions, extracted from WikiHow. Finally, we also collected annotations using the longer contexts present in the GLUE RTE training data, which came from the RTE5 dataset BIBREF29. We trained an even stronger RoBERTa model by adding the training set from the second round (A2) to the training data.The ANLI dataset improves upon previous work in several ways. First, and most obviously, the dataset is collected to be more difficult than previous datasets, by design. Second, it remedies a problem with SNLI, namely that its contexts (or premises) are very short, because they were selected from the image captioning domain. We believe longer contexts should naturally lead to harder examples, and so we constructed ANLI contexts from longer, multi-sentence source material.Following previous observations that models might exploit spurious biases in NLI hypotheses, BIBREF12, BIBREF13, we conduct a study of the performance of hypothesis-only models on our dataset. We show that such models perform poorly on our test sets.With respect to data generation with naïve annotators, Geva2019taskorannotator noted that models might pick up on annotator bias, modelling the annotators themselves rather than capturing the intended reasoning phenomenon. To counter this, we selected a subset of annotators (i.e., the “exclusive” workers) whose data would only be included in the test set. This enables us to avoid overfitting to the writing style biases of particular annotators, and also to determine how much individual annotator bias is present for the main portion of the data. Examples from each round of dataset collection are provided in Table TABREF2.Furthermore, our dataset poses new challenges to the community that were less relevant for previous work, such as: can we improve performance online without having to train a new model from scratch every round, how can we overcome catastrophic forgetting, how do we deal with mixed model biases, etc. Because the training set includes examples that the model got right but were not verified, it might be noisy, posing filtering as an additional interesting problem.The dataset statistics can be found in Table TABREF7. The number of examples we collected increases per round, starting with approximately 19k examples for Round 1, to around 47k examples for Round 2, to over 103k examples for Round 3. We collected more data for later rounds not only because that data is likely to be more interesting, but also simply because the base model is better and so annotation took longer to collect good, verified correct examples of model vulnerabilities.For each round, we report the model error rate, both on verified and unverified examples. The unverified model error rate captures the percentage of examples where the model disagreed with the writer's target label, but where we are not (yet) sure if the example is correct. The verified model error rate is the percentage of model errors from example pairs that other annotators were able to confirm the correct label for. Note that this error rate represents a straightforward way to evaluate model quality: the lower the model error rate—assuming constant annotator quality and context-difficulty—the better the model.We observe that model error rates decrease as we progress through rounds. In Round 3, where we included a more diverse range of contexts from various domains, the overall error rate went slightly up compared to the preceding round, but for Wikipedia contexts the error rate decreased substantially. While for the first round roughly 1 in every 5 examples were verified model errors, this quickly dropped over consecutive rounds, and the overall model error rate is less than 1 in 10. On the one hand, this is impressive, and shows how far we have come with just three rounds. On the other hand, it shows that we still have a long way to go if even untrained annotators can fool ensembles of state-of-the-art models with relative ease.Table TABREF7 also reports the average number of “tries”, i.e., attempts made for each context until a model error was found (or the number of possible tries is exceeded), and the average time this took (in seconds). Again, these metrics represent a useful way to evaluate model quality. We observe that the average tries and average time per verified error both go up as we progress through the rounds. The numbers clearly demonstrate that the rounds are getting increasingly more difficult.Table TABREF13 reports the main results. In addition to BERT BIBREF10 and RoBERTa BIBREF25, we also include XLNet BIBREF30 as an example of a strong, but different, model architecture. We show test set performance on the ANLI test sets per round, the total ANLI test set, and the exclusive test subset (examples from test-set-exclusive workers). We also show accuracy on the SNLI test set and the MNLI development (for the purpose of comparing between different model configurations across table rows) set. In what follows, we briefly discuss our observations.Notice that the base model for each round performs very poorly on that round's test set. This is the expected outcome: For round 1, the base model gets the entire test set wrong, by design. For rounds 2 and 3, we used an ensemble, so performance is not necessarily zero. However, as it turns out, performance still falls well below chance, indicating that workers did not find vulnerabilities specific to a single model, but generally applicable ones for that model class.As already foreshadowed by the dataset statistics, round 3 is more difficult (yields lower performance) than round 2, and round 2 is more difficult than round 1. This is true for all model architectures.Generally, our results indicate that training on more rounds improves model performance. This is true for all model architectures. Simply training on more “normal NLI” data would not help a model be robust to adversarial attacks, but our data actively helps mitigate these.We obtain state of the art performance on both SNLI and MNLI with the RoBERTa model finetuned on our new data The RoBERTa paper BIBREF25 reports a score of $90.2$ for both MNLI-matched and -mismatched dev, while we obtain $91.0$ and $90.7$. The state of the art on SNLI is currently held by MT-DNN BIBREF31, which reports $91.6$ compared to our $92.9$.However, the base (RoBERTa) models for rounds 2 and 3 are outperformed by both BERT and XLNet. This shows that annotators have managed to write examples that RoBERTa generally struggles with, and more training data alone cannot easily mitigated these shortcomings. It also implies that BERT, XLNet, and RoBERTa all have different weaknesses, possibly as a function of their training data (BERT, XLNet and RoBERTa were trained on very different data sets, which might or might not have contained information relevant to the weaknesses)—an additional round with a wider model variety would thus be interesting to investigate as a next step.Even though ANLI training data is different from SNLI and MNLI, adding this data to the training set does not harm performance on those tasks. Furthermore, as Table TABREF21 shows, training only on ANLI is transferable to SNLI and MNLI, but not vice versa. This suggests that methods could successfully be applied for many more consecutive rounds.In order to avoid the possibility that models might pick up on annotator-specific artifacts, a concern raised by Geva2019taskorannotator, we included an exclusive test subset with examples from annotators never seen in the training data. We find that the differences between this exclusive subset and the test set are small, indicating that our models do not over-rely on individual annotator's writing styles.For SNLI and MNLI, concerns have been raised about the propensity of models to pick up on spurious artifacts that are present just in the hypotheses BIBREF12, BIBREF13. To study this in the context of our results and task difficulty, we compare models trained on (context, hypothesis) pairs to models trained only on the hypothesis (marked $H$). Table TABREF21 reports results on the three rounds of ANLI, as well as SNLI and MNLI. The table shows some interesting take-aways:We corroborate that hypothesis-only models obtain good performance on SNLI and MNLI. Performance of such models on ANLI is substantially lower, and decreases with more rounds.On the two rounds where RoBERTa was used as the base model, its performance is not much better than the hypothesis-only model. This could mean two things: either the test data is very difficult, or the training data is not good. To rule out the latter, we trained only on ANLI ($\sim $163k training examples): doing so with RoBERTa matches the performance of BERT on MNLI when it is trained on the much larger, fully in-domain SNLI+MNLI combined dataset (943k training examples), with both getting $\sim $86, which is impressive. Hence, this shows that our new challenge test sets are so difficult that the current state-of-the-art model cannot do better than a hypothesis-only prior.We perform two types of model error analysis. First we evaluate two popular existing test sets that were created to expose model weaknesses, and show that our dataset discourages models from learning spurious statistical facts, relative to other large popular datasets (e.g., SNLI and MNLI). Secondly, we explore, by round, the types of inferences our writers successfully employed to stump models, by performing hand-annotation on 500 examples from each round's development set.Recently, several hard test sets have been made available for revealing the biases NLI models learn from their training datasets BIBREF32, BIBREF17, BIBREF12, BIBREF33. We examined model performance on two of these: the SNLI-Hard BIBREF12 test set, which consists of examples that hypothesis-only models label incorrectly, and the NLI stress tests BIBREF33, in which sentences containing antonyms pairs, negations, high word overlap, i.a., are heuristically constructed. We test our models on these stress tests, after tuning on each test's respective development set to account for potential domain mismatches. For comparison, we also report accuracies from the original papers: for SNLI-Hard we present the results from BIBREF12's implementation of the hierarchical tensor-based Densely Interactive Inference Network BIBREF34 on MNLI, and for the NLI stress tests, we present the performance of BIBREF33's implementation of InferSent BIBREF3 trained on SNLI. Our results are in Table TABREF23.We observe that all of our models far outperform the models presented in original papers for these common stress tests, with our two RoBERTa models performing best. Both perform well on SNLI-Hard and achieve accuracy levels in the high 80s on the `antonym' (AT), `numerical reasoning' (NR), `length' (LN), `spelling error'(SE) sub-datasets, and show marked improvement on both `negation' (NG), and `word overlap' (WO). Training a RoBERTa model also on ANLI appears to be particularly useful for the NR, WO, NG and AT NLI stress tests.A dynamically evolving dataset offers the unique opportunity to track how model error rates change over time. Since each round's development set contains only verified examples, we can investigate two interesting questions: which types of inference do writers employ to fool the models, and are base models differentially sensitive to different types of reasoning? The results are summarized in Table TABREF27.We employed an expert linguist annotator to devise an ontology of inference types that would be specific to NLI. While designing an appropriate ontology of types of inference is far from straightforward, we found that a unified ontology could be utilized to characterize examples from all three rounds, which suggests that it has at least some generalizeable applicability. The ontology was used to label 500 examples from each ANLI development set.The inference ontology contains six types of inference: Numerical & Quantitative (i.e., reasoning about cardinal and ordinal numbers, inferring dates and ages from numbers, etc.), Reference & Names (coreferences between pronouns and forms of proper names, knowing facts about name gender, etc.), Basic Inferences (conjunctions, negations, cause-and-effect, comparatives and superlatives etc.), Lexical Inference (inferences made possible by lexical information about synonyms, antonyms, etc.), Tricky Inferences (wordplay, linguistic strategies such as syntactic transformations/reorderings, or inferring writer intentions from contexts), and reasoning from outside knowledge or additional facts (e.g., “You can't reach the sea directly from Djibouti”). The quality of annotations was also tracked; if a pair was ambiguous or had a label that seemed incorrect (from the expert annotator's perspective), it was flagged. Round 1–3 development sets contained few `Quality' tags; the incidence of quality issues was stable at between 3% and 4% per round. Any one example can have multiple types, and every example contained at least one tag.As rounds 1 and 2 were both built with contexts from the same genre (Wikipedia), we might expect writers to arrive at similar strategies. However, since the model architectures used in the first two rounds differ, writers might be sufficiently creative in finding different exploits in each. For round 3, we expect some difference in reasoning types to be present, because we used source material from several domains as our contexts. In sum, any change between rounds could be due to any of the following factors: inherent differences between data collection, model architectures and model training data, random selection of contexts, or slight differences in writer pool or writer preferences.We observe that both round 1 and 2 writers rely heavily on numerical and quantitative reasoning in over 30% of the development set—the percentage in A2 (32%) dropped roughly 6% from A1 (38%)—while round 3 writers use numerical or quantitative reasoning for only 17%. The majority of numerical reasoning types were references to cardinal numbers that referred to dates and ages. Inferences predicated on references and names were present in about 10% of rounds 1 & 3 development sets, and reached a high of 20% in round 2, with coreference featuring prominently. Basic inference types increased in prevalence as the rounds increased, ranging from 18%–30%, as did Lexical inferences (increasing from 13%–33%). The percentage of sentences relying on reasoning and outside facts remains roughly the same, in the mid-50s, perhaps slightly increasing after round 1. For round 3, we observe that the model used to collect it appears to be more susceptible to Basic, Lexical, and Tricky inference types. This finding is compatible with the idea that the models trained on adversarial data are more impressive and perform better, encouraging writers to devise more creative examples containing harder types of inference in order to stump them.Machine learning methods are well-known to pick up on spurious statistical patterns. For instance, in image captioning, a simple baseline of utilizing the captions of nearest neighbors in the training set was shown to yield impressive BLEU scores BIBREF35. In the first visual question answering dataset BIBREF36, biases like “2” being the correct answer to 39% of the questions starting with “how many” allowed learning algorithms to perform well while ignoring the visual modality altogether BIBREF37, BIBREF38. The field has a tendency to overfit on static targets, even if that does not happen deliberately BIBREF39.In NLI, Gururangan2018annotation, Poliak2018hypothesis and Tsuchiya2018performance showed that hypothesis-only baselines often perform far better than chance. It has been shown that NLI systems can often be broken merely by performing simple lexical substitutions BIBREF15, and that they struggle with quantifiers BIBREF40 and certain superficial syntactic properties BIBREF17. In reading comprehension and question answering, Kaushik2018howmuch showed that question- and passage-only models can perform surprisingly well, while Jia2017adversarial added adversarially constructed sentences to passages, leading to a drastic drop in performance. Many text classification datasets do not require sophisticated linguistic reasoning, as shown by the surprisingly good performance of random encoders BIBREF41. Similar observations were made in machine translation BIBREF42 and dialogue BIBREF43. In short, the field is rife with dataset bias and papers trying address this important problem. This work can be viewed as a natural extension: if such biases exist, they will allow humans to fool the models, adding useful examples to the training data until the bias is dynamically mitigated.Concurrently with this work, Anonymous2020adversarialfilters proposed AFLite, an iterative approach for filtering adversarial data points to avoid spurious biases. Kaushik2019learningdifference offer a causal account of spurious patterns, and counterfactually augment NLI datasets by editing examples to break the model. The former is an example of a model-in-the-loop setting, where the model is iteratively probed and improved. The latter is human-in-the-loop training, where humans are used to find problems with one single model. In this work, we employ both strategies iteratively, in a form of human-and-model-in-the-loop training, to collect completely new examples, in a potentially never-ending loop BIBREF18. Relatedly, lan2017ppdb propose a method for continuously growing a dataset of paraphrases.Human-and-model-in-the-loop training is not a new idea. Mechanical Turker Descent proposes a gamified environment for the collaborative training of grounded language learning agents over multiple rounds BIBREF19. The “Build it Break it Fix it” strategy in the security domain BIBREF44 has been adapted to NLP BIBREF20 as well as dialogue BIBREF45. The QApedia framework BIBREF46 continuously refines and updates its content repository using humans in the loop, while human feedback loops have been used to improve image captioning systems BIBREF47. wallace2018trick leverage trivia experts to create a model-driven adversarial question writing procedure and generate a small set of challenge questions that QA-models fail on.There has been a flurry of work in constructing datasets with an adversarial component, such as Swag BIBREF48 and HellaSwag BIBREF49, CODAH BIBREF50, Adversarial SQuAD BIBREF51, Lambada BIBREF52 and others. Our dataset is not to be confused with abductive NLI BIBREF53, which calls itself $\alpha $NLI, or ART.In this work, we used a human-and-model-in-the-loop entailment training method to collect a new benchmark for natural language understanding. The benchmark is designed to be challenging to current state of the art models. Annotators were employed to act as adversaries, and encouraged to find vulnerabilities that fool the model into predicting the wrong label, but that another person would correctly classify. We found that non-expert annotators, in this gamified setting and with appropriate incentives to fool the model, are remarkably creative at finding and exploiting weaknesses in models. We collected three rounds, and as the rounds progressed, the models became more robust and the test sets for each round became more difficult. Training on this new data yielded the state of the art on existing NLI benchmarks.The ANLI benchmark presents a new challenge to the community. It was carefully constructed to mitigate issues with previous datasets, and was designed from first principles to last longer—if the test set saturates, the field can simply train up a new model, collect more data and find itself confronted yet again with a difficult challenge.The dataset also presents many opportunities for further study. For instance, we collected annotator-provided explanations for each example that the model got wrong. We provided inference labels for the development set, opening up possibilities for interesting more fine-grained studies of NLI model performance. While we verified the development and test examples, we did not verify the correctness of each training example, which means there is probably some room for improvement there.The benchmark is meant to be a challenge for measuring NLU progress, even for as yet undiscovered models and architectures. We plan for the benchmark itself to adapt to these new models by continuing to build new challenge rounds. As a first next step, it would be interesting to examine results when annotators are confronted with a wide variety of model architectures. We hope that the dataset will prove to be an interesting new challenge for the community. Luckily, if it does turn out to saturate quickly, we will always be able to collect a new round.YN and MB were sponsored by DARPA MCS Grant #N66001-19-2-4031, ONR Grant #N00014-18-1-2871, and DARPA YFA17-D17AP00022.

Q: Do they compare human-level performance to model performance for their dataset?

Q: What are the weaknesses found by non-expert annotators of current state-of-the-art NLI models?

Q: What data sources do they use for creating their dataset?

Q: Do they use active learning to create their dataset?