SlowGuess/NAACL_2022_OCR huggingface dataset:
CURRICULUM: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
Zeming Chen Qiyue Gao Rose-Hulman Institute of Technology {chenz16, gaoq}  $@$  rose-hulman.edu
In the age of large transformer language models, linguistic evaluation play an important role in diagnosing models' abilities and limitations on natural language understanding. However, current evaluation methods show some significant shortcomings. In particular, they do not provide insight into how well a language model captures distinct linguistic skills essential for language understanding and reasoning. Thus they fail to effectively map out the aspects of language understanding that remain challenging to existing models, which makes it hard to discover potential limitations in models and datasets. In this paper, we introduce CURRICULUM as a new format of NLI benchmark for evaluation of broad-coverage linguistic phenomena. CURRICULUM contains a collection of datasets that covers 36 types of major linguistic phenomena and an evaluation procedure for diagnosing how well a language model captures reasoning skills for distinct types of linguistic phenomena. We show that this linguistic-phenomena-driven benchmark can serve as an effective tool for diagnosing model behavior and verifying model learning quality. In addition, our experiments provide insight into the limitation of existing benchmark datasets and state-of-the-art models that may encourage future research on re-designing datasets, model architectures, and learning objectives.<sup>1</sup>
With the rising power of pre-trained language models, large-scale benchmarks serve as an important factor driving the future progress of NLP. These benchmarks can provide a tool for analyzing the strengths and weaknesses of pre-trained language models. In recent years, many benchmarks (Wang et al., 2019, 2020; Rajpurkar et al., 2018) have been proposed that offer a diverse set of evaluation objectives. However, recent criticisms have been
Figure 1: We propose a broad-coverage diagnostic benchmark for linguistic-phenomena-driven evaluation. Our benchmark includes both a dataset collection and an evaluation procedure for evaluating model performance and diagnosing linguistic skills captured by a model. We evaluate models fine-tuned on large NLI datasets through four types of diagnostic tests: zero-shot, inoculation, hypothesis-only, and cross-distribution.
made that these benchmarks fail to serve as effective measures of progress in machine learning (Raji et al., 2021). In particular, the task design does not formulate specific linguistic skills required for understanding. They lack the effectiveness in helping researchers understand how certain systems or models work and how they fail. Although many state-of-the-art language models have shown impressive performance on these common benchmarks, their performance degrades considerably on adversarial or out-of-distribution samples (Bras et al., 2020). The performance drop shows that models may not be learning the required linguistic skills for solving the tasks of these benchmarks but exploit spurious dataset biases (Poliak et al., 2018b). Overall, the current benchmark format seems to be more like a contest than a tool that can explain how well a language model captures distinct linguistic skills essential to language understanding and reasoning.
In this paper, we propose a new form of benchmark that serves as a diagnostic evaluation tool for analyzing model linguistic skills. We present CURRICULUM benchmark: a framework for diagnosing
<sup>1</sup>Our code and data are publicly available at https://github. com/eric11eca/curriculum-ling
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3204 - 3219 July 10-15, 2022 ©2022 Association for Computational Linguistics
neural language models through broad-coverage linguistic phenomena. Our benchmark includes (1) a large-scale collection of natural language inference (NLI) datasets covering 36 linguistic phenomena and (2) an evaluation procedure for probing and evaluating how well a language model captures reasoning skills for distinct types of linguistic phenomena. Targeted linguistic phenomena in CURRICULUM range from fundamental properties like named entity and coreference to complex ones like commonsense and deductive reasoning. With the CURRICULUM benchmark, we aim to investigate the following research questions:
- Q1: Do language models trained on benchmark datasets have the ability to reason over a wide range of linguistic phenomena?
- Q2: Are linguistic phenomena missing from the training data recoverable through inoculation (i.e., continuing to train models on a small sample of examples) (Liu et al., 2019a)?
- Q3: Do language models learn a general reasoning skill of a phenomenon through inoculation?
To address the above questions, we empirically analyze NLI models trained on popular benchmark datasets through a pipeline of evaluations that includes: a zero-shot diagnostic test, inoculation retraining, hypothesis-only sanity check, and cross cross-distribution generalization tests.
For Q1, we observe that models trained on benchmark datasets, including adversarial data, do not have the reasoning ability for a large set of linguistic phenomena. Our results show that training on more datasets can help the model learn more types of reasoning but does not help the model acquire complex reasoning skills such as deductive and commonsense reasoning. Our benchmark exposes multiple knowledge gaps in large NLI models regarding diverse linguistic phenomena, particularly in the categories of commonsense and comprehension. For Q2, our analysis provides empirical evidence that exposes the lack of recoverable linguistic phenomena in benchmark datasets and models' inability to learn certain linguistic phenomena. We also show that, on some phenomena, models may rely heavily on spurious dataset bias existing in the hypothesis to reach high accuracy. For Q3, Our experiments show that models can adapt between distributions with different difficulties only on  $22.2\%$  of the phenomena such as Boolean, conditional, and comparative logic. In the majority  $(58.3\%)$  of the phenomena, models fail to gen
eralize when the difficulties of the train and test distributions are different, for example, relational knowledge, puns, and contextual commonsense reasoning. A model's learning performance may not align with its generalization ability, suggesting the lack of a general reasoning skill.
Overall, our proposed benchmark systematically maps out a wide range of specific linguistic skills required for language understanding and inference. We envision linguistic-phenomena-based evaluation to be an integral component of general linguistic intelligence. We hope CURRICULUM can serve as a useful evaluation tool that can map out which aspects of the problem space remain challenging for existing systems and models.
NLU Benchmarks In recent years, multiple large-scale benchmarks for evaluating models' general language understanding performance have been proposed. Similar to our benchmark's task format, SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018) are the two common benchmarks for Natural Language Inference (NLI). GLUE and SuperGLUE are the two most popular benchmarks that aim to provide a straightforward comparison between task-agnostic transfer learning techniques. They cover various task formats, task domains, and training volumes, with datasets all collected from publicly available sources. The construction of our benchmark is similar in that we also collect publicly available datasets from peer-reviewed papers. Adversarial NLI (ANLI) is a new benchmark collected "via an iterative, adversarial human-and-model-in-the-loop procedure." (Nie et al., 2020). ANLI is shown to be a more difficult challenge than previous benchmarks. Different from these benchmarks, our work aims to map out and evaluate specific linguistic skills a model needs for language understanding.
Fine-grained NLU Evaluation On top of large-scale benchmarks, there are several works (Joshi et al., 2020; Tarunesh et al., 2021) contributing to the fine-grained analysis of model performance. They collect data examples from existing benchmarks by attaching taxonomic category labels to each data. Or, they build semi-synthetic data allowing analysis on 17 reasoning dimensions. Our data collection and categorization concepts are similar to them. However, our work covers more linguistic phenomena that are difficult but important such as
Category	Description	Phenomena
Lexical	Testing a model's Word-level reasoning skill on lexical semantic, direct, transitive, and compositional lexical relationships.	Lexical Entailment (lex-ent), Named Entity (ner)
Hypernymy (hyper), Hyponymy (hypo)
Veridicality & Transitivity (transit)
Syntactic	Testing a model's reasoning skill on syntactic structure and compositionality.	Syntactic Alternation (syn-alt), VerbNet (vbn)
Syntactic Variation (syn-var), VerbCorner (vbc)
Semantic	Testing a model's reasoning skill on sentence-level reasoning involving diverse semantic properties: entity relations, context, events, subjectivity, and semantic proto roles.	Sentiment (semi), Relational Knowledge (kg-rel), Puns (puns), Semantic Proto Label (sprl)
Context Alignment (ctx-align), Coreference (coref)
Logical	Testing a model's reasoning skill on logical operations: propositional structure, quantification, negation, and monotonicity reasoning.	Boolean (bool), Counting (count), Conditional (cond)
Comparative (comp), Negation (negat)
Monotonicity (monot), Quantifier (quant)
Analytical	Testing a model's knowledge exploitation ability: drawing accurate conclusions based on domain-specific knowledge, symbolic knowledge, and interpretable reasoning steps.	Entailment Tree (ent-tree)
Analytical Reasoning (analytic)
Commonsense	Testing a model's reasoning skill on commonsense knowledge independent of cultural and educational background.	Physical (physic), Social (social), HellaSwag (swag)
Contextual Commonsense Reasoning (cosmo)
Comprehension	Testing a model's reasoning skill on complex reading comprehension and inference, covering aspects of semantic, context, logic, and numerical	Event Semantics (ester), Discrete Reasoning (drop)
Deductive Reasoning (logi)
Long Contextual Reasoning (control)
Special	Testing a model's everyday reasoning skill. Including non-monotonic reasoning about valid but defeasible hypothesis from hypothetical context and spatial-temporal reasoning.	Spatial Reasoning (spat), Temporal Reasoning (temp)
Defeasible Reasoning (defeas)
Counterfactual Reasoning (counter)
Table 1: This table lists the eight categories of linguistic phenomena covered by our dataset collection. We provide a brief introduction for each category describing the types of linguistic skills they intend to evaluate. We also list the dataset names and abbreviations each category contains.
commonsense and non-monotonic reasoning.
Challenge Datasets for NLU Many challenge datasets have been developed to evaluate models on specific linguistic skills for understanding. These datasets are in different formats such as NLI, Question Answering (QA), and Reading Comprehension (RC). They target a large set of skills including monotonicity (Yanaka et al., 2019a), deductive logic (Liu et al., 2020), event semantics (Han et al., 2021), physical and social commonsense (Sap et al., 2019; Bisk et al., 2019), defeasible reasoning (Rudinger et al., 2020), and more. Our work brings together a set of challenge datasets to build a benchmark covering a large set of specific linguistic skills. We also merge different evaluation methods proposed by these works into a complete evaluation pipeline for our benchmark.
Probing Linguistic Knowledge Several works have found evidence that pre-trained models' representations encode knowledge about linguistic phenomena. Tenney et al. (2019) probe contextual representations from four pre-trained language models through the edge-probing method across tasks ranging from syntactic and semantic phenomena. They find that pre-trained models encode rich in
formation on syntactic phenomena but only weakly encode information on semantic tasks compared to non-contextual baselines. Chen and Gao (2021)'s linguistic-information-probing framework extends the edge-probing study by focusing on different semantic phenomena that are important for logical inference in natural language. Their results show that pre-trained contextual embeddings encode more linguistic information on simple semantic phenomena than complex phenomena. Our work is partly motivated by this line of work in which our evaluation is based on the fact that pre-trained models can capture specific linguistic skills from learning.
Other work investigates if models use specific linguistic skills to solve a downstream task. The DNC benchmark (Poliak et al., 2018a) provides a collection of datasets for analyzing if models use distinct linguistic phenomena to conduct natural language inference. Several tasks in our benchmark come directly from this collection. However, our benchmark covers a wider range of linguistic phenomena from more categories than DNC. In particular, our benchmark contains semantic phenomena and includes phenomena from fundamental linguistic properties to complex reasoning types. In addition, our benchmark includes a sys
tematic evaluation methodology that allows a more in-depth analysis of model behavior.
3 The CURRICULUM Benchmark
3.1 A New Form of Benchmark
Recently, Raji et al. (2021) suggested that good benchmark construction should focus on mapping out a specific set of linguistic skills required for language understanding. They recommend a future benchmark should provide interpretation on how systems work and how they fail on particular aspects of a problem space. Following this suggestion, we propose a new form of benchmark: linguistic-phenomena-driven evaluation. Our main objective is to reformulate the benchmark not simply to be a scoreboard for SOTA model contest but rather as a real measurement and standardization tool for (1) analyzing model performance, (2) exposing model and dataset weakness and (3) providing insights for future research directions.
The curriculum benchmark aims to map out a specific set of linguistic skills required for language understanding. Our benchmark will serve as a diagnostic framework for linguistic-phenomena-driven probing and evaluation. The targeted linguistic skills should range from fundamental linguistic properties to complex reasoning types. Our linguistic phenomena selection is motivated by three benchmarks: GLUE Diagnostic, Rainbow, and DNC. In addition, we include many more phenomena focusing on complex reasoning types such as deductive logic and analytical thinking. Our finalized benchmark covers eight categories of linguistic phenomena. Each linguistic phenomenon is considered one task, and one should train, evaluate, and analyze models on each phenomenon individually. We briefly describe the types of reasoning skill each category focus on in Table 1. Appendix A and B shows a list of references and dataset details for the train and test datasets used for each linguistic phenomenon.
3.2 Dataset
We collect many challenge NLI or NLU datasets and filter them individually with the following criteria: (1) We focus on datasets that evaluate a specific or a set of specific linguistic phenomena. (2) We focus on English monolingual datasets that are institutional and publicly available. (3) We exclude tasks that require domain-specific knowledge that we would not expect a model to learn through pre
training, such as medical knowledge. We finalize our selection with 36 datasets. Figure 1 shows a detailed ontology of our selected linguistic phenomena and their abbreviations. Our motivation for dataset selection is mainly based on the linguistic phenomena categories that we aim to cover which will range from a simple to complex setting.
3.3 Unified Task Format
We unified the task formats into a single linguistic task, Natural Language Inference (NLI). NLI is a task for Natural Language Understanding. The task requires a model to classify the logical relationship between premise and a hypothesis. This logical relationship can either be Entailment (premise is true implies the hypothesis is absolutely true), Contradiction (premise is true implies the hypothesis is absolutely false), and Neutral (one cannot determine if the hypothesis is true or false based on the premise) (Dagan et al., 2013). We select NLI as the universal task format because NLI often serves as a general evaluation method for models on different downstream tasks. A model would need to handle nearly the full complexity of natural language understanding in order to solve the NLI task (Poliak et al., 2018b). Our benchmark contains two types of NLI problems: (1) the 3-way NLI with Entailment, Contradiction, and Neutral; (2) the 2-way NLI with Entailed and Not-Entailed. Each example has a premise and a hypothesis with 2-way or 3-way labels.
3.4 Automatic Recast
To convert non-NLI datasets into the NLI task format, we follow the dataset recast procedure (Poliak et al., 2018b): automatically convert from non-NLI datasets with minimum human intervention. We design algorithmic ways to generate sentence pairs from the input text and convert the original labels into the NLI labels. Question Answering (QA) and Reading Comprehension (RC) are the two major tasks we need to convert. To convert datasets into NLI format, we follow the standard procedure (Khot et al., 2018). In QA datasets, if choices are given as declarative statements, we consider them as hypotheses and the question context as the premise. If choices are given as phrases answering the question, we concatenate the context and question to form a premise and consider the answers as hypotheses. Several datasets are tasks with free-response problems, and an answer can only be converted to an entailed hypothesis. To
P	Iv	P	Iv	P	Iv
lex-ent	0.31	transit	0.41	hyper	-0.99
hypo	-0.10	ner	0.19	vbn	0.55
vbc	-0.40	syn-alt	0.10	syn-var	0.11
bool	1.12	cond	1.13	cont	0.75
comp	0.98	negat	1.13	quant	0.78
monot	-1.57	kg-rel	0.05	coref	-0.38
senti	0.42	ctx-align	-0.79	puns	0.14
sprl	-0.11	ent-tree	0.50	analytic	0.00
temp	0.10	spat	0.49	counter	0.47
defeas	-0.39	social	-0.40	physic	-0.17
swag	-0.66	cosmo	-0.57	drop	0.19
ester	-0.10	logi	-0.71	control	-0.07
generate non-entailed hypotheses, we use several techniques during recasting. We show more details on our conversion techniques in Appendix C. As a sanity check on our resulting datasets, we empirically find low performance on standard partial-input baselines (Poliak et al., 2018b), suggesting that our conversion yields data of high quality.
3.5 Dataset Difficulty
To enhance our benchmark to provide more information on each dataset for in-depth evaluation and analysis, we provide each phenomenon a difficulty level. We use the predictive  $\nu$ -information (Ethayarajh et al., 2021) as a measurement for dataset difficulty. The  $\nu$ -information can measure how much information an input variable X can provide about Y when constrained to functions  $\nu$ . Intuitively, more usable infromation X can provide, the easier a dataset is for the functions  $\nu$ . Formally, let  $\varnothing$  denote a null input that provides no information about Y and  $\nu$  as a predictive family, we can compute the  $\nu$ -information  $I_v(X \to Y)$  as follows:
$$
\mathrm {H} _ {v} (Y) = \inf  _ {f \in \mathcal {V}} \mathbb {E} [ - \log f [ \varnothing ] (\mathrm {Y}) ]
$$
$$
\mathrm {H} _ {v} (\mathrm {Y} | X) = \inf  _ {f \in \mathcal {V}} \mathbb {E} [ - \log f [ \mathrm {X} ] (\mathrm {Y}) ]
$$
$$
\mathrm {I} _ {v} (\mathrm {X} \rightarrow \mathrm {Y}) = \mathrm {H} _ {v} (Y) - \mathrm {H} _ {v} (Y | X)
$$
where  $\mathrm{X},\mathrm{Y}$  denote random variables with sample spaces  $\mathcal{X},\mathcal{Y}$ . According to Ethayarajh et al. (2021),  $\varnothing$  can be an empty string here as  $f[\varnothing]$  models the label entropy. This framework can naturally adapt to the calculation of the point-wise
Table 2: Dataset difficulty measured by the amount of usable information  $(\mathrm{I}_v)$  from input data instances. The lower  $\mathrm{I}_v$  is the more difficulty a dataset will be for the model.  $\mathcal{P}$  here are the abbreviations of linguistic phenomena listed in Table 1
Name	Model	Train/Test	Accuracy
roberta-mnli	RoBERTa(Liu et al., 2019b)	MNLI/MNLI	90.2%
bart-mnli	BART(Lewis et al., 2020)	MNLI/MNLI	89.9 %
roberta-anli-mix	RoBERTa	SNLI, MNLI,FEVER, ANLI/ANLI	53.7 %
xlnet-anli-mix	XLNet(Yang et al., 2019)	SNLI, MNLIFEVER, ANLI/ANLI	55.1 %
Table 3: Details on models used in our experiments. All four models are large models and publicly available.
$\nu$  -information (PVI) where we measure the difficulty of each data example. Given a training dataset  $\mathcal{D}_{train} = \{(x_i,y_i)\}_{i = 1}^n$  , and the predictive family  $\nu$  , the PVI of a data instance  $(x,y)\in \mathcal{D}_{train}$  is computed as:
$$
\operatorname {P V I} (x \rightarrow y) = - \log_ {2} f [ \varnothing ] (y) + \log_ {2} f ^ {\prime} [ x ] (y),
$$
where  $\varnothing$  is an empty string (null input) and  $\{f, f'\} \subseteq \mathcal{V}$ .  $f'$  and  $f$  are models fine-tuned from  $\mathcal{D}_{train}$  and  $\{(\varnothing, y_i) | (x_i, y_i) \in \mathcal{D}_{train}\}$  respectively. The  $\mathcal{V}$ -information framework can also serve as a difficulty measurement for datasets and can be computed explicitly by averaging over PVI:
$$
\mathrm {I} _ {v} (\mathrm {X} \rightarrow \mathrm {Y}) = \frac {1}{n} \sum_ {i} \mathrm {P V I} (x _ {i} \rightarrow y _ {i})
$$
As Table 2 shows, the difficulty level ranges from negative to positive. The higher the  $\nu$ -information is, the easier a dataset is for the model.
Dataset Controlled Split For our model evaluation pipeline, we are interested in verifying model's ability to learn a generalizable reasoning skill on linguistic phenomena. In particular, we want to check if a model can generalize when its training and testing data distributions have different measurement of difficulty. Thus, we need to conduct controlled split on datasets based on the point-wise difficulty, i.e. the point-wise  $\nu$ -information of their data examples. We first calculate the  $\mathrm{PVI}(x \to y)$  for each phenomenon dataset, then we split each dataset into two portions: simple and hard, based on the calculation of each example's PVI.
4 Evaluation Methodology
We define an evaluation process for the CURRICULUM benchmark that aims to bring different types of evaluation and diagnosing methods used by previous challenge NLI datasets. Following Raji et al.
(2021)'s suggestion, we want our evaluation process to both to analyze the model output in detail and explore which aspects of the inference problem space remain challenging to current models.
Zero-shot Diagnostic Test This test is motivated by the diagnostic test in GLUE. We focus on providing fine-grained analysis of zero-shot system performance on a broad range of linguistic phenomena. We follow the GLUE diagnostic dataset and use the Matthews Correlation Coefficient (MCC) (Jurman et al., 2012) as the evaluation metric. MCC computes the correlation coefficient of the predicted labels and the true labels. The correlation coefficient value is between -1 and +1. A coefficient of +1 indicates a perfect prediction. A 0 indicates average random prediction A -1 indicates the classifier always miss-classifies. MCC is perfectly symmetric, so it can be used even if the dataset has classes with different sizes.
Inoculation by Fine-tuning We use inoculation (Liu et al., 2019a) to further analyze model failures on target linguistic phenomena. This method fine-tunes the model on a down-sampled training section of a phenomenon dataset (inoculation). One can interpret inoculation performance in two ways:
1. Good performance: the original training set of the model, prior to inoculation, did not sufficiently cover the target phenomenon, but it is recoverable through through additional training on a small sample of data.
2. Poor performance: there exists a model weakness to handle the target phenomenon.