SlowGuess/NAACL_2022_OCR huggingface dataset:
Roy Schwartz Gabriel Stanovsky
School of Computer Science, The Hebrew University of Jerusalem {roy.schwartz1, gabriel.stanovsky}@mail.huji.ac.il
Recent work has shown that deep learning models in NLP are highly sensitive to low-level correlations between simple features and specific output labels, leading to overfitting and lack of generalization. To mitigate this problem, a common practice is to balance datasets by adding new instances or by filtering out "easy" instances (Sakaguchi et al., 2020), culminating in a recent proposal to eliminate single-word correlations altogether (Gardner et al., 2021). In this opinion paper, we identify that despite these efforts, increasingly-powerful models keep exploiting ever-smaller spurious correlations, and as a result even balancing all single-word features is insufficient for mitigating all of these correlations. In parallel, a truly balanced dataset may be bound to "throw the baby out with the bathwater" and miss important signal encoding common sense and world knowledge. We highlight several alternatives to dataset balancing, focusing on enhancing datasets with richer contexts, allowing models to abstain and interact with users, and turning from large-scale fine-tuning to zero- or few-shot setups.
Effective human communication relies on our ability to understand extra-textual context based on common sense, world knowledge or shared cultural experiences, a property often cited as Grice's second maxim of quantity: "Do not make your contribution more informative than is required" (Grice, 1975, 1989). Studies have estimated that only  $12\%$  of the information conveyed by text is mentioned explicitly (Graesser, 2013; Tandon et al., 2020). To illustrate this, consider the question "who is the president of the U.S.?" To answer it, a human reader is likely to presume many unstated propositions, as exemplified in Tab. 1.
In contrast to humans, supervised models often fail to generalize and understand implicit context, instead resorting to low-level correlations in
Figure 1: A high-level overview of the current state of supervised NLP research. Dataset developers create more aggressive filtering techniques (left), leading to larger models that are able to solve them by finding more elusive spurious correlations (right).
Who is the president of the U.S.?
Context	Answer
∅	Joe Biden
The year 2019	Donald Trump
The West Wing, season 1	Josiah “Jed” Bartlet
Table 1: Context, whether explicit or implicit, matters in textual understanding, as exemplified by the question "who is the president of the U.S.?" E.g., in the first line, given no other context, a QA system should provide the most sensible fallback answer (Joe Biden, at the time of writing).
the data, leading to amplified bias (Zhao et al., 2017; Stanovsky et al., 2019) and brittle performance (Schwartz et al., 2017; Gururangan et al., 2018). To address this, recent approaches have suggested mitigating such correlations by balancing the dataset via either adding or removing certain instances (Goyal et al., 2017; Hudson and Manning, 2019; Zellers et al., 2018; Sakaguchi et al., 2020). In parallel, developers keep building larger and larger pretrained models (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020), which, when fine-tuned on these datasets, consistently manage to reach human performance. Taken together, these trends lead to an arms-race between data curation
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2182 - 2194
and model development (Fig. 1).
In this position paper, we question the value of mitigating spurious correlations via dataset balancing, by showing that their existence in large training sets is both inevitable and to some extent even desired, as they are an inherent property of natural language understanding. We build on a recent result by Gardner et al. (2021), who assumed that every single-word feature correlation is spurious, i.e., can be used to mislead a model. We extend their argument, showing that balancing single-word features is insufficient for eliminating all spurious correlations, and that balancing feature combination is needed for that purpose. On the other hand, we show that balancing too much leads to datasets that contain no learnable signal either. We conclude by questioning whether mitigating all spurious correlations via dataset balancing is practical.
Following, we show that this practice is also undesired. We show that ignoring these correlations will hinder the learning of fallback options for both world knowledge facts (Joe Biden is the president of the U.S.) and common sense knowledge (a person is happy when receiving a gift), thus preventing models from using this knowledge in cases of uncertainty. We conclude that the existence of spurious correlations in training sets should not be solved by creating more balanced datasets. $^{1}$
We then discuss alternatives to mitigating spurious correlations. We argue that models should be trained to understand constructions emanating from an apriori theory of language, such as negation, sarcasm, humor, and metaphors. We also suggest adopting modeling approaches that identify when the context is insufficient. We argue that in such cases, the model should not fallback to default assumptions, but rather abstain or interact with the user to clear ambiguities. Finally, we question the basic procedure of large-scale fine-tuning, and suggest focusing on zero- and few-shot learning instead (Liu et al., 2021b).
2 Dataset-Model Arms Race
This section provides a view of recent research in NLP as an arms race between models and datasets. Below we describe the conditions leading to this
Who is wearing glasses?
Figure 2: An example of dataset balancing (adapted from Goyal et al., 2017). For each (question, image) pair in the VQA dataset (left), VQA2.0 adds another image, for which the answer is different (right).
arms race, and present our main research question, challenging its value for making progress in NLP.
Models exploit spurious correlations While pretrained models consistently perform well across multiple tasks, various studies have pointed out that this is often achieved by exploiting spurious correlations in datasets, rather than improving on the underlying task (Glockner et al., 2018; Gururangan et al., 2018; Elazar et al., 2021), and that this phenomenon becomes more prominent as the models grow in size (Li et al., 2021).
Mitigating spurious correlations via balancing Various dataset curators have tried to prevent models from learning spurious correlations by modifying their training data via a careful control for the training label distribution, effectively striving for a balanced dataset. One approach is to add examples in order to balance the dataset (Goyal et al., 2017; Sharma et al., 2018; Hudson and Manning, 2019). For instance, the VQA2.0 dataset (Goyal et al., 2017) is built by taking every (question  $q$ , image  $i$ , answer  $a$ ) triplet in the VQA dataset (Antol et al., 2015), and adding another triplet with the same question  $q$ , but a different image  $i'$ , guaranteed to lead to a different answer  $a'$ . See Fig. 2 for an example.
Filtering as balancing A complementary balancing approach to augmentation is filtering examples out from datasets such that spurious correlations are minimized. This approach was taken in the creation of the SWAG dataset (Zellers et al., 2018), using "adversarial filtering" (AF). In AF, dataset instances that are easily solved by an adversarial model are filtered out. The AF approach and similar approaches were picked up by many datasets such as ReCoRD (Zhang et al., 2018), DROP (Dua et al., 2019), HellaSWAG (Zellers et al., 2019),  $\alpha$ NLI (Bhagavatula et al., 2020), and
<sup>1</sup>We emphasize that balancing methods are still useful as they can lead to mitigation of some spurious correlations, and therefore better generalization (Le Bras et al., 2020; Swayamdipta et al., 2020), as well as potentially more efficient training. We argue that these methods are inherently limited in their ability to mitigate all spurious correlations.
WinoGrande (Sakaguchi et al., 2020).
Here we argue that approaches like AF converge to removing all low-level correlations, and therefore a fully balanced dataset. As this approach relies on an external model, applying it with ever stronger models with higher capacity, will allow these models to pick up on subtler correlations (Li et al., 2021). At the extreme, the remaining instances that could not be solved by a fully capable model will have no statistical signal that can be exploited by that model, i.e., a balanced dataset. We henceforth refer to both augmentation and filtering as balancing methods.
Large models solve the new datasets In parallel to the efforts in dataset balancing, the leading modeling approach in recent years in NLP is pretraining large language models on raw text corpora, followed by fine-tuning them on supervised downstream applications. These models continue to grow in size (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019; Raffel et al., 2020), and their fine-tuning performance improves accordingly. This in turn leads to more aggressive balancing, setting in motion a kind of arms race between datasets and models (Fig. 1).
Evidently, a similar trend emerges for the previously mentioned datasets: (1) the first baselines, reflecting the state of the art at the time of dataset creation, perform relatively poorly, e.g.,  $59\%$  on SWAG,  $47\%$  on ReCoRD, 47 F1 on DROP,  $47\%$  on HellaSWAG,  $69\%$  on  $\alpha$  NLI, and  $79\%$  on Wino-Grande; (2) model developers introduce increasingly larger and heavily-parameterized models, hill-climbing on these datasets; and eventually (3) models essentially solve the dataset within a year or two, often outperforming humans:  $86\%$  on SWAG (Devlin et al., 2019),  $94\%$  on ReCoRD (He et al., 2021b), 88 F1 on DROP (Chen et al., 2020),  $93\%$  on HellaSWAG (He et al., 2021b),  $92\%$  on  $\alpha$  NLI (He et al., 2021a), and  $90\%$  on Wino-Grande (Raffel et al., 2020). (4) new large-scale datasets are collected with more aggressive pruning techniques, thus repeating the cycle.
Based on these findings, our main research question is whether dataset balancing is the most promising method for mitigating spurious correlations. We note that an arms race between models
Name	Description
ingenuine	Correlations between features and output labels for no reason.
ungenerizable	Correlations that do not generalize to new contexts.
every-word	Correlations between every single-word feature and output label.
Table 2: Different definitions of spurious correlations.
and datasets might spur advances. Here we question a specific aspect of this arms race: the improvement of datasets by using more aggressive filtering techniques. Next we turn to present practical and conceptual limitations of this practice.
3 The Lost Battle Against Spurious Correlations
So far we have identified dataset balancing as a common way to mitigate spurious correlations. Next, we outline how different works define spurious correlations (Sec. 3.1), and then question whether dataset balancing is a viable way for mitigating them; we note that balancing too little is bound to leave spurious correlations in the data (Sec. 3.2), while balancing too much discards meaningful signal (Sec. 3.3). We finish by questioning whether this practice is even desired (Sec. 3.4).
3.1 What are Spurious Correlations?
Mitigating spurious correlations is frequently used as motivation for developing new balancing approaches. However, the term spurious correlations is often not clearly and consistently defined. The basic definition is a set of features that are correlated but not causally related. $^3$
In NLP, several definitions of spurious correlations are typically used. One conceptual definition, denoted here ingenuine (e.g., Wang and Culotta, 2020; Rogers, 2021) is a feature correlated with some output label for no apparent reason. Such features often result from the annotation process (referred to as annotation artifacts; Gururangan et al., 2018). For instance, Gururangan et al. (2018) have shown that the words "cat" and "sleeping" are correlated with contradictions in the SNLI dataset (Bowman et al., 2015).
This definition is appealing: we want our models to learn real information about the world, and not properties of a given dataset. However, it is also
2Indeed, AFLite, an extension of AF, was designed to "systematically discover and filter any dataset artifact in crowdsourced commonsense problems" (Le Bras et al., 2020, emphasis in the original).
<sup>3</sup>https://en.wikipedia.org/wiki/ Spurious relatonship
somewhat subjective, and could include features that might be referred to as genuine, such as the word "not" indicating NLI contradictions. Further, genuine features, i.e., those representing a real phenomenon in the world (e.g., "amazing" as a feature for positive sentiment), are also likely to lead models make to erroneous predictions in some contexts (e.g., negation or sarcasm; Gardner et al., 2021). Such features could thus harm generalization, so some might consider them spurious as well.<sup>4</sup>
In an alternative definition, denoted ungeneralizable, a spurious feature is one that works well for specific examples but does not hold in general (Chang et al., 2021; Yaghoobzadeh et al., 2021). This definition does not address the nature of the feature (genuine or not), but does make an implicit assumption that such features are of high importance (e.g., high pointwise mutual information values with the corresponding label; Gururangan et al., 2018). This definition is no longer subjective in terms of the genuineness of the feature, but is still subjective in the level of effect on generalizability (i.e., what is a high value of PMI?).
Gardner et al. (2021) relaxed the last constraint, and assumed that every simple correlation between single word features and output labels is spurious (henceforth every-word). They then defined a class of competent datasets, where the marginal probability for every feature is uniform over the class label, i.e., for any feature  $x_{i}$  and label  $y \in Y$ ,  $p(y|x_{i}) = \frac{1}{|Y|}$ , thus limiting models from picking up any correlation between single features and output labels.
We next extend the every-word approach beyond single words, showing that models that can exploit single word features can also exploit some feature interactions, and therefore these should also be considered spurious. Tab. 2 summarizes the different definitions of spurious correlations.
3.2 Balancing too Little Leaves some Spurious Features
Gardner et al. (2021) assumed that as each word can appear in certain contexts that change its semantic meaning (e.g., negation, sarcasm), each word is potentially spurious. Here we note that the same argument can be applied to feature interactions, such as word  $n$ -grams. We start with a toy
Split	Text	Label
Train	very good	+
very bad	-
not good	-
not bad	+
Test	not very good	-
good	+
Table 3: A toy example of a training set (Train), which is balanced for unigrams, but not for bigrams. Relying on the bigram correlations (e.g., memorizing that "very good" leads to a positive sentiment) will lead to mispredictions on the test set (Test).
example to illustrate our argument for bigrams, and then extend it for larger values of  $n$ .
Consider the toy dataset for the task of sentiment analysis shown in Tab. 3, with vocabulary  $V = \{ \text{good, bad, not, very} \}$ , and label set  $Y = \{ +, - \}$ . The Train split is balanced with respect to single-word features, i.e., it is a balanced or competent dataset:
$$
\forall w \in V, y \in Y: p (y | w) = \frac {1}{| Y |}
$$
Assume the semantics of this dataset is that of English, while  $+ \cdot$  means positive sentiment and  $-$  means negative.
A model trained on Train can achieve perfect training accuracy by learning the correct semantics. However, achieving perfect training accuracy can also be done by learning correlations between two-word features and the target label (i.e., memorizing all the training examples). In this case, the model would make the wrong prediction for the first test example in Test (as it has learned that very good is a feature that indicates positive sentiment), and similarly, will make a random prediction for the second test example, which does not contain any two-word feature seen during training.
This example highlights that balancing single-word features does not guarantee resiliency to spurious correlations, and therefore in order to mitigate all spurious correlations, balancing pairs of features is also required. One can construct similar examples for larger values of  $n$ , by similarly considering multi-word expressions and common co-occurrences (e.g., "jaw dropping", "worst day ever"). These could serve as spurious correlations in the same way single words do.
Another example is sarcasm. A model that fails to understand sarcastic contexts will misinterpret
4See Eisenstein (2022) for discussion of different feature types.
Original Train Set	Augmented Samples
Input	Label	Input	Label
0 0	0	*0 0	1
0 1	1	*0 1	0
1 0	1	*1 0	0
1 1	0	*1 1	1
Table 4: Left: a training set for the XOR function, balanced for unigrams. Right: requiring that bigrams are also balanced would prevent models from learning.
statements that appear in such contexts, even if it perfectly understands the base meaning of these statements. Thus, the entire reasoning process of such a model, whether relying on simple features, feature interactions, or other types of understanding, will result in mispredictions of certain inputs, and thus can be considered spurious.
As a result, to truly mitigate all spurious correlations in a dataset, balancing feature combinations is required as well. Accordingly, balancing too little will leave some spurious correlations in the dataset.
3.3 Too much Balancing Prevents Learning Valuable Semantic Knowledge
We observed that balancing too little does not allow models to fully eliminate spurious correlations. Here we show that too much balancing can prevent models from learning valuable knowledge.
Consider the training data for learning the XOR function presented in Tab. 4 (left). This dataset contains enough learnable signal when considering feature interactions despite being balanced for single words. Nonetheless, balancing this dataset for pairs of features would result in no information, and thus prevent any model from learning this function (Tab. 4, right).
Now consider a given natural language dataset  $D$ . Define  $n$  to be the length of the longest document in  $D$ . By definition, balancing every combination of up to  $n$  features leaves no learnable signal in  $D$ . We conclude that balancing too much can prevent models from learning semantic knowledge.
Combining the two observations, we are left with the question of the potential intersection between balancing too much and balancing too little: does a sweet spot exist for which no spurious correlations are found in the dataset, but enough learnable signal is left? And even if so, would a balancing
algorithm, whether by augmentation or filtering, be able to find it? We leave these questions for future work, but note that addressing them is a prerequisite for the theoretical and practical application of dataset balancing for mitigating spurious correlations.
3.4 Dataset Balancing is Undesired
Even if a sweet spot exists between balancing too little and too much, do we really want to find it? Here we argue that perhaps not.
The practice of dataset balancing is designed to prevent models from learning that some words or expressions have a common fallback meaning that can stem from dataset artifacts (e.g., "cat" as an indicator of contradiction) but also from cultural and historical contexts (e.g., Biden is the U.S. president in 2022). Fallback meanings are crucial for understanding language, as contexts are often underspecified (Graesser, 2013). Indeed, relying on fallback meanings might make models fail to process some inputs correctly, and might not generalize to other domains where the fallback meaning is different. We argue that the ability to use them is a central ability of language understanding.
For example, substantial efforts are made to teach models world knowledge, such as that the president of the U.S. is Joe Biden, the capital of Brazil is Brasília, and France is the soccer world champion. These efforts include building world knowledge datasets (Wang et al., 2021), developing methods for enhancing models with this information (Zhang et al., 2019; Peters et al., 2019), and evaluating how well models capture it (Rubinstein et al., 2015; Roberts et al., 2020). But many of these world-knowledge facts are context dependent: the capital of the Brazil has changed in 1960, the president of the U.S., as well as soccer world champions potentially change every 4 years, etc.
Another example is common sense knowledge, such as "people are happy when they receive a gift", "an elephant is taller than a zebra", and "a statue that doesn't fit into a suitcase is too large". A large body of work has been carried out to create benchmarks that measure the common sense abilities of models (Liu and Singh, 2004; Levesque et al., 2012; Zellers et al., 2018; Sakaguchi et al., 2020; Bisk et al., 2020), as well as augmenting models with such abilities (Qin et al., 2020; Bosselut et al., 2021).
Common sense reasoning is, by definition, stochastic and reliant on understanding presup
<sup>5</sup>We assume the standard data collection process when using AF, in which the last step is balancing (Zellers et al., 2018; Dua et al., 2019), and longer instances cannot be added.
posed, underspecified context. One could imagine a person unhappy to receive a gift (e.g., because it is not what they wanted), a fantastically large zebra compared to a tiny elephant, and a suitcase with multiple compartments which prevent a small statue from fitting in it.
These examples illustrate that a model that learns these correlations and relies exclusively on them to make predictions is limited and is bound to make mistakes in some contexts. One way to avoid these mistakes is to balance these correlations out, and prevent models from knowing these assertions to begin with. We argue that this solution is not a desired solution. In essence, an interpreter's task (be it human or machine) is to infer the most probable context in which a statement is made, and as a result, it should have a fallback option for such world knowledge and common sense assertions.
Discussion We recognize that a balanced dataset may not be balanced with respect to the appearance of common-sense or world-knowledge assertions in a given context. E.g., a model might balance-out the general fact that Joe Biden is the U.S. president, but not that he is the president in 2022. As in many cases much of the context is unobserved (Graesser, 2013), the question is whether we want models to make a prediction in cases of uncertainty based on the fallback option. We argue that doing so is a desired strategy in many cases (though a preferred strategy might be to interact of abstain from making a decisive prediction, see Sec. 4.2).
We also acknowledge that correlations in the real world can be misleading. For instance, people often mistake the biggest commercial city in some countries for their capital (e.g., Istanbul in Turkey), potentially due to the high correlation between the two. In such cases, relying on the fallback option might lead to prediction error. However, we argue that following the human strategy of relying on a fallback option in cases of uncertainty will promote models' communication abilities.[6]
We want to stress that balancing methods can result in mitigating some of the spurious correlations, and therefore lead to increased generalization (Le Bras et al., 2020; Swayamdipta et al., 2020). Moreover, the process of filtering the data naturally results in smaller datasets, which leads to lower training costs (Swayamdipta et al., 2020). While such
Current Practice	Proposal
Dataset balancing	Richer contexts (§4.1)
A closed label set	Abstain/interact (§4.2)
Large-scale fine-tuning	Few-shot learning (§4.3)