Venue: NAACL 2022

# Informativeness and Invariance:

Two Perspectives on Spurious Correlations in Natural Language

 Jacob Eisenstein

Google Research

jeisenstein@google.com

###### Abstract

Spurious correlations are a threat to the trustworthiness of natural language processing systems, motivating research into methods for identifying and eliminating them. However, addressing the problem of spurious correlations requires more clarity on what they are and how they arise in language data. Gardner et al. (2021) argue that the compositional nature of language implies that _all_ correlations between labels and individual "input features" are spurious. This paper analyzes this proposal in the context of a toy example, demonstrating three distinct conditions that can give rise to feature-label correlations in a simple PCFG. Linking the toy example to a structured causal model shows that (1) feature-label correlations can arise even when the label is invariant to interventions on the feature, and (2) feature-label correlations may be absent even when the label is sensitive to interventions on the feature. Because input features will be individually correlated with labels in all but very rare circumstances, domain knowledge must be applied to identify spurious correlations that pose genuine robustness threats.

+
Footnote †: _Corresponding of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4326 - 4331

## 1 Introduction

Spurious correlations are a growing source of concern in machine learning Geirhos et al. (2020) and related fields including natural language processing Gururangan et al. (2018); McCoy et al. (2019)_inter alia_). While the intuition is fairly clear -- spurious correlations are features that are useful in the training data but unreliable in general -- the notion is frequently referenced without a formal definition. Gardner et al. (2021) propose a definition in terms of conditional probabilities: a feature \(X_{i}\) is spuriously correlated with the label \(Y\) unless \(P(Y~{}|~{}X_{i})\) is uniform. The definition can be generalized from uniformity to independence (\(X_{i}~{}\botleftharpoons[Y]\)) without affecting the claims of the paper. They go on to argue that "in a language understanding problem, ..._all_ simple correlations between input features and output labels are spurious" (emphasis in the original). The property that individual input features should be independent of labels -- which I will call _marginally uninformative input features (UIF)_1 -- is treated as an assumption about the nature of language processing and also as a desideratum that datasets should satisfy: if the label can be predicted from input features alone, then the dataset is in some sense too easy.2

Figure 1: An instance from the toy model. The upper part of the figure corresponds to \(f_{X}\), the function that generates the text via a PCFG (see fig. 2): nodes represent non-terminals in the grammar and edges represent context-free derivations. The lower part of the figure corresponds to the causal model of the sentiment \(Y\) and target \(Z\). Here nodes represent random variables and edges represent causal relationships.



Footnote 2: The term “\(X_{1},Y_{2}\)” is the sum of a significant amount of morphological composition and are therefore capable of carrying complex relational meanings. Conversely, multiword expressions can function analogously to single word features, so there is no reason in principle why only single-word features should be considered spurious Schwartz and Stanovsky (2022).

The principle of UIF is based on the insight that linguistic context can modulate the semantics of any subspan of a text, using mechanisms such as syntactic negation or discourse markers. Furthermore, the frequency of negation and other forms of semantic inversion may vary across datasets and deployment settings. A predictor that relies on negation being rare (to pick one example) cannot be said to have truly achieved competence in the language processing task. Such a predictor may perform poorly in domains in which these high-level distributional properties shift.

An especially provocative assertion of Gardner et al. is that all correlations between labels and individual input features have the same status. In the sentence _the pizza was amazing_, suppose that both _pizza_ and _amazing_ are correlated with positive sentiment because the reviewers like pizza. There are at least two intuitive differences between these two correlations. First, while one can easily imagine a benighted subpopulation of reviewers who do not like pizza, it is not so easy to imagine reviewers who think that the word "amazing" carries negative sentiment. Second, if we modify the subject (e.g., _the movie was amazing_), the label will usually be unaffected, but there are many perturbations to the adjective that flip the label (e.g., _the pizza was greasy_). This second intuition can be described using the framework of causality, which has generally treated spurious correlations as those that arise without a direct causal explanation Simon (1954). Given a causal model of the data generating process, we can compute an _interventional_ distribution \(P(Y~{}|~{}\text{do}(X_{1}:=x),X_{2},X_{3}),\) which corresponds to the distribution over \(Y\) in a data generating process in which the variable \(X_{1}\) is surgically set to the value \(x\)Pearl (1995); Peters et al. (2017); Feder et al. (2021).3 When such interventions do not affect \(Y\) for any given example, we say that \(Y\) and \(X_{1}\) are _counterfactually invariant_Veitch et al. (2021). Violations of UIF are particularly troubling when they are accompanied by counterfactual invariance, because non-causal correlations often do not transfer to other domains Scholkopf et al. (2012); Buhlmann (2020).

Footnote 3: Space does not permit a discussion of the distinction between interventions and counterfactuals see Pearl (2009).

This paper uses a toy example to relate the UIF property to (1) the production probabilities in probabilistic context-free grammars (PCFGs), and (2) counterfactual invariance in structured causal models. The connection to PCFGs provides additional motivation for the UIF criterion from the perspective of domain generalization, while clarifying the scenarios that can give rise to violations of UIF, which Gardner et al. attribute too narrowly to "bias and priming effects" in annotators. The connection to counterfactual invariance highlights the ways in which these concepts do and do not align. Efforts to remove artifacts from the training and evaluation of NLP systems will be most productive when focused at the intersection of these two views of spurious correlations: violations of UIF for input features to which the label is counterfactually invariant according to a plausible causal model.

## 2 Toy Example

Consider a simplified targeted sentiment analysis task Mitchell et al. (2013), in which the sentiment is \(Y\), the target is \(Z\), and the sentences are all of the form \((X_{1},X_{2},X_{3})\), with \(X_{1}\) specifying a target noun phrase, \(X_{2}\) a copula-like expression, and \(X_{3}\) a predicative adjectival phrase. For example, \(Y=\textsc{Pos},Z=\textsc{Pizza},X_{1}=\textit{the pizza},X_{2}=\textit{turned out to be}, X_{3}=\textit{cribspy and delicious}\). We will treat this data as generated from the causal model shown in fig. 2. This causal model can be summarized by two assertions: (1) the target \(Z\) is a direct effect of only the span \(X_{1}\); (2) the sentiment label \(Y\) is a direct effect of only the spans \(X_{2}\) and \(X_{3}\). The function \(f_{X}\) can represent any generative model of text: an n-gram model, a grammar-based formalism, a deep autoregressive network, etc.

Figure 2: Causal model for the toy example shown in fig. 1. \(N_{U},N_{X},N_{V},N_{Z}\) indicate independent noise variables, and \(f_{X},f_{Y},f_{Z}\) indicate deterministic functions that map from causes to effects (for more details on the notation, see Peters et al., 2017).

 

Aside on the direction of causation.We treat the text as the cause of the labels, rather than the converse. This distinction is somewhat vexed (Scholkopf et al., 2012; Jin et al., 2021). In some cases the direction of causation is clear from the task (e.g., table-to-text generation, summarization, and translation), but often the problem could be framed in either direction: perhaps the writer had the label in mind when producing the text, and thus the text is an effect of the label; or perhaps it is better to think of the annotator, who must read the text to arrive at the label, regardless of the writer's intentions. When the labels cause the text, the notion of counterfactual invariance can be restated in terms of the invariance of text features to perturbations on labels, e.g. \(P(X_{1}~{}|~{}\text{do}(Y:=y),Z)\). As the toy example is meant to serve only an expository purpose, we leave elaboration of the relationship of UIF to such models for future work.

### Counterfactual invariance \(\mathbf{\gg}\) Uif

The causal model implies several counterfactual invariance properties: intervention on \(X_{1}\) will not affect \(Y\), nor will intervention on \(X_{2}\) or \(X_{3}\) affect \(Z\). This is because \(X_{1}\) blocks the influence of \(X_{2}\) and \(X_{3}\) on \(Z\), and vice versa for \(Y\). Conversely, \((X_{3},Y)\) are not counterfactually invariant in general because \(X_{3}\) is an ancestor of \(Y\) in the causal graph, and similarly for \((X_{2},Y)\) and \((X_{1},Z)\).

Counterfactual invariance does not imply that the associated input features are marginally uninformative of the label. Consider a classical spurious correlation in which pizza tends to receive positive sentiment and sushi receives negative sentiment. This correlation is produced when \(f_{X}\) encodes a PCFG with the top-level production:

\[S\rightarrow \tilde{Z}_{\text{pizza}}~{}\tilde{Y}_{+} (1+\alpha)/4 \tag{5}\] \[\tilde{Z}_{\text{sushi}}~{}\tilde{Y}_{-} (1+\alpha)/4\] \[\tilde{Z}_{\text{pizza}}~{}\tilde{Y}_{-} (1-\alpha)/4\] \[\tilde{Z}_{\text{sushi}}~{}\tilde{Y}_{+} (1-\alpha)/4,\]

with the right column indicating the probability of each rule expansion and \(\alpha\in[-1,1]\).4 The nonterminal symbols \(\hat{Z}_{\text{pizza}},\hat{Z}_{\text{sushi}},\hat{Y}_{+},\tilde{Y}_{-}\) are intentionally chosen to correspond to the labels \(Z\) and \(Y\). Subsequent rules in the grammar can then be designed to ensure that \(\tilde{Z}_{\text{pizza}}\) usually produces values of \(X_{1}\) that make \(Z=\textsc{PIZZA}\) likely, and analogously for the other non-terminals and associated labels. The unification of PCFGs and structured causal models is shown in fig. 1.

Footnote 4: The stochasticity of the grammar is encoded in the deterministic function \(f_{X}\) through the noise variable \(N_{X}\). Let \(N_{X}\sim\text{Uniform}(0,1)\), and choose the first rule expansion of \(S\) when \(N_{X}<(1+\alpha)/4\), the second rule expansion when \((1+\alpha)/4\leq N_{X}<(1+\alpha)/2\), and so on.

When \(\alpha\neq 0\), there may be an association between \(X_{1}\) and \((X_{2},X_{3})\). As a result, there exist pairs of values \((x_{1},x^{\prime}_{1})\) such that,

\[P( Y|X_{1}=x_{1}) \tag{6}\] \[=\sum_{X_{2},X_{3}}P(Y~{}|~{}X_{2},X_{3})P(X_{2},X_{3}~{}|~{}X_{1 }=x_{1})\] \[\neq\sum_{X_{2},X_{3}}P(Y~{}|~{}X_{2},X_{3})P(X_{2},X_{3}~{}|~{}X_ {1}=x^{\prime}_{1})\] \[=P(Y|X_{1}=x^{\prime}_{1}),\]

creating a violation of UIF. The same argument can be applied to \(P(Z~{}|~{}X_{2})\) and \(P(Z~{}|~{}X_{3})\). UIF is also violated in \(P(Z~{}|~{}X_{1})\), \(P(Y~{}|~{}X_{2})\), and \(P(Y~{}|~{}X_{3})\), but for a different reason: these distributions are conditioned on the direct causal parents of the labels in \(f_{Y}\) and \(f_{Z}\). Manipulation of the data distribution to ensure that \(\alpha=0\) (deconfounding \(\tilde{Y}\) and \(\tilde{Z}\)) can remove only the violations of UIF induced by \(f_{X}\), but not those induced by the direct causal relationships encoded in \(f_{Y}\) and \(f_{Z}\): for example, if \(\Pr(X_{3}=\textit{delicious}|\tilde{Y}_{+})>\Pr(X_{3}=\textit{delicious}| \tilde{Y}_{-})\) then the feature _delicious_ will be associated with positive sentiment regardless of the rule probabilities in eq. (5).

Discussion.The example shows how violations to UIF can emerge via confounding, creating classical spurious correlations in the sense of Simon (1954): informativeness despite counterfactual invariance. Such correlations are unlikely to be robust because it is not difficult to imagine a domain in which the sign of \(\alpha\) changes, impairing the performance of predictors that have learned the spurious correlation. In contrast, feature-label correlations that arise directly from the causal model, such as \((Z,X_{1})\), are only damaging under more extreme forms of concept shift, in which the meanings of the features themselves change.

Aside on causality and robustness.The distinct interpretations of spuriousness as (1) non-causal and (2) non-robust are noted by Schwartz and Stanovsky (2022) in concurrent work. However, these interpretations can be reconciled by 

the argument that non-causal features are inherently unlikely to be robust, which is sometimes formalized as the _principle of sparse mechanism shift_Scholkopf et al. (2021). The principle states that complex causal systems are usually composed of smaller independent parts, with domain shifts affecting only a few components of the system at a time. A related principle arises in the context of natural language: distributional frequencies are more likely to change across domains, while categorical facts about language are generally stable. Biber (1991), for example, makes this argument explicitly in the analysis of register. In our model, the implication is that the probabilistic rule expansions in \(f_{X}\) are more likely to change than the basic properties of the lexicon, which govern which terminal symbols can be emitted by each non-terminal.

Uif \(\nRightarrow\) Counterfactual Invariance Violations of counterfactual invariance can occur even when UIF is satisfied. To show this, we supply two more productions for the grammar:

\[\begin{array}{rcll}\tilde{Y}_{+}&\rightarrow&\text{Cop}_{+}~{}\textsc{AdjP} _{+}&&\beta_{+}\\ &\text{Cop}_{-}~{}\textsc{AdjP}_{-}&&1-\beta_{+}\\ \tilde{Y}_{-}&\rightarrow&\text{Cop}_{+}~{}\textsc{AdjP}_{-}&&\beta_{-}\\ &\text{Cop}_{-}~{}\textsc{AdjP}_{+}&&1-\beta_{-}\end{array} \tag{8}\]

Here the non-terminal Cop\({}_{+}\) produces a "positive" copula in \(X_{2}\) (_is, was, is universally agreed to be_), Cop\({}_{-}\) produces a negated copula in \(X_{2}\) (_isn't, wasn't, was the furthest possible thing from_), AdjP\({}_{+}\) produces positive-sentiment adjectival phrases in \(X_{3}\) (_great_, _delicious_), and AdjP\({}_{-}\) produces negative-sentiment adjectival phrases in \(X_{3}\) (_disappointing_, _totally unappetizing_). There are two special cases of interest:

* When \(\beta_{+}=\beta_{-}\), the probability of using a negated copula is independent of \(Y\), so \(X_{2}\) satisfies UIF with regard to \(Y\), while \(X_{3}\) generally does not.
* When \(\beta_{+}=1-\beta_{-}\), the use of negation is balanced to make the distribution over sentiment terms independent of \(Y\), so \(X_{3}\) satisfies UIF with \(Y\), while \(X_{2}\) generally does not.

Combining these cases, both \(X_{2}\) and \(X_{3}\) satisfy UIF with \(Y\) when \(\beta_{+}=\beta_{-}=\frac{1}{2}\), meaning that negated and non-negated copula are equally likely and are independent of \(Y\).

Discussion.UIF is violated not only by confounding, as discussed in the previous section, but also in mild settings that do not meet any reasonable definition of bias: unless \(\beta_{+}=\beta_{-}=1/2\) then at least one of \(X_{2}\) and \(X_{3}\) is marginally informative of \(Y\). Furthermore, UIF has no impact on the counterfactual invariance of \(X_{2}\) and \(X_{3}\) on \(Y\). Neither is counterfactually invariant even when the generative model is parametrized to make UIF hold for all input features (see also Pearl (2009); page 185). This is because the overall sentiment can be directly affected by adding or removing negation and by flipping the polarity of the sentiment-carrying adjective.

## 3 Conclusions

In the toy example, violations of UIF arise from three distinct phenomena: confounding between the sentiment and the target (\(\alpha\neq 0\), leading to \(X_{1}\not\mathrel{\hbox{$\mskip-4.0mu \perp\mskip-4.0mu \perp\mskip-4.0mu \perp \mskip-4.0mu \perp\mskip-4.0mu \perp\mskip-4.

 

UIF is a necessary but insufficient condition for a spurious correlation in the causal sense.

Moving beyond toy examples, it is unlikely that we can construct fully-specified causal models of language that supply useful invariances while handling every possible fluent utterance. How then can we use causal insights to design better benchmarks and more robust language understanding systems? In some cases it is possible to elaborate partial causal models of a task, with associated invariance properties: for example, the sentiment of a movie review should be invariant to (though not independent of) the identities of the actors in the movie. Several existing approaches can be viewed as instantiations of partial causal models: for example, data augmentation, causally-motivated regularizers, stress tests, and "worst-subgroup" performance metrics (and associated robust optimizers) can be seen as enforcing or testing task-specific invariance properties that provide robustness against known distributional shifts (e.g., Lu et al., 2020; Ribeiro et al., 2020; Kaushik et al., 2021; Koh et al., 2021; Veitch et al., 2021). Such approaches generally require domain knowledge about the linguistic and causal properties of the task at hand -- or to put it more positively, they make it possible for such domain knowledge to be brought to bear. Indeed, the central argument of this paper is that no meaningful definition of spuriousness or robustness can be obtained without such domain knowledge.

A final observation, pertaining to both UIF and counterfactual invariance, is the parallel treatment of \(X_{2}\) (the copula) and \(X_{3}\) (the adjectival phrase). From a lexical semantic perspective, only \(X_{3}\) is directly associated with sentiment, while \(X_{2}\) plays a functional role by potentially reversing \(X_{3}\). It may therefore seem undesirable to learn a correlation between \(X_{2}\) and \(Y\), and preferable to attach that relationship exclusively to \(X_{3}\). Indeed, one of the main catalysts of interest in spurious correlations in natural language processing was the observation that the presence of syntactic negation is a strong predictor of contradiction label in the natural language inference task, which should require reasoning about pairs of sentences Gururangan et al. (2018); Poliak et al. (2018). Yet neither UIF nor counterfactual invariance is capable of making any distinction between \(X_{2}\) and \(X_{3}\) in this model. While it is possible to enforce uninformativeness on \(X_{2}\) heuristically, e.g. by sampling or augmenting the data to ensure \(\beta_{+}=\beta_{-}\), those same heuristics could be applied to enforce uninformativeness on \(X_{3}\) by making \(\beta_{+}=1-\beta_{-}\). Singling out \(X_{2}\) requires additional justification. Such a principle might be found in the multitask setting, in which we prefer feature-label informativeness to be sparse, with each feature directly informing only a few labels.

Acknowledgments.Thanks to Alex D'Amour, Amir Feder, Katja Filippova, Matt Gardner, Tal Linzen, Ellie Pavlick, Noah Smith, Ian Tenney, Kristina Toutanova, and Victor Veitch for feedback on this work.

## References

* Biber (1991) Douglas Biber. 1991. _Variation across speech and writing_. Cambridge University Press.
* Buhlmann (2020) Peter Buhlmann. 2020. Invariance, causality and robustness. _Statistical Science_, 35(3):404-426.
* Feder et al. (2021) Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. 2021. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. _arXiv preprint arXiv:2109.00725_.
* Gardner et al. (2021) Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021. Competency problems: On finding and removing artifacts in language data. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 1801-1813.
* Geirhos et al. (2020) Robert Geirhos, Jorn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. _Nature Machine Intelligence_, 2(11):665-673.
* Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 107-112.
* Jin et al. (2021) Zhijing Jin, Julius von Kugelgen, Jingwei Ni, Tejas Vaidhya, Ayush Kaushal, Mrnimaya Sachan, and Bernhard Schoolkopf. 2021. Causal direction of data collection matters: Implications of causal and anticausal learning for NLP. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 9499-9513 

Divyansh Kaushik, Amirth Setlur, Eduard Hovy, and Zachary C Lipton. 2021. Explaining the efficacy of counterfactually augmented data. In _International Conference on Learning Representations_.
* Koh et al. (2021) Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergeye Levine, Chelsea Finn, and Percy Liang. 2021. WILDS: A benchmark of in-the-wild distribution shifts. In _Proceedings of the 38th International Conference on Machine Learning_, pages 5637-5664.
* Lu et al. (2020) Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2020. Gender bias in neural natural language processing. In _Logic, Language, and Security_, pages 189-202. Springer.
* McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3428-3448.
* Mitchell et al. (2013) Margaret Mitchell, Jacqui Aguilar, Theresa Wilson, and Benjamin Van Durme. 2013. Open domain targeted sentiment. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1643-1654.
* Pearl (1995) Judea Pearl. 1995. Causal diagrams for empirical research. _Biometrika_, 82(4):669-688.
* Pearl (2009) Judea Pearl. 2009. _Causality_. Cambridge university press.
* Peters et al. (2017) Jonas Peters, Dominik Janzing, and Bernhard Scholkopf. 2017. _Elements of causal inference: foundations and learning algorithms_. The MIT Press.
* Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_, pages 180-191.
* Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4902-4912, Online. Association for Computational Linguistics.
* Scholkopf et al. (2012) Bernhard Scholkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. 2012. On causal and anticausal learning. In _Proceedings of the 29th International Coference on International Conference on Machine Learning_, pages 459-466.
* Scholkopf et al. (2021) Bernhard Scholkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. 2021. Toward causal representation learning. _Proceedings of the IEEE_, 109(5):612-634.
* Schwartz and Stanovsky (2022) Roy Schwartz and Gabriel Stanovsky. 2022. On the limitations of dataset balancing: The lost battle against spurious correlations. In _Findings of the Association for Computational Linguistics: NAACL 2022_.
* Simon (1954) Herbert A Simon. 1954. Spurious correlation: A causal interpretation. _Journal of the American statistical Association_, 49(267):467-479.
* Veitch et al. (2021) Victor Veitch, Alexander D'Amour, Steve Yadlowsky, and Jacob Eisenstein. 2021. Counterfactual invariance to spurious correlations in text classification. _Advances in Neural Information Processing Systems_, 34.