Counterfactual Maximum Likelihood Estimation for Training Deep Networks
Authors: Xinyi Wang
Wenhu Chen
Michael Saxon
William Yang Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: causality
spurious correlation
deep learning
data augmentation
out-of-distribution generalization
natural language processing

Abstract: Although deep learning models have driven state-of-the-art performance on a wide array of tasks, they are prone to spurious correlations that should not be learned as predictive clues. To mitigate this problem, we propose a causality-based training framework to reduce the spurious correlations caused by observed confounders. We give theoretical analysis on the underlying general Structural Causal Model (SCM) and propose to perform Maximum Likelihood Estimation (MLE) on the interventional distribution instead of the observational distribution, namely Counterfactual Maximum Likelihood Estimation (CMLE). As the interventional distribution, in general, is hidden from the observational data, we then derive two different upper bounds of the expected negative log-likelihood and propose two general algorithms, Implicit CMLE and Explicit CMLE, for causal predictions of deep learning models using observational data. We conduct experiments on both simulated data and two real-world tasks: Natural Language Inference (NLI) and Image Captioning. The results show that CMLE methods outperform the regular MLE method in terms of out-of-domain generalization performance and reducing spurious correlations, while maintaining comparable performance on the regular evaluations.

Counterfactual Maximum Likelihood Estimation for Training Deep Networks
Xinyi Wang, Wenhu Chen, Michael Saxon, William Yang Wang
Department of Computer Science
University of California, Santa Barbara
edu, wenhuchen&ucsb. edu, saxonBucsb. edu, william@cs.. wcs..u...... edu
Abstract
Although deep learning models have driven state-of-the-art performance on a wide array of tasks, they are prone to spurious correlations that should not be learned predictive clues. To mitigate this problem, we propose a causality-bers framework to reduce the spurious correlations caused by observed confounders. We give theoretical analysis on the underlying general Structural Causal Model (SCM) and propose perform Maximum Likelihood Estimation (MLE) the interven- tional distribution instead of the observational distribution, namely Counterfactual Maximum Likelihood Estimation (CMLE). As the interventional distribution, in general, is hidden from the observational data, we then derive two different upper bounds of the expected negative log-likelihood and propose two general algorithms, Implicit CMLE and Explicit CMLE, for causal predictions of deep learning models using observational data. We conduct experiments real-world tasks: Natural Language Inference (NLI) and Image Captioning. The results show that CMLE methods outperform the regular MLE method in terms of out-of-domain generalization performance and reducing spurious correlations, while maintaining comparable performance on the regular evaluations
training
both simulated data and
Introduction
Deep neural networks have been tremendously successful across variety of tasks and domains in recent years. However, studies have shown that deep learning models trained with traditional supervised learning framework tend to learn spurious correlations example, in computer vision. deep learning models can rely on surface-level textures background environment [2] 6 instead of the object of interest in images. In natural language processing (NLP), question-anss answering models are insensitive the choice question 7 and natural language inference models are surprisingly accurate at predicting the logical relationship between a pair of sentences from just one of them In image captioning. a phenomenon called object hallucination is observed where models tend to include nonexistent objects in caption based on their common association with other objects that are actually present in the image [4.
predictive clues [ For or
Another related problem in the current associative learning framework, like maximum likelihood estimation (MLE), is that neural networks can achieve almost perfect performance on one dataset but dramatically fail to generalize on another because of naturally occurring or ad adversarially enforced distribution shift [I. One explanation is that the model learns 'fake' features induced by spurious correlations instead of invariant features that would hold in different domains for the same task [12]
this kind of invariance can be explained by the underlying causal mechanism of the task [3]
Our code is released at http:////ggggguooo//////aaaaaaaa
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

It is usually impractical to identify and disentangle all of the potential sources of spurious correlations. Various training algorithms have been proposed reduce the influence of spurious correlations by learning the underlying causal mechanism. One approach for discovering the underlying causal mechanism is by invariant prediction based on the invariance of causal mechanisms across different environments [13 12 [4]. However, constructing such diverse environments is usually unrealistic. Another approach is counterfactual data augmentation [15H19), which directly modifies the part of the input that causes the target variable, but usually involves expensive human efforts in generating the counterfactual examples. We propose Counterfactual Maximum Likelihood Estimation (CMLE), which tries to perform MLE on interventional distribution instead of the observational distribu- tion. Here, obervational distribution means the distribution that the train data is sampled from. Interventional distribution means the distribution with one some of the variables intentionally intervened (set to some fixed value). More specifically, we aim to remove observable confounders and reduce spurious correlations by directly intervening on the label variable, without requiring human annotators or the curation of diverse environments.
In this paper, we focus on the setting of predicting the outcome Y of a certain action I on X, with the underlying causal model as X + Y < ' and an observed train dataset in the form of (X, T) Such setting enables us to assume that there exist some observed confounders in X that influence both Y and T. This means there is potentially a false causal relation from X to T, indicated by a red arrow in Figure la In our framework, instead of trying to identify all the confounders, we try rule out the effect of the confounders by considering the interventional distribution that directly deleting the false causal edge from X to T with resulting causal graph shown in Figure Id This causal graph can be viewed as a Randomized Controlled Trial (RCT), which is the gold standard approach for estimating the treatment effect (20)
Our method is inspired by individual treatment effect (ITE) prediction 21 22), which tries to predict the expected effect E[Y; YolX xx] (e.g. differnce in blood pressure) of treatment T' {0, 1} (e.g. drug/surgery) on a individual unit X = x (e.g. a patient). In this case, the observed confounder in X can be explained by selection bias [22]. which means the treatment T applied to an individual X is dependent on X. For example, young patients are more likely to be treated by surgery, while elder patients more likely to be treated by drugs.
There are two broad kinds of tasks to which our proposed CMLE framework can apply: relationship prediction tasks that predicts the relation T given two inputs X and Y (e.g. natural language inference
23 24, paraphrase identification (25] 26), natural language for visual reasoning [27 etc.), and conditional generation tasks that generate Y given precondition X and constraint T (e.g. style transfer
28 22], controllable image/text generation (31,21,, controllable image captioning [33 34] etc.). For conditional generation tasks, the causal relation of X, Y. T fits in our setting. For relation prediction tasks, we first assume there is an underlying conditional generation process of Y for given pair of X and T. This is natural assumption this is usually how the datasets for this type of task is generated. Then we augment the original dataset using the counterfactual examples generated by our method. In this paper, try to reduce the spurious correlations contained in both kinds of tasks.
Since we generally cannot observe the interventional distribution, we derive two different upper bounds of the expected interventional negative log-like likeliiel using only the observational distribution: one is Implicit CMLE, as it does not involve any explicit generation of counterfactual examples; while the other is Explicit CMLE, as it explicitly generates counterfactual examples during the training process. We test our framework with deep neural networks on a simulated dataset and two real-world tasks with well-known spurious correlations: natural language inference [23] and image captioning (35. Compared to regular MLE, we improve the out-of of-domain accuracy of NLI on a hard adversarial dataset by 2.9% relatively and beat baselines on human preference evaluations by more than 10%, while maintaining comparable performance on automated evaluations. Our results show that our learning framework can better capture the underlying causal mechanism and reduce spurious correlations without degrading performance.
2 Related Work
A growing body of work has investigated algorithmic improvements for machine learning model training by leveraging underlying causal mechanisms. One line of work tries to utilize the invariance of causal mechanisms across different environments. Invariant Causal Prediction (ICP) [36] aims to

---

Tu?
7-m
T=1
T-m
Y
Y
Y
Y=
Y
Y2
Ya
(a) Observed p (X, ),1)
(d) Intervened
(b) Expand all possible outcomes
(c) Intervene on '''
p' x,),
Figure I: The Structural Causal Model (SCM) assumed by Counterfactual MLE, where (a) and (b) have the same distribution, (c) is the resulting causal graph intervening on T. (d) the intervened distribution by only removing the X + 'I' edge from (a). Shaded nodes means being observed and red arrows denote the causal relation causing the spurious correlation.
learn linear invariant causal predictor to identify the causal parents of given target variables with data obtained from different environments. Further refinements on this line of work include nonlinear and nonparametric ICP assessment methods [37] and Invariant Risk Minimization (IRM) where out-of-distribution generalization is enabled by learning a nonlinear invariant causal predictor shared across different environments [2]. However, these methods generally require a set of environments that are both sufficiently diverse to eliminate confounders while maintaining the underlying causal structure. Though this can be achieved by intervening in the non-causal variables in a synthetic interactive setting. it is very difficult to obtain even define such environments with complex high nigh-dimensional observational data like text or images. Our proposed CMLE framework only requires single observational dataset, making it much more practical.
Another line of work approaches this problem with data augmentation. Methods including program- matically editing text to address gender bias [16188, employing crowd workers augment training data to capture potential unmeasured variables [38], and manually editing minimal text spans to flip the predicted label [15 have all been proposed to produce counterfactually augmented data (CAD) for improving out-of-domain generalization. Conceptually, in contrast to the first approach that potentially large number of non-causal variables, CAD directly intervenes on a target variable [39]. which is more practical for a specific task. Refinements on this work attempt to better utilize the manually edited or synthetically generated counterfactual data by pairing the factual and counterfactual examples together [40] [4], and leverage local causal structures to generate counterfactual experience in training reinforcement learning algorithms
involves interventions
[19]. While human annotators can generate high-quality counterfactual examples that are effective in reducing spurious correlations, they are expensive obtain and do not easily scale. Meanwhile, our proposed CMLE framework is fully automated, requiring no human annotations.
Our CMLE framework is inspired by technique for predicting the Individual Treatment Effect (ITE) from observational data [22]. This method only observes one possible outcome of an individual X with one binary treatment T = 0 or T 1, and it assumes there are hidden confounders but potentially observed ones. Assuming a similar underlying causal model as ours (m 2) gives an error bound the sum of the standard generalization error and the distance between the control and treated distributions in a representation space. Improvements on this work include disentangling the observable confounders in ITE using representation learning 42), and estimating the causal effects of linguistic properties [43]. While in this work we do not aim to separate the confounders from the causal factors in X, we try reduce the spurious correlations caused by the possible confounders contained in X using our proposed CMLE training framework.
3 Method
In this section, we first introduce our problem setting and the goal of Counterfactual Maximum Likelihood Estimation (CMLE). As cannot observe the counterfactual outcomes in observational data, we propose two different ways of estimating the counterfactual likelihood by deriving two different upper bounds: Implicit CMLE, which is faster training, and Explicit CMLE, which is more flexible. We also briefly introduce the deep learning architecture we use in our experiments.
3.1 Problem Setting
We consider the scenario with one input variable X E X' that contain all the confounders, one discrete treatment variable T {1, m) where m > 2, and one outcome variable Y E 3,
3

---

as shown in Figure la Note that X' and Y can be very complicated high dimensional space like text or image and all three variables are observed in the dataset. Suppose the data we observed is D = {(11, 41 y1 ): (x2, l2; 12), .... (Ii, In Un)} and only x; is are sampled i.i.d. In this paper, we adopt the causal notations from [44] Below we give formal definition of Structural Causal Model
(SCM) 44) that we use to model our framework:
Definition 1. A Structural Causal Model (SCM) is a pair (M,p(u)), where M is a triple (U, V, F) and p(u) is probability function defined over the domain of U. U {U1, Uu., Un} is set of background variables (exogenous) that are determined by factors outside the model. V = {V1, .2...., Vn } is set of variables (endogenous) and F f2,, f2...., is a set of functions s.t. V; = ((Pa(VV), U.), i= 1,2, ..., n. where Pa(V)) E VV,, are the parents of V₁. We say that "V, is direct cause of V;" Vf Pa(V:) and illustrate this relation V - V₂ in a causal graph.
In Figure T. we omit the exogenous and only show the causal relations between endogenous variables. The shaded nodes indicate that the corresponding variables are observed. Intervention on a specific variable is formalized as fixing the value of this variable to a specific value. This is known as the do-operation, changing the causal graph by removing all edges incident to the intervened variable. We denote the outcome of Y corresponding to do = as Y,. We expand all the possible outcomes of Y in Figure Tb while we actually can only observe one outcome for each data point in the observational dataset. We also make the following standard assumption [2 45 49]:
Assumption 1. (Strong Ignorability Condition) (Y1, Y2, ..., .m) 1 TIX and 0 (T < 1.
Under this condition, the counterfactual outcomes are identifiable since p(Y; = y| p(Y u|XX = T = t). i.e. The counterfactual likelihood can be written as observable conditional probabilities estimated from data.
Our goal is to eliminate the false causal relation from X to T represented by the red arrow in Figure
la To achieve this goal, define our objective over the interventional distribution of (X, Y, Y,T)) as shown in Figure ld instead of over the observed data distribution p²(X, (X, Y, I") as shown in Figure To construct the interventional distribution, we only delete the causal edge between X and T and keep all the other causal relationships unchanged. In this distribution. T is uniformly sampled from {1,2, m.. Note that p' (Y| p' (X) = (Yr (X= p (Y 2). As p(Yr X) is the same for both observational distribution and interventional distribution, we omit the superscript for simplicity. Then for predicting Y. the objective of counterfactual maximum likelihood estimation
Ial
(CMLE) would be:
E EX - log P6 = Ex
Ey,xx log Po (o, XV]]
(1)
Similarly, for predicting T, we have:
Ep((x log / = Ex
By x log (T
Here 0 is the learnable parameters. In the remaining part of paper, the probabilities without su- perscripts are default to be observational probabilities. In the following sections, we first focus on deriving two different upper bounds of the CMLE objective for predicting Y: Implicit CMLE and Explicit EMLE. In Implicit CMLE, we add a regularizer to balance the distribution of X given T in the representation space. For Explicit CMLE, training. To predict T, we train a classifier using normal MLE on a augmented dataset with the generated counterfactual examples estimate the interventional distribution p' (X, Y,T). We define a counterfactual example as follows:
directly generate counterfactual examples during
Definition 2. counterfactual example is an example (x, t, y) with y sampled from p(Y; z) s.t. (x,t,y) & D and there is an example (z, t', t',v/) E DD where t' f' t.
ETT Lo 1)). by defining the expected We can rewrite our objective function in Equation 1 as E. loss function of a specific input and treatment as: Definition 3. We denote the loss function for predicting Y as T. Lo( 1, = log (V, x|Xx = x)) Then the expected loss function given X = x and T = t is Co t) = 5 Y.)].

---

WASS(pg mp
Classifier Ps (T]
11 (He,.),)
y())
y02
y(
(()
(4)
Encoder
Decoder
Encoder x00) x(0)
Decoder
x x(1)
(MM)
y(k-1)
x
t/K, y(t) y(
(x,
Figure 2: The model architecture of Implicit CMLE (left) and Explicit CMLE (right).
Because the Strong Igmorability Assumption, we have Ca t) > O. Then we define an expected factual loss e and expected counterfactual loss eCF when observing a specific treatment T t:
Definition 4. The expected factual loss and expected counterfactual loss given T t are ef Ex(r (X,t)) and e ExITyriCC (X, t)), respectively.
Here e- measures how well the model performs on the observational distribution while " measures how well the model would perform on an alternative world where different treatment is chosen. Then we can rewrite our CMLE objective as:
Cola, (o),,)]== DlT =
t)eF + + + ()
T='
(3)
t=1
3.2 Implicit CMLE
We consider learning a representation function of X, $ - R.d, where Rdd is the representation space. Here make the following assumption about T::
Assumption 2. T X - Rd is invertible function. We denote v Rd X as the inverse of Φ. For continuous X, V also needs to be differentiable to calculate the Jacobian matrix.
Note that this assumption are invertible. We denote the distribution induced by & by pq. We can easily transform p(X (T) to po (X))) by standard change of variable [50,
be fulfilled for neural networks as long as all the activation functions
Definition 5. For i {1, 2,., m) and Rd. denote (r) := P+(r|T = i). For discrete
X. we simply have pT-i ((r) p(X (r) i), while for continuous X, have ph (r)
p(X (r) T det 8
We adopt a similar idea as [22] and use the Integral Probability Metric (IPM) to derive an upper bound of the CMLE objective. IPM is class of metrics between probability distributions with the following definition [52]:
Definition 6. For two probability density functions p and q defined over S, we have
/ g(()(p(s) - (()))ds
IPMc q) = sup gEG
Where is a family of functions
IR
Note that here (and in the rest of paper) abuse the notation of integration. If X' is discrete, the integration would become a sum over it. Then we can derive an upper bound for our CMLE objective by Assumption 2 Definition 5 and Equation 3 (See Appendix A. full proof.)
Theorem 1. For function family G and a constant B& 0 s.t. Bg (o) for any e Rd and 21,2, ..., m), the CMLE objective is bounded by:
E
(a
T P(T + B)BoolPM T=t PP Tx')
In this upper bound, we are only left with observationallfactual quantities that can be estimated from the observational data. We choose G' to be J-Lipschitz functions, which implies IPMg to be the Wasserstein distance (53]54] 54), denoted by WAss(..).. Then the empirical CMLE objective using an unbiased estimator of this upper bound can be written as:
5

---

n Lo ti 91) + (1 4 (B)))) (4)
arg min
Ut
1=1
Where u, = - -, and I is the indicator function. a is a hyperparameter accounting for Ba and we abuse the notation of WA to denote an empirical version of Wasserstein distance. The resulting empirical training objective consists weighted version of the normal factual loss for empirical risk minimization and a regularization term to draw pb and P T*i
closer. " accounts "1, for different number examples with each label and 1 "i account for p(T + (see Theorem D. The empirical Wasserstein distance is calculated among examples with different labels in the representation space. In practice, to perform back propagation, we directly compute the gradient of
WAss Ps T+1) over a single mini-batch, using the algorithm proposed by [55]
(See Appendix
B.I for details.)
3.3
Explicit CMLE
While Implicit CMLE tries reduce the spurious correlation by learning a better representation of X, we consider directly learning a better approximation of the log likelihood log po (Y, by explicitly generating counterfactual examples at training. Here we assume that there a neural network with parameter & that approximates the interventional distribution P (TIY, X) defined as follow:
Definition 7. (T|Y, X) is
approximate of (T( X) parameterized by & s.s.
K LL Y) P (T|X, Y)) < B, where B > 0 is a real constant. We also define an asso- ciated loss function (t, x, y) = (T f|Y = y. = z).
Here (p(pll denotes the KL. divergence between two distributions p and q. Our intuition is that is directly feasible to learn the normal MLE objective Ex,y. [log pa ( X)) from the observation data, while is infeasible directly learn the CMLE objective E,, [E, [z, Also, T is categorical variable, its distribution can be simplified as a Multinomial distribution. We then derive an upper bound of the CMLE objective using Bayes' rule and Jensen's Inequality: (See Appendix A.2 for full proof.)
Theorem 2. By Assumption there exist 81 < 82 < I.s. 81 <T([X)= < 82, then we have
m
E
Co(x, ()] < Ex,r
1,y)l +
(X Jo(i,,,,)11 +
Where 1
)), μ = (md2 mōj - m + 1) log m.
p(t)
This theorem indicates that we can transfer the counterfactual likelihood of predicting into the counterfactual likelihood of T predicted by another model. Note that the resulting bound would be tighter when Ps (T X) a better approximate of (T)Y, X) as we use the fact (L) X) (T), > in the proof (see Appendix A..)p. In practice, we can either pretrain a Po (T X) on the observational data when Py Y. X) and p X) are not too far away, or we can simultancously train a Ps (T with po XY to take advantages of the generated counterfactual examples.
Since is expensive to take m - 1 samples of yi from po (Y for each training example at train time, introduce another random variable K, for each E {1, ..., m), such that is K, uniformly distributed on {1, 2,,,,.., m.,........... ML4. Then the Explicit CMLE objective can be rewritten as:
m EK, Epo Xx [ (o x, Yk.).]]
arg min Eg,
E (Y, [L(x, t, YJ]
Here we ignore the constant term j in the upper bound and multiply the remaining terms by J. We can then write the Explicit CMLE objective empirically
1 [Lo t. v4) + a(ts) Fzs (z,, ke,)))
arg min
(5)
0

---

Where a(ts) is hyperparameter corresponding to m==1, kt is randomly sampled from p(Ke,) and yo (zig, R2,) ) is sampled from po(Y,, X z). We only sample K2 once and sample the corre- sponding Yx, once for computational efficiency. In our implementation, to backpropagate through PY,, X x), we adopt the Cumbel--Softmal approach [56) deal with the discrete text data, which creates a differentiable sample to replace the non-differentiable discrete variable. For alternative approaches for discrete and continuous )/ see Appendix B.2
3.4 CMLE Implementation
While CMLE is a general causal training framework that can be applied to almost any deep learning architecture, in this paper, we primarily consider the Transformer [57] architecture, which excels at modeling sequential data and has been successfully applied to various domains, especially with X, we can write as pretraining large amount of data [58 [59. Here assume for any
{z!00) .x(1,, .... } and for any E Y we have y 1 ()) ,..., y. (()) We input 7 = t
by prepend special token/vector corresponding to as a prefix to the decoder input y. The model
architectures of Implicit CMLE and Explicit CMLE are shown in Figure 2
4 Experiments
We consider one synthetic dataset and two real-world tasks, Natural Language Inference and Image Captioning, to evaluate the effectiveness of our proposed CMLE causal training framework. Both real-world tasks have prominent datasets containing well-known spurious correlations; we aim to reduce these correlations' influence on each target task. In the following sections, we will introduce our experiments in details and compare the performance of our proposed CMLE against vanilla MLE.
4.1 Simulated Experiments
We consider a synthetic dataset generated by the following procedure inspired by 45
Given the followigng functions 91 (z) -
= = (2
-2 2 sin((x) Io = e -z) { (x)) =2 =0) = x, go() =
cos( (1 t) = +12) t and g13(2,t) sin(t + z)):
1. Sample r; from (0 (1,2.....000 100).
2. Let = g] + (2 + 93(13) + (1,))) + (I()). Let = 1 if S 4; f = 2 if
4 < 5
3. Let μ1 (0(x1) + (22) + (xx) + (0 t) + 912(17, t), #2 go(2s) + (1)
911 (1x) + 913( Øg. t). Then sample from N ((11,H2).1)
4. Repeat steps 1-3 for N times.
Here X real vector of length 100, T takes value in (1,2,3) and is real vector of length The task is to predict Y given X and T. Then we generate 10000 train data, and 5000 validation data and 5000 in distribution (observational) test data by setting N=10000 and 5000 respectively. We also generate corresponding ground truth counterfactual test data for testing how well our models can estimate the counterfactuals. OODI, OOD2, OOD3 are three distr--iisttibution test sets that generates T using different mechanisms as follows:
0ODI: t is uniformly sampled from 31.
00D2: Let 91 (æe) + + (x7) + ga(ss) + (x (x(0)) Let i S 4; t if < 5; 3 if >
*
00D3: Let s = + (x2) (x(a)) + (x (10( Lett 1 if S 4; t 2
if4 5
We use multi-layer perceptrons for all of our models and use the mean--suuared-error (MSE) loss at training. This means we assume Y to follow Gaussian distribution given X and T. In Table [ report MSE of predicting Y on different test sets.
Here Explicit CMLE* means using p& (TIX Y) that is trained on the interventional distribution instead of the observational distribution. The numbers are averaged over ten runs and the standard deviation is also reported. We take = 0.01 for Implicit CMLE and 0.1 for Explicit CMLE.
7

---

test set
MLE
Implicit CMLE
Explicit CMLE
Explicit CMLE.
Observational
3.63:0.20
3.57#0.17
3,45:0.29
3,28=0.27
Counterfactual
5.25+0.40
5.11+0.38
4.77+0.38
4.61=0.42
OOD1
4.32+0.28
4.46+0..37
4.31+0.32
3.98=0.36
OOD2
7.25+0.56
7.1740.56
6,9440.73
6.39=0.68
OOD3
4.51+0.33
4.35+0.36
4.15+0.40
3.91=0.44
Table 1: MSE of predicting Y given X and T.. The numbers are averaged over ten runs.
We can see that all CMLE methods perform better than MLE on the counterfactual test set and the distiidtittiittionnn test sets, while Explicit CMLE performs better than Implicit CMLE. And the performance of Explicit CMLE would be better when po (T]T,, (2,XXXYY))) ) is closer to the ground truth interventional distribution.
4.2 Natural Language Inference (NLI)
Natural language inference (NLD) 23 is a fundamental task in natural language understanding, which aims to infer the logical relation between a pair of sentences, premise and hypothesis. Unfortunately, NLI datasets are known to present significant spurious correlations. For example, [ demonstrated that a machine learning model can non-trivially outperform the majority baseline using only the hypothesis without the premise. One known spurious correlation comes from the background knowledge contained in the premises. i.e. The prediction is based on the background knowledge instead of the logical relation between the premise and the hypothesis. For example, the hypothesis "TV viewing and consumption both begin at about the same time." " would usually be considered to be true from common knowledge. While fine-tuned RoBERTa (60) also predict this example as entailment, it contradicts its premise "Regular TV viewing typically begins between 2 and 3 years of age, consuming about 10.". We view the premise sentence as X, the hypothesis sentence as Y. and the label (entailment, neutral, or contradiction) as T. The NLI annotation procedure [23 61
naturally fits in causal framework in Figure 1] the crowd workers are instructed to write hypothesis that is ontailed/neurrr///ontaaiiii to a given premise.
We consider three large-scale NLI datasets: SNLI [23), MNLI (24] and ANLI (62). SNLI and MNLI 522/10,00//////1///1 092,702/20,00//20.00 examples for train/dev/test split respectively. ANLI was constructed adversarial dataset to eliminate spurious correlations and balance the data distribution, thus it serves great benchmark to evaluate model's generalizability. ANLI contains three subsets of increasing Lifficulty-A.. A1, A2, and A3-with about about 392, 012,700220,00000000 200.00000220000000000000 examples, 000,46001,000/1,,11 examples and 100,459/1 2,00/112000 examples each for train/dev/test split respectively.
are considered standard benchmarks, with each containing
examples and hard
For generating counterfactual hypotheses we fine-tune the pre-trained BART large model provided by HuggingFace [64] with 12 encoder and decoder layers. We fine-tune BART on the combined dataset of SNLI and MNLI using both Implicit and Explicit CMLE, We choose a = 0.003 for Implicit CMLE and 0.1 for Explicit CMLE. We predict Ps (T|X, Y) for Explicit CMLE with a conventionally fine-tuned RoBERTa large model with 24 layers. We call this model NLI-RoBERTa.
We evaluate how consistent our generated hypotheses Y are to their corresponding labels T using both NOI-RoBERTa and human evaluation in (rightt. In both cases, we perform this assessment by comparing the model- or human-predictt RoBERTa is shown the first row of Table Both of our CMLE methods significantly outperform the MLE baseline by a large margin on the labeling consistency of the generated hypothesis, which indicates that our CMLE framework can generate Y, that truly influenced by t.
L' against the ground truth 1. The accuracy of the NI.I-
To construct a dataset that mimics sampling from the interventional distribution, we generate two counterfactual examples for each example in the original train dataset of SNLI and MNLI. We filter out the inconsistent examples using NLI-RoBERTa. We generate three augmented datasets using standard MLE, Implicit CMLE, and Explicit CMLE and fine-ture tne-tune a copy of RoBERTa on each one. The results for these models are presented in rows MLE (w/ Aug). Implicir CMLE and Explicit CMLE of Table 2 respectively. The baseline MIEE is NEETRoBERTa itself. We test the accuracy of all the fine-tuned RoBERTas simultaneously on the regular test sets (MNLI, SNLI) and the out-of-domain adversarial test sets (ANLI). The results Table 2 show that, while maintaining performance on
2 All of our human evaluations are conducted via Amazon Mechanical Turk. See Appendix C.3 for details.
8

---

NLI Accuracy
Generation Consistency
Method
A1
A2
A3
ANLI
SNLI
MNLI-m/mm
SNLI (R/H))
MNLI (R/H)
MLE
47.3
26.1
22.4
31.3
92.6
90.6/90.5
N/A
N/A
MLE (w/ Aug)
47.2
25.4
21.7
30.8
92.4
90.4/90.3
65.5/46.8
66.0/48.3
Implicit CMLE
47.0
27.1
24.0
32.2
92.4
90.4/90.5
88,5/69.6
75. 75.1/7 85.8/62.4
Explicit CMLE
47.7
26.8
23.7
32.2
92.4
90.5/90.5
95.2/70.8
Table 2: Accuracy of RoBERTa fine-tuned on augmented SNLI + MNLI (left), along with label consistency (aceuracy) of the generated hypothesis (right). We report all the results on test sets except on MNLI we use dev sets, where 'm' means matched and 'mm' means mismatched. 'R' indicates evaluations by NLI-RoBERTa and 'H' indicates evaluations by human.
regular test sets, our proposed CMLE methods significantly outperform both MLE baselines on the adversarial test set. This indicates that CMLE can improve the out-of-domain generalization performance of large deep learning models without hurting the in-domain performance.
4.3 Image Captioning
Image captioning is a fundamental task in studying multimodal interaction. However, image cap- tioning models also suffer severely from spurious correlations 665 that lead to hallucinations. This type of spurious correlation mainly caused by the typical co-occurrences of objects in images. For example, as toilets are frequently present in bathroom pictures, a captioning model trained by MLE is very likely to hallucinate a 'toilet' whenever it sees a 'bathroom' even if no toilet is present.
To reduce these hallucination artifacts, we apply our CMLE framework by regarding the image as X, the caption as Y, and whether the caption is hallucinated T. In setting, the non- hallucinated captions are ground truth captions written by a human, and the hallucinated captions are generated by image captioning models. The dataset we use is the MSCOCO 2014 dataset [66] which contains 123,287 images of common objects with 5 human-annotated captions per image. We use the Karpathy 667 split with 1 5,000 images in the rrain/validation/testt set respectively. Our implementation is based on [68].
The Transformer architecture we use directly follows (57), with 6 layers in each of the encoder and decoder layers. To construct a dataset with both hallucinated and non-hallucinated captions, we synthetically generate captions by masking out object word for each caption and use BERT (large) (58) to fill in an alternative word that aligns with the language model. We estimate the caption hallucination by the CHAIR score (65) which parses the object words in caption and compares them to the ground truth objects annotated in MSCOCO images and captions. In total, we synthesize 648,550 hallucinated captions for 93,004 images in the training set.
Then we train the transformer model on this augmented dataset of MSCOCO captions with synthetic hallucinated captions using regular MLE and our proposed CMLE methods. We choose 0.0002 for Implicit CMLE and a = 0.0001 for Explicit CMLE. For Explicit CMLE, we fine-tune a pre- trained LXMERT (59 this augmented dataset (T), Here we consider two MLE baselines: The MLE baseline is the Transformer model regularly trained on the original MSCOCO dataset, while the MLE wngg baseline is trained the augmented dataset.
We conduct user studies on the generated captions as shown in Table Bl In the user study, we let the user choose the best caption among our two baselines and two proposed CMLE methods from the following aspects: Faithfulness, Expressiveness and Informativeness, Faithfulness means how faithful is the generated caption to the corresponding image, which partially measures the extend of hallucinations. Expressiveness means if the generated caption is coherent, grammatically, and semantically correct. Informativeness means if the caption includes most of the information in the image instead of giving a general description. We also let the user choose the caption that best describes the image in their opinion, which is referred to Preference in Table We can see that our proposed CMLE methods outperform the two MLE baselines with a large margin on all the evaluation dimensions which indicates that methods can not only reduce the hallucination contents in the generated captions but also increase the quality of generated captions in general.
The automatic evaluation results are shown in Table where CHAIRs is the percentage of hallucinated sentences and CHAIRi is the percentage of hallucinated instances (objects). We can see that our CMLE methods outperform both MLE baselines CHAIR scores, which indicates that our methods

---

Choice (%)
MLE
MLE (w/ Aug)
Implicit CMLE
Explicit CMLE
Tie
Faithfulness
20.7
13.3
28.7
32.0
5.3
Expressiveness
22.7
14.0
27.9
32.7
2.7
Informativeness
21.4
14.7
27.2
32.7
4.0
Preference 17.4
12.6
30.6
32.7
6.7
Accuracy 60.2
54.3
66.9
69.1
N/A
Table Human evaluations of the quality of generated captions. We ask users to choose the hest one among four generated captions according to different criterion. Accuracy is obtained by asking the user whether given caption is accurate to the corresponding image or not.
Method BLEU@4 4 69
METEOR 770
CIDEr 71
SPICE 722
CAHIRs/CHIRI 65
MLE
33.1
26.9
106.7
20.0
7.3/5.3
MLE (w/ Aug)
31.3
26.2
102.1
19.6
8.0/5.8
Implicit CMLE
32.0
26.8
105.2
20.0
6.8/4.9
Explicit CMLE
32.1
26.6
105.5
19.9
7.0/5.0
Table 4: Automatic metrics on MSCOCO dataset for image captioning. All the results are reported on the Karpathy test split. Lower CAHIRs and CHAIRi means less object hallucinations.
can effectively reduce hallucination in image captions. At the same time, the regular scores (BLEU@4 4, METEOR, CIDEr, SPICE) that measure the similarities between the generated captions and ground truth captions, are comparable between CMLE and the MLE baseline. We note that the MLE (w/ Aug) baseline on augmented data is significantly worse than the MLE baseline both the automatic scores and human evaluations. We conjecture that this is because the model trained by vanilla MLE is not good at finding the causal relation between the hallucination label T and the caption Y and thus would be confused by the mixture of hallucinated captions and halluucccc captions.
4.4 Discussions and Limitations
From both NLI and image captioning experiments, observe that Explicit MLE is significantly better than Implicit MLE terms of human evaluations of the generated text, and comparable on other automatic evaluations. We conjecture that this is because that Implicit CMLE only incorporates counterfactual constraints into the representation of X while Explicit CMLE directly optimizes PY{Y] (X) when generating the counterfactual examples.
also note that improvement on automatic metries not that large both real-worte tasks. For NLI, our data augmentation cannot perfectly simulate sampling from the interventional distribution as we only augment two counterfactual examples for each factual example. For Image captioning, as it is difficult to simulate the true distribution of hallucinated captions, the models we trained on the augmented data can be affected by these newly introduced artifacts. On the other hand, the automatic metrics like BLEU only measure the similarity of generated sentences with the observed ground truth which cannot demonstrate the full strength our methods.
Conclusion
Our proposed CMLE framework can be applied wide range of learning scenarios with paired inputs and corresponding labels or involving conditional generation. While currently we only consider the spurious correlations caused by the observable confounders under the Strong Ignorabilityy assumption, it is often the case that the confounders are not observed. For example, annotators may have biased pattern of annotation. As we do record the information of the annotators, this confounder is considered unobserved. Expanding the current CMLE framework by including the unobserved confounders into the causal model would be an interesting and important future direction.
Acknowledgments
This work was supported in part by the National Science Foundation Graduate Research Fellowship under Grant No. 16501 14. This material based on work that partially funded by an unrestricted gift from Google.
10

---

References
[1] J., Y. Bengio. Measuring the tendency of cnns to learn surface statistical regularities, 2017.
[2] Beery, S., G. Van Horn. P. Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV). 2018.
[3] Poliak, A., J. Naradowsky, A. Haldar, et al. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference Lexical and Computational Semantics, pages 180-191, Association for Computational Linguistics, New Orleans, Louisiana, 2018.
[4] Rohrbach, A., L. A. Hendricks, K. Burns, et Object hallucination in image captioning. In Empirical Methods in Natural Language Processing (EMNLP). 2018.
[5] Geirhos, R., P. Rubisch, C. Michaelis, al. Imagenet-trained are biased towards texture; increasing shape bias improves accuracy and robustness, 2019.
[6] Zhang, Y. H. Tan, M. Bansal. Diagnosing the environment bias in vision-and-language navigation. In Bessiere, ed., Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 890-897. ijcai.org, 2020.
[7] Kaushik, D., Z. C. Lipton. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010-5015. Association for Computational Linguistics, Brussels, Belgium, 2018.
[8] Quifionero-Cnnella, J., M. Sugiyama, N. D. Lawrence, et al. Dataset shift in machine learning. Mit Press, 2009.
[9] Szegedy, C., W. Zaremba, Sutskever, et al. Intriguing properties of neural networks, 2014.
[10] Ovadia, Y., E. Fertig, J. Ren, et al. Can you trust your model's uncertainty? evaluating predictive uncertainty under datasct shift. NeurIPS. 2019.
[11] Filos, A., P. Tigkas, R. Mcallister, et al. Can autonomous vehicles identify, recover from, and adapt distribution shifts? In H. D. III, Singh, eds., Proceedings of the 37th International Conference on Machine Learning, vol. 119 of Proceedings of Machine Learning Research, pages 3145 -3533. PMLR, 2020.
[12] Arjovsky, M., L. Bottou, I. Gulrajani, et al. Invariant risk minimization, 2020.
[13] Peters, J., P. Buhlmann, N. Meinshausen, Causal inference using invariant prediction: identifi- cation and confidence intervals. arXiv: Methodology, 2015.
[14] Chang, S., Y. Zhang, M. Yu, et al. Invariant rationalization. In International Conference on Machine Learning, pages 1448-18 1458. PMLR, 2020.
[15] Kaushik, D., E. Hovy, Z. C. Lipton. Learning the difference that makes difference with counterfactually augmented data. International Conference on Learning Representations (ICLR), 2020.
[16] Lu, K., P. Mardziel, F. Wu, et al. Gender bias neural natural language processing. arXiv preprint arXiv: 1807.1117 2018.
[17] Zmigrod, R., Mielke, H. Wallach, al. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Association for Computational Linguistics (ACL). 2019.
[18] Maudslay, R. H., H. Gonen, R. Cotterell, et al. It's all in the name: Mitigating gender bias with name-based counterfactual data substitution. arXiv preprint a90iv: 00871, 2019.
[19] Pitis, S., F. Creager, A. Garg. Counterfactual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems, 33, 2020.

---

[20] Greenland, J. Pearl, J. M. Robins. Causal diagrams for epidemiologic research. Epidemiology, pages 37-48, 1999.
[21] Shpitser, J. Pearl. Identification of conditional interventional distributions. In Proceedings of the Srrenty-Second Conference on Uncertainty in Artificial Intelligence, UAI' page 437 444. AUAI Press, Arlington, Virginia, USA, 2006.
[22] Shalit, U., F. D. Johansson, D. Sontag, Estimating individual treatment effect: generalization bounds and algorithms. In International Conference on Machine Learning, pages 3076-3085. PMLR, 2017.
[23] Bowman, S. R.,. G. Angeli, C. Potts, et al. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.00 053266,, 2015.
[24] Williams, A., N. Nangia, S. Bowman. broad-coverage challenge corpus for sentence under- standing through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume (Long Papers), pages 1112-1122. Association for Computational Linguistics, 2018.
[25] Zhang, Y., J. Baldridge, He. PAWS: Paraphrase Adversaries from Word Scrambling. In Proc. of NAACL. 2019.
[26] Lan, W., Xu. Neural network models for paraphrase identification, semantic textual similarity, natural language inference, and question answering. In Proceedings of COLING 2018. 2018.
[27] Suhr, A., M. Lewis, J. Ych, et al. A corpus of natural language for visual reasoning. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pages 217-223, Association for Computational Linguistics, Vancouver, Canada, 2017.
[28] Gatys, L., A. Ecker, M. Bethge. A neural algorithm of artistic style. Journal of Vision, 16(12):326-33) 326, 2016.
[29] Madaan, A., A. Setlur, T. Parekh, al. Politeness transfer: A tag and generate approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1869-1881. 2020.
[30] Mirza, M.. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 1411.174 2014.
[31] Hu, Z., Z. Yang, X. Liang, et al. Toward controlled generation of text. In ICML, pages 1587-1596. 2017.
[32] Khalifa, M., H. Elsahar, M. Dymetman. A distributional approach to controlled text generation. In International Conference on Learning Representations. 2021.
[33] Chen, L., Jiang, J. Xiao, et al. Human-like controllable image captioning with verb-specific semantic roles. arXiv preprint
2021.
[34] Deng, C., N. Ding, M. Tan, et al. Length-controllable image captioning. arXiv preprint arXiv:2007.09588, 2020.
[35] You, Q., H. Jin, Z. Wang, et al. Image captioning with semantic attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4651 4659. 2016.
[36] Peters, J., P. Bühlmann, N. Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 78(5):947-0012. 2016.
[37] Ghassami, A., S. Salehkaleybar, N. Kiyavash, et al. Learning causal structures using regression invariance. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
[38] Srivastava, M., Hashimoto, Liang. Robustness to spurious correlations via human annota- tions. International Conference Machine Learning (ICML), 2020.
12

---

[39] Kaushik, D., A. Setlur, E. Hovy, et al. Explaining the efficacy of counterfactually augmented data. International Conference on Learning Representations (ICLR), 2021.
[40] Teney, D., Abbasnedjad, A. v. d. Hengel. Learning what makes a difference from counterfac- tual examples and gradient supervision. arXiv preprint WxXiv:20440090:4 2020.
[41] Liang, Z., W. Jiang, H. Hu, al. Learning to contrast the counterfactual samples for robust visual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3285-3292. Association for Computational Linguistics, Online, 2020.
[42] Hassampour, N., R. Greiner. Learning disentangled representations for counterfactual regression. In International Conference on Learning Representations. 2019.
[43] Pryzant, R., D. Card, D. Jurafsky, et al. Causal effects of linguistic properties. arXiv preprint htt::20000/29199 2020.
[44] Pearl, Causality, Cambridge university press, 2009.
[45] Li, S., Fu. Matching on balanced nonlinear representations for treatment effects estimation. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett, eds., Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., 2017.
[46] Johansson, F, U. Shalit, D. Sontag. Learning representations for counterfactual inference. In International conference on machine learning, pages 3020-3029. PMLR, 2016.
[47] Alaa, A. M., M. van der Schaar. Bayesian inference of individualized treatment effects using multi-task gaussian processes. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 3427-3435. 2017.
[48] Yoon, J., Jordon, M. Van Der Schaar. Ganite: Estimation of individualized treatment effects using generative adversarial nets. In International Conference on Learning Representations. 2018.
[49] Yao, L., S. Li, Y. Li, et al. Representation learning for treatment effect estimation from observational data. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Besa-Bianchi, R. Garnett, eds., Advances Neural Information Processing Systems, vol. 31. Curran Associates, Inc., 2018
[50] Rezende, D., S. Mohamed. Variational inference with normalizing flows. In F. Bach, D. Blei, eds., Proceedings of the 32nd International Conference on Machine Learning, vol. 37 of Proceedings of Machine Learning Research, pages 1530-1538. PMLR, Lille, France, 2015.
[51] Tran, D., Vafa, K. Agrawal, al. Discrete flows: Invertible generative models of discrete data. Advances in Neural Information Processing Systems, 32:14719-14722 2019.
[52] Müller, A. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, pages 429-443, 1997.
[53] Villani, C. Optimal transport: old and new, vol. 338. Springer Science & Business Media, 2008.
[54] Sriperumbudur, B. K., Fukumizu, A. Gretton, et On the empirical estimation of integral probability metrics. Electronic Journal of Statistics, 6:1550-1599, 2012.
[55] Cuturi, M., A. Doucet. Fast computation of wasserstein barycenters. In International conference on machine learning, pages 685.693. PMLR, 2014.
[56] Jang, E., S. Gu, B. Poole. Categorical reparamcterization with gumbel-softmax. arXiv preprint arXiv:1611.01 144. 2016.
[57] Vaswani, A., N. Shazeer, N. Parmar, et Attention is you need. arXiv preprint 13.772,06.03762, 2017.
13

---

[58] Devlin, J., M.-W. Chang, K. Lee, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[59] Tan, H., M. Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5103-51 103-5114, 14. 2019.
[60] Liu, Y., M. Ott, N. Goyal, et al. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xrv: 1907.116999 2019.
[61] Williams, A., N. Nangia, S. R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint WoXiv:170.004422 2017.
[62] Nie, Y., Williams, Dinan, al. Adversarial NL.: understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguisties, 2020.
new benchmark for natural language
[63] Lewis, M., Y. Liu, N. Goyal, et al. BART: Denoising wwwuueene-t--squueee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 787 1871-7 Association for Computational Linguistics, Online, 2020.
[64] Wolf, T., L. Debut, V. Sanh, al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Association for Computational Linguistics, Online, 2020.
[65] Rohrbach, A., L. A. Hendricks, K. Burns, et al. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035 4045. 2018.
[66] Chen, X., H. Fang, T.-Y. Lin, et al. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXhv:1504.00325, 2015.
[67] Karpathy, A., L. Fei-Fei. Deep sisual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137. 2015.
[68] Luo, R., B. Price, S. Cohen, et al. Discriminability objective for training descriptive captions. arXiv preprint arXiv: 1803: 04376, 2018.
[69] Papineni, K., S. Roukos, T. Ward, et al. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318. 2002.
[70] Banerjee, S., A. Lavie. Meteor: with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, 2005.
An automatic metric for mt evaluation with improved correlation
[71] Vedantam, R., C. Lawrence Zitnick, D. Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 1566645555 2015.
[72] Anderson, P., Fernando, M. Johnson, al. Spice: Semantic propositional image caption evaluation. In European conference on computer vision, pages 382-398. Springer, 2016.
[73] Kingma, D. P., M. Welling. Auto-encoding variational bayes. arXiv preprint 1r11v:1312.6114, 2013.
[74] Williams, R. J. Simple statistical gradient-followwe algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256 1992.
14


Reviews:
[{"review_id": "pkFJJXgZ2JM", "reviewer": "Reviewer_i8Y6", "summary": "In this work, authors propose the Counterfactual Maximum Likelihood Objective CMLE, where the parameters of the statistical model are inferred by maximizing the likelihood of the data under the interventional distribution, as opposed to the observational data. For the assumed model X -> Y <- T and X -> T, where X is the covariate, T is the treatment and Y is the outcome, the objective is to learn the parameters \\theta that predict Y|X=x,T=t using just the causal link T->Y and not the spurious features in X=x. Ideally, if one could intervene on T and set do(T=t), it is possible to collect data where the outcome is observed upon the intervention done on T. Then maximizing the following likelihood: (CMLE objective) \\argmax_\\theta \\sum_t E_{X|t}  E_{Y|x,t} p_\\theta(Y|X=x,do(T=t)) would give us the MLE estimate of the parameter \\theta under the set of interventional distributions do(T=t), thereby eliminating any effect of the spurious features in X on the prediction of the outcome label Y. The paper proposes two upper bounds for the above CMLE objective: i) Implicit MLE: focuses on obtaining better representations of the covariates X, such that the distribution of these representations is similar (by Wasserstein distance) for every value of T; and ii) Explicit MLE: where counterfactual examples are generated during training. Expectations in the theoretical derivations of the above upper bounds, can be estimated by Monte Carlo evaluations using only the observational data. Results are provided on two downstream tasks where models often rely upon spurious correlations: i) Natural Language Inference (NLI); and ii) Image Captioning (IC).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The empirical results on NLI and image captioning seem fairly convincing, especially the human evaluations. But the improvements in automated metrics don’t seem very significant. \n- The paper is mostly clear and easy to understand. But, I would recommend the authors to improve the notation schema for Explicit MLE while providing an intuitive explanation for the upper bound.\n- Since Explicit MLE does much better on human evaluation for both NLI and IC, it is unclear if practitioners should use Explicit MLE over Implicit MLE for all downstream tasks which require OOD generalization. It would be useful to be aware of at least one setting where Implicit MLE is useful. Although the authors conjecture a plausible explanation, the true reasoning behind this phenomena is not investigated in this work. Undoubtedly, it would be useful for the community to have a better understanding of why modeling the representations of X to be invariant under different treatments is not helpful---especially since multiple works have been proposed along these lines. Is the reason for this purely statistical, wherein the computation of the Wasserstein distance in high dimensions is inefficient or is it more fundamental, i.e. does the upper bound not closely track the CMLE objective to begin with?\n- The theoretical derivations for the upper bounds seem correct. But, both upper bounds are in expectation, i.e. the authors don’t provide probabilistic tail bounds. This is especially important since even though the final optimization objective is an unbiased estimate of the upper bound, the statistical error is very high for those in high dimensions. \n- The authors have provided full access to the code for both NLI and IC tasks. The implementation details have also been made fairly clear.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Most of the concerns I laid above could be addressed in an expanded limitations section. I think if you condense your introduction section, you'll have a bit more space to expand your discussion. Parts of your broader impact discussion in the Appendix can also be brought here once you have more space.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "YDtRaZvdPPe", "reviewer": "Reviewer_PCZg", "summary": "The paper tackles the problem of spurious correlations caused by observed confounders and offers two potential solutions: implicit and explicit counterfactual maximum likelihood estimation. Specifically the paper deals with the setup of predicting the outcome Y of some action T on X and considers X to potentially be a confounder. The paper compares the performance of the proposed methods to on a natural language inference and an image captioning task and shows improvements in terms of human evaluation but little difference in terms of automatic evaluation.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The paper tackles the important problem of deep learning models picking up spurious correlations and proposes an interesting approach to dealing with this problem.\n- The causal angle to reducing spurious correlations is currently being studied from various angles and seems to be a promising direction to handle this problem.\n\nCons:\n- The paper assumes full observability of the confounder, that the exogenous variables are observed and only models a very specific causal model with three variables. This greatly limits the applicability of this work to the general problem of reducing reliance on spurious correlations.\n- The general problem setup of the causal relations and the observability of exogenous variables seems like it lacks some clarity.\n- The main benefits of the method seem to only be observable under the setting of human evaluation. It would be useful to turn to more standard benchmarks or at least similar set ups that test domain generalisation to other types of spurious correlation. The usual benchmarks such as coloured MNIST or ideas from e.g. [2] do not fit the assumed causal model though.  \n\nDetailed comments:\n- Line 61: differnce -> difference\n- The paper assumes the exogenous variable to be included in X - as such it can be argued that the paper does not actually consider counterfactuals but rather only interventions. Throughout the paper the definition of counterfactuals is not precise.\n- Section 3.2 mentions that neural networks are invertible as long as the activations functions are invertible. This is not exactly true as not every weight matrix is invertible, e.g. check out [1] for more on this.\n- Section 4.1 mentions that CMLE significantly outperforms the other baselines the adversarial test set. This is not possible to judge without any confidence intervals. \n- The evaluation of the MLE with augmentation in section 4.2. seems flawed as the augmentations introduce noisy data which can be assumed to deteriorate the performance.\n- The implicit CMLE seems similar to ideas in [2]. Is this correct? It would be good to discuss the relationship to this work.\n\n[1] Kingma, Diederik P., and Prafulla Dhariwal. \"Glow: Generative flow with invertible 1x1 convolutions.\" arXiv preprint arXiv:1807.03039 (2018).\n[2] Makar, Maggie, et al. \"Causally-motivated Shortcut Removal Using Auxiliary Labels.\" arXiv preprint arXiv:2105.06422 (2021).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See other comments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "IaED1YXJq-", "reviewer": "Reviewer_XKW8", "summary": "This paper applies methodology from CATE estimation to do maximum likelihood with respect to samples from the interventional distributions Y|do(T),X, which they call Counterfactual MLE (CMLE). They provide two algorithms, Implicit CMLE and Explicit CMLE. The implicit variant uses the CATE generalization bounds from Shalit's CATE generalization bounds paper to get bounds on the likelihood from the desired counterfactual sampling distribution. The Explicit variant writes bounds in terms of the distribution T|Y,X. The two methods are applied to natural language inference and image captioning tasks. The authors suggest that this methodology and the surrounding literature on causally-inspired prediction should play a large role in the standard approach of deep learning in order to avoid shortcut learning / relying on spurious correlations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. In the intro, it is not clear why you introduce a graph that does not have an arrow from X to T but you call X a confounder. I will ask a series of subquestions on this topic: \n\n- I understand that in general you are concerned with some spurious correlations. However, the *Precise* context in which the red X->T arrow appears was not totally clear in the paper. Y is a common child of X,T so conditioning on Y correlates X and T. But when do we condition on Y? We would usually not do this when predicting Y. \n\n- Separately, you mentioned X may contain some confounders (mutual parent of T,Y). Wouldn't that necessitate a regular (black) arrow from X to T? Why is that not in the graphical models? \n\n- Finally, usually spurious correlations are defined as a correlation between the outcome and some variable that we do not want to predict with. This can arise when they have a common child that is conditioned on, or a common parent that is unobserved. Then, people worry about spurious correlations because the correlation may not hold in new populations / test sets. Can you elaborate on the precise meaning of spurious correlation, conditioning on which variables makes it happen, why do you assume it is bad if a model takes advantage of the correlation (eg some assumption about this correlation not holding in the test data)\n\n\n2. Why do you call ptheta(T|X,Y) in eq. (2) the \"interventional distribution\" and call it the \"observational distribution\" in line 223? By interventional do you mean \"distribution over possible interventions (treatments)\" and likewise by observational do you mean \"distribution that determines which Yt we observe\"? I would maybe rephrase both of these since \"interventional distribution\" is usually used for p(outcome|do(treatment))\"\n\n3. I would suggest omitting some vague descriptions/adjectives/adverbs such a \"much easier\" in 228 and \"potentially complicated\" in 229 or rephrase with more precise terms.\n\n4. Implicit CMLE: it's pretty unclear what is the contribution from the Shalit generalization bounds paper versus what you do on top of it. Can you try to restructure so that you can make that more clear? Do I understand that the main step here is to bound the MLE objective where expectation is over interventional distribution p(Y|do(T),X), and do I understand this bound is achieved by re-arranging the terms in the Shalit generalization bound?\n\n5. Explicit CMLE: when you assume you can estimate  P(T|Y,X) can you provide more info on what making this assumption implies? And if there is any insight on what to expect when it is misestimated?\n\n5. M-1 samples versus tuning gumbel-softmax's temperature parameter. I'm really not sure which is worse. I suggest a dedicated synthetic experiment where you compare these two methods (does gumbel-softmax work well enough).\n\n6. More generally, I think the omission of synthetic experiments is one of the main weaknesses of the paper. The setup has so many moving parts (estimating IPMs, learning auxiliary models, gumbel-softmax). It's so important to isolate the effects of any of these steps and the effect on the entire objective/procedure when any one of these step is wrong. For example, you could try a controlled simulation where the p(T|Y,X) model perfectly matches the ground truth.\n\nside note not factoring into my review: for unobserved confounders, I would check out the recent work on proxy variables. There they discuss observed children of confounders and it may be a useful basis for extensions.\n\nTo summarize, these should be improved:\n- writing clarity about X->T edge\n- writing clarity about contributions over Shalit bounds (e.g. we re-arrange their equation (XYZ) to get our .....)\n- synthetic experiment should be added to test individual parts of the setup.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "3USzFVfOHqa", "reviewer": "Reviewer_715u", "summary": "This paper proposed to use counterfactual maximum likelihood estimation to learn deep learning models that are less susceptible to spurious correlation relationships. In particular, it proposes two general algorithms, Implicit CMLE and Explicit CMLE, for learning causal predictions of DL models under observational data. The proposed method shows improvement over the regular MLE method in two real data sets.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this is a high quality paper with solid works done. Let me summarize my comments in some different aspects:\n\n1. Originality: the proposed algorithm is novel and the idea itself is very interesting.\n\n2. Clarity: this work is well-presented in general. With intuitive explanation following each formula/theorem.\n\n3. Significance: The method itself is very general. The two real data experiments are very enlightening and I think the proposed method has great application potential in more real life tasks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "2_RESnCvWA5", "reviewer": "Reviewer_S8oA", "summary": "This work proposes counterfactual maximum likelihood estimation (CMLE), a new training objective for suppressing the spurious effect of confounding variables. The authors derive two types of upperbound formulation for interventional log-likelihood (i.e. implicit CMLE and explicit CMLE). Using natural language inference (NLI) and image captioning as the intervention modeling tasks, the proposed method demonstrated superior performance in terms of human-perceived performance.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n- The proposed method performed superior performance in terms of human evaluation for NLI and image captioning, compared to simple MLE baselines.\n\nCons\n- In contrast to the effort that went into deriving the CMLE objective, the experiments seem to lack some important factors as below.\n- Lack of baseline models: the aim of this paper is counterfactual estimation (both NLI and image captioning experiments were set up as counterfactual estimation tasks). However, the authors fail to include any existing works related to counterfactual estimation (or treatment effect estimation) such as [1-8].\n- Strange choice of datasets: Most existing counterfactual estimation studies use both simulated datasets and real-world dataset. The former is especially used because simulated datasets contain both factual and counterfactual samples, therefore the model can be tested to see if it actually learned the intervention effect (i.e. treatment effect) properly. However, in this work, the authors employ rather unconventional NLP datasets to test their method. If the authors would like to claim their work as a counterfactual estimation model, they should test their model on simulated datasets as well, rather than only relying on human perception of how appealing the generated outputs are.\n\nReferences\n1. Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. In International conference on machine learning, pp. 3020–3029, 2016.\n2. Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: general- ization bounds and algorithms. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 3076–3085. JMLR. org, 2017.\n3. Ahmed M Alaa and Mihaela van der Schaar. Bayesian inference of individualized treatment effects using multi-task gaussian processes. In Advances in Neural Information Processing Systems, pp. 3424–3432, 2017.\n4. Sheng Li and Yun Fu. Matching on balanced nonlinear representations for treatment effects estimation. In Advances in Neural Information Processing Systems, pp. 929–939, 2017.\n5. Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Ganite: Estimation of individualized treatment effects using generative adversarial nets. International Conference on Learning Repre- sentations (ICLR), 2018.\n6. Liuyi Yao, Sheng Li, Yaliang Li, Mengdi Huai, Jing Gao, and Aidong Zhang. Representation learning for treatment effect estimation from observational data. In Advances in Neural Information Processing Systems, pp. 2633–2643, 2018.\n7. Bryan Lim, Ahmed Alaa, and Mihaela van der Schaar. Forecasting treatment responses over time using recurrent marginal structural networks. In Advances in Neural Information Processing Systems, pp. 7493–7503, 2018.\n8. Bica, I., Alaa, A.M., Jordon, J. and van der Schaar, M., 2020, September. Estimating counterfactual treatment outcomes over time through adversarially balanced representations. In International Conference on Learning Representations.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: We thank the authors for their engaging submission on an important topic of research. 

A major shared concern of the reviewers is the lack of baseline methods in the empirical study.  While the related work section describes the state of existing invariant learning algorithms, the authors make no direct comparisons in the experiments.  And while there may be no directly comparable algorithm, the wealth of causal representation learning algorithms enumerated by reviewer S8oA could still be used to form predictions that highlight the key differences of the CMLE framework. 

Still though, this work introduces and empirically investigates a new objective for learning invariant representations.  With the additional simulation study and reviewer responses, this work was viewed favorably by most reviewers.  We encourage to the authors to incorporate insights from the simulation study into the manuscript.  
Decision: Accept (Poster)