Improving Multi-Task Generalization via Regularizing Spurious Correlation
Authors: Ziniu Hu
Zhe Zhao
Xinyang Yi
Tiansheng Yao
Lichan Hong
Yizhou Sun
Ed H. Chi
Venue: NeurIPS 2022 Accept (2022)
Keywords: spurious correlation
multi-task learning
multi-task generalization
out-of-distribution generalization

Abstract: Multi-Task Learning (MTL) is a powerful learning paradigm to improve generalization performance via knowledge sharing. However, existing studies find that MTL could sometimes hurt generalization, especially when two tasks are less correlated. One possible reason that hurts generalization is spurious correlation, i.e., some knowledge is spurious and not causally related to task labels, but the model could mistakenly utilize them and thus fail when such correlation changes. In MTL setup, there exist several unique challenges of spurious correlation. First, the risk of having non-causal knowledge is higher, as the shared MTL model needs to encode all knowledge from different tasks, and causal knowledge for one task could be potentially spurious to the other. Second, the confounder between task labels brings in a different type of spurious correlation to MTL. Given such label-label confounders, we theoretically and empirically show that MTL is prone to taking non-causal knowledge from other tasks. To solve this problem, we propose Multi-Task Causal Representation Learning (MT-CRL) framework. MT-CRL aims to represent multi-task knowledge via disentangled neural modules, and learn which module is causally related to each task via MTL-specific invariant regularization. Experiments show that MT-CRL could enhance MTL model's performance by 5.5% on average over Multi-MNIST, MovieLens, Taskonomy, CityScape, and NYUv2, and show it could indeed alleviate spurious correlation problem.

Improving Multi-Task Generalization via Regularizing Spurious Correlation
Ziniu Hul Zhe Zhao?, Xinyang Yi', Tiansheng Yao?, Lichan Hong", Yizhou Sun', Ed H. Chi
University of California, Los Angeles, [bull, yzsun fccs uul.l.aaeee. edu Google Research, Brain Team, ( (zhezhao, xinyang, tyao, lichan, edchi eeeeeeeeogoohhoooooooooo @egoogle.com con
Abstract
Multi-Task Learning (MTL) is powerful learning paradigm to improve generaliza- tion performance via knowledge sharing. However, existing studies find that MTL could sometimes hurt generalization, especially when two tasks are less correlated. One possible reason that hurts generalization is spurious correlation, i.e., some knowledge is spurious and not causally related to task labels, but the model could mistakenly utilize them and thus fail when such correlation changes. In MTL setup, there exist several unique challenges of spurious correlation. First, the risk of having non-causal knowledge higher, as the shared MTL model needs to encode all knowledge from different tasks, and causal knowledge for one task could be potentially spurious to the other. Second, the confounder between task labels brings in different type of spurious correlation to MTL. Given such label-label confounders, we theoretically and empirically show that MTL is prone to taking non-causal knowledge from other tasks. To solve this problem, propose Multi- Task Causal Representation Learning (MT-CRL) framework. MT-CRL. aims to represent multi-task knowledge via disentangled neural modules, and learn robust task-to-module routing graph weights via MTT.-specific invariant regularization. Experiments show that MT-CRL could enhance MTL model's performance by 5.5% on average over Multi-MNIST, MovieLens, Taskonomy, CityScape, and NYUv2, and show it could indeed alleviate spurious correlation problem.
Introduction
Multi-Task Learning (MTL), learning paradigm (Caruana. 1997; Zhang Yang, 2018) aiming to train a single model for multiple tasks, is expected to benefit the overall generalization performance than single-task learning (Maurer al., 2016; Tripuraneni al., 2020) given the assumption that there exists some common knowledge to handle different tasks. However, recent studies observed that, when two tasks are less correlated, MTL could lead even worse overall performance (Parisotto et al., 2016; Zhang et al., 2021). A line of works (Yu et al., 2020; Wang et al., 2021; Fifty et al., 2021) resort performance drop to optimization challenge because conflicting tasks might compete for model capacity. However, both Standley al. (2020) and our analysis in Section 2.2 show that, even with an pver-parameterized model that achieves low MTL training loss, the final generalization performance could be worse than single-task learning. This finding motivates us to think about the following question: Are there any intrinsic problems in MTL that hurt generalization?
One widely studied issue that influences generalization is the spurious correlation problem (Geirhos al., 2019, 2020), i.e., correlation that only existed training datasets due to unobserved con- founders (Lopez-Paz, 2016), but not causally correct. For example, as Beery et al. (2018) discussed, when we train an image classification model to identify cows with biased dataset where cows mostly
"Tii work was done when Ziniu was an intern at Google.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

appear in pastures, the trained cow classification model could exploit the features of background (e.g.. pastures) to make prediction. Thus, when we apply the classifier to another dataset where cows also appear in other locations such as farms or rivers, it will fail to generalize (Nagarajan et al., 2021).
When it comes to MTL setting, there exist several unique challenges to handle spurious correlation problem. First, the risk of having non-causa features is higher. Suppose each task has different sets of causal features. To train single model for all these tasks, the shared representation should encode all required features. Consequently, the causal features for one task could be potentially spurious to the other tasks, and such risk could be even higher with an increasing number of tasks. Second, the confounder that leads to spurious correlation is different. Instead of the standard confounders between feature and label, the nature of MTL brings in a unique type of confounders between task labels, c.g., correlation between tasks' labels could change in different distributions. For example, when we train a MTL model to solve both cow classification and scene recognition tasks, its encoder needs to capture both foreground and background information, and the spurious correlation between the two tasks training set could mislead per-task model to utilize irrelevant information, e.g., use background to predict cow. Given such label-label confounders that are unique for MTL, we theoretically prove that MTL is prone to taking non-causal knowledge learned from other tasks. We then conduct empirical analysis to validate the hypothesis. In summary, we point out the unique challenges of spurious correlation in MTL setup, and show that it indeed influences multi-task generalization.
In light of the analysis, we try to solve the spurious correlation problem in MTL. Among all the knowledge learned in the shared representation layer through end-to-end training, an ideal MTL framework should learn to leverage only the causal knowledge to solve each task by identifying the correct causal structure. Following the recent advances that enable causal learning in an end-to-end learning model (Schölkopf et al., 2021; Mitrovic et al., 2021), we propose a Multi-Tast Task Causal Representation Learning (MT-CRL) framework, aiming to represent the multi-task knowledge via set of disentangled neural modules instead of single encoder, and learn the task-to-module causal relationship jointly. We adopt de-correlation and sparsity regularization over popular Mixture-of- Expert (MoE) architecture (Shazeer et al., 2017). The most critical and challenging step is to learn the causal graph in the MTL setup. which requires distinguishing the genuine causal correlation from spurious ones for all tasks. Motivated by the recent studies that invariance could lead to causality (Ahuja et al., 2020; Koyama & Yamaguchi, 2021), we propose penalize the variance of gradients w.r.t. causal graph weights across different distributions. On a high level, this invariance regularization encourages the causal graph to assign higher weights to the modules that are consistently useful. In contrast, the modules encoding spurious knowledge that cannot consistently achieve graph optimality are assigned lower weights and be discarded by task predictors.
We evaluate our method on existing MTL benchmarks, including Multi-MNIST, MovieLens, Taskon- omy, CityScape, and NYUv2. For each dataset, to mimic distribution shifts, we adopt some attribute information given in the dataset, such as the released time of the movie or district of a building. to split train/valid/test datasets. The results show that MT-CRL could consistently enhance the MTL model's 's performance by 5.5% on average, and outperform both the MTL optimization and robust machine learning baselines. We also conduct case studies to show that MT-CRL indeed alleviate spurious correlation problem in MTL setup.
The key contributions of this paper are as follows:
We are the first to analyze spurious correlation problem in MTL setup, and point out several key challenges unique to MTL with theoretical and empirical analysis.
We propose MT-CRL with MTL-specific invariant regularizers to elleviate spurious correla- tion problem, and enhances the performance on several MTL benchmarks.
2 Analyzing Spurious Correlation in MTL
To systematically analyze the spurious correlation problem in MTT. we first assume that data and task labels are generated by ground-trute causal mechanisms deseribed in Suter et al. (2019). We denote X as the variable of observed data, and each data is associated with K latent generative factors IF = [ representing different semanties of the data (e.g., color, shape, background of an image). We follow Schölkopf et al. (2021) to assume that the data X is generated by disentangled causal mechanisms P(X such that PI II (.))..).

---

CAITL
Y1
Fi
Fi
..
F
Spurtous ractors
as Causal factors F
Cousal factors
Ground-Truth Model
(
(1-)
Could
Need to encode factors for lank
Need to encode factor for lask
causal factors
00O ...O +0-00P Z
O0O ... ...OO)
0
Tal-)
Figure Spurious correlation in Single-Taske Learning is mainly caused by factor-label con- founders CSTT. We could remove spurious fac- tors IF from representation
Figure 2: Spurious correlation Multi-Task Learning could be caused by label-label con- founders CMTL, Factors for both tasks F and mgg need to be encoded and potentially spurious.
As IF represents high-level knowledge of the data, could naturally define task label variable Y, for task as the cause of a subset of generative factors. We denote E as subset of causal feature variables within F that causally related to each task variable Y,. and could define F F F as a subset of non-causal feature variables to task t. such that P(Y = Ff) In other words, changing the values of any non-causal factors in F does not change the conditional distribution.
Note that the discussion so far based on the assumption that the ground-truth causal generative only given supervised dataset
model is known. In real- weal-world learning setting, however, (X, Y) without access to generative factors IF. To solve the task, a neural encoder ((-) required to extract representation Z from the data that encodes the information about the causal factors, top of which a task predictor f(-) could predict the label.
2.1 Spurious Correlation Problem
Based on the ground-truth generative model, an ideal predictor for each task should only utilize the causal factors, and keep invariant to any intervention on non-causal factors. However, in real- world problems, it is hard to achieve an invariant predictor due to the spurious correlation issue due to unobserved confounders Cdist (Lopez-Paz, 2016). Formally, confounders are variables that influence the two connected variables' correlation, and such correlation could change under different distribution (different value Cdist), thus the model exploiting such spurious correlation will fail to generalize. Below we summarize the differences of spurious correlation problems for single-tase task and multi-task learning settings:
Single- Task Learning (STL). As illustrated in Figure 1, the label-factor confounders for single LSTL
task learning dist connects non-causal factors F E FS' and task label Y, bringing in spurious correlation. For example, temperature could confound crime and ice cream consumption. When the weather is hot, both crime rates and ice cream sales increase, but these two phenomena are not causally related. Based on the proof by Nagarajan al. (2021); Khani & Liang (2021), such spurious correlation could lead the model to use non-causal factors, and thus hurt generalization performance.
Multi-Task Learning (MTL). In the MTL setting, there exist several unique challenges to handle spurious correlation. First, the risk of having non-causal features is higher. As is illustrated in Figure 2, the shared encoder & needs encode all the factors causally related each task in the representation Z. Therefore, for each task, all non-overlapping factors from other tasks could be potentially spurious. Second, besides the standard label-factor confounders CSTI. for each single task introduced above, we define label-label confounders Cdist connecting multiple tasks' label {Y}. Such confounder is unique to MTL setting.
As example, consider two binary classification tasks, with Y, and Y, as variables from {+1} } for task label. The two labels' correlation P(Y. = Y,) = mc could change with different confounder Cdist C. We assume the two tasks have nve-lverlap factors F. and F; drawn from Gaussian distribution. We then show MTL model with both two factors as input will utilize non-causal factors:
3

---

Multi-SEM
Multi-MNIST
STL
MTL
STL
MTL
AcCtrain
0.931
0.936
0,981
0.987
Acceal
0,906
0.882
0.874
0.846
Pepar 0.128
0.246
0.261
0.328
(4) Saliency Map of Single Tosk Model
[b] Saliency Map Multi-Takk Model
Figure 3: The gradient saliency map of right- side digit classifier. The model trained by MTL exploits left pixels (spurious) more.
Table 1: Empirical results of multi-task (MTL.)
and single-task learning (STL) model on synthetic datasets with changing dist
Given mc + 0.5. the Bayes Optimal per-task classifier has non-zero weights to non-causal factor. Given mc 0.5 and limited training dataset, the trained per-task classifier will assign non-zero weights to non-causal factor noise.
Proposition 1
Detailed proof is in Appendix A. Therefore, in this linear classification example, when we deploy the trained model to new distribution with changed label-label confounder Cdist C'ATLL the model trained by MTL that utilizes non-causal factors generalize relatively worse. On the contrary, the model trained by STL don't need to encode all causal factors from two tasks. Assuming there is no task-label confounder dist in each task's i dataset, the trained model could remove non-causal factors from representation.
2.2 Empirical Experiments
In the following, we conduct experiments to validate the claims. As there is no existing MTL datasets specifically designed to analyze spurious correlation problem, we construct synthetic Multi- SEM (Rosenfeld et al., 2021) 1) and Multi-MNIST (Harper & Konstan, 2016) 6) datasets with known causal structure to study whether the model trained by MTL indeed exploits more non-causal factors, and how the spurious correlation influences multi-task generalization. Dataset details are in Appendix D.1.
Spurious Score. As we know the ground-trus causal structure for the two datasets, we could quantify how much model utilizes the non-causal factors. Following the gradient saliency map proposed by Simonyan et al. (2014), we calculate the average absolute gradients w.r.t each factor as a ((((()))))))
Grad(F) = (((()))))
which measures how much a model leverage this factor to make prediction. We then define the spurious score Papur the proportion of average gradients over Eress Grad(F) Erer Grod(F)
OF
non-causal feature Papur
Empirical Results. We train a shared-bottons model via Multi-task learning (MTL) and single-task learning (STL) over the two datasets and report both the training and test accuracy with spurious ratio Pepur Table 1. As illustrated, the test accuracies of MTL for both Multi-SEM and Multi-MNIST datasets are both worse than STL. The training accuracies of MTL are very similar to STL, meaning that the performance drop is not due to the optimization difficulty that many previous works try to address. The spurious ratio Papur of MTL is much higher than the STL, which means that it exploits more non-causal factors. To give a more straightforward illustration, we plot the gradient saliency map of the right-side digit classifier for Multi-MNIS" in Figure 3. The model trained by MTL utilizes more left-side pixels, which are non-causal to the final prediction. We also show the results of Multi-SEM with more than 2 tasks in Appendix B. These findings support our hypothesis that with spurious correlation caused by label-label confounder CMTL L models trained by MTL is more prone leverage non-causal knowledge than STL, and thus influence generalization performance.
3 Method
Based on the previous analysis of the spurious correlation problem in MTL, we now introduce Multi-Tase Causal Representation Learning (MT-CRLL) framework with the goal that the per-task predictor only leverages the causal knowledge instead of spurious correlation. The high-level idea

---

of the framework is to reconstruct the ground-truie causal mechanisms introduced in section 2 through end-to-end representation learning. To accomplish this goal, the framework aims to model multi-task knowledge via a set of disentangled neural modules; 2) learn the task-to- graph that is optimal across different distributions. With the correct causal graph as routing layer, per-task predictor only utilizes outputs from causally-related modules, thus alleviating the spurious correlation problem. We introduce the two crucial designs as follows.
causal
3.1 Modelling via Disentangled Neural Modules
In order to alleviate spurious correlation, an ideal MTL model should learn the multi-task knowledge in the shared representation while identifying which part of the knowledge is causally related to each task. However, directly conducting causal discovery is impossible all the knowledge is fused in a single shared encoder. Thus, we seek to adopt modularized architecture in which each module encodes disentangled knowledge, and thus enable modeling causal relationship between task and modules. We adopt Multi-gate Mixture-of-E et al., 2017) architecture tailored for MTL setting, as our underlying model. Specifically, we have K' different neural modules as shared encoders + = [0:+]] --:" Given batch of input data
(MMoE) (Ma et al., 2018), variant of MoE (Shazeer
K
X [ with batch size B, we extract k representations via different neural modules, i.e., Z: = (X) E Bxd, Based on sparsity assumption of the causal mechanisms Parascandolo et al. (2018); Bengio et al. (2020); Lachapelle et al. (2021), only a few modules should be causally related to each task. Therefore, on top of the learned neural modules, graph, aiming to estimate which module is causally related to each task. We model the bipartite adjacency (a.k.a. bi-adjacency) matrix A sigmoid(O) E [0, 1jTxK by applying sigmoid over learnable parameter 0 to enforce the range constraint. Note that original MMoE adopts softmax to get gate vector, which encourages only a small portion of modules being utilized for each task. Our graph modelling allows multiple modules utilized for each task. With the correct graph weights A as routing layer, we could utilize only the causally related modules and make predictions with per-task predictor f-() as (X(X) f.(E. Ar,: (X))..
learn task-to-module routing
Disentangling Modules. One of the main properties of the causal mechanisms we introduced in section 2 is disentanglement, such that each factor represents a different view of the data, and changing the value of one factor does not influence the others. If without explicit constraints, the learned modules' outputs could still be correlated and hinder the causal structure learning. Therefore, we need to add regularization disentangle these modules during training.
Most existing disentangled representation learning methods are under the generative modeling frame- work, e.g. VAE (Higgins al., 2017) or GAN (Chen et al., 2016). However, Locatello et al. (2019) argues that without explicit supervision, it is hard for generative models to learn correct disentangled factors. We therefore only borrow the regularization methods utilized in existing generative disen- tangled representation works (Cheung et al., 2015; Cogswell et al., 2016) to directly penalize the correlation of learned modules. Specifically, we regularize the in-batch Pearson correlation p (Z,, Z,) between every pair of output dimensions from different representation matrices Z; and Zj. as:
Cov Z,) ZZ. Z.) /Co Z,)
p(Z,, Z,)
where Cov (Z, Z,) [Z.
Z,
Z
Z (1)
By minimizing the Frobenius norm of the correlation matrix p for every two different representation pairs, we could enforce the encoder & to extract disentangled representations.
k
Edecor + Adecor
Σ
(X),
Task-to-Module Graph Regularization. Based on sparsity assumption of the causal mecha- nisms (Parascandolo et al., 2018; Bengio et al., 2020; Lachapelle et al., 2021), each task is causally related to only a few modules. To learn the graph structure, existing works (Zheng et al., 2018: Ng et al., 2019; Lachapelle et al., 2020) propose to to fit structural equation model (SEM) with sparsity regularization over the graph weights. We adopt a similar sparse regularization with an entropy balancing term (Hainmueller, 2012) over the bi-adjacency matrix A weights of the task-to-moduu
5

---

routing graph:
E, A
grraph (A) = Aaps ||A||1 - Abot Entropy
Note that the entropy term aims at keeping the causal weights for each module summing all the tasks to be balanced. This could help avoid degenerate solutions in which only a few modules utilized. By combining the two regularizations in (2) and (3) with per-task supervised risk term Re (i, At, ft) E(XY.JD T.e (X), Y.), we get the regularized loss as:
(00, = Σ (o, Av, St) + Liccoor ()) + Legraph (A).
(4)
LET
3.2 Causal Learning via Graph-Invariant Regularization
It is critical and challenging learn the correct causal graph, which requires distinguishing the true causal correlation from spurious ones. Motivated by the recent studies of robust machine learning that a predictor invariant to multiple distributions could learn causal correlation (Ahuja al., 2020; Koyama & Yamaguchi, 2021), we assume the true causal relationship to be optimal across different distributions. To do so, we assume have access to multiple slices of datasets collected from different environments e E & in which the confounder CATL. that controls task correlation might change. For example, one natural choice is consider train/valid dataset split (the setting we utilize in experiment), or assume the training set is split into multiple slices with different attributes. We desire the moskll graph weights A and per-task predictor fe to be optimal across all environments e E &. Formally, we aim to solve the following bi-level optimization problem:
f) s.t. AG, fa E arg min RE (Ø, A, T.EEE. a,A.f
min
A.J
where R& denotes the risk over data slice in environment e. This optimization problem could be regarded as a multi-task version of IRM. Based on Theorem described in Ahuja et al. (2020), by enforcing invariance over sufficient number of environments that exhibit distribution shifts (i.e., changes of confounder dist CMT,), per-task predictors should only utilize modules that are consistently helpful to the task, and assign zero weights to modules that encode non-causal factors to the task, and thus alleviate spurious correlation and help oft-o--distribution generalization. Even if all data are sampled from the same distribution and there are no distribution shifts, invariance could also help eliminate noisy correlation due to the limited training dataset and help in-distribution generalization.
Invariant Optimality of Mask-to-Modde Graph for MTL. As discussed in IRM, the bi-leveled optimization problem in (5) is highly intractable, especially with complex and non-linear Φ. To implement a practical optimization objective, IRM proposes to softly regularize the gradient of the task-predictor at different enviromments enforce it to be optimal:
&.A.f (É(0) A,
ΣΣ LET 3EE
V VA-AN A.. S. R2 ( (.,
min
(6)
However, as is discussed in IRM paper, if the complexity of a task predictor f is much larger than the number of environments, it could learn an over-fitted solution that makes gradient zero but does not achieve invariance. IRM adopts a fixed all-one vector as predictor to reduce complexity. This approach is not applicable to MTL. setup, as the optimal task aesk-predictors fi for different task t could be very distinctive and complex, and we cannot use a fixed uniform predictor for all tasks.
To strike balance between invariance and complexity of multi-task predictors, we propose only to regularize the gradient of the task-to-module routing graph while assuming the complex predictor ft for each task is fixed each iteration. We call this modification as Graph-Invarias Risk Minimization (G-IRM), which is designed specifically to MTL setup:
min o,4.f
(2(0, A, + AG-IRM LG-IRM ( (.,A()) AIf))
By adopting the similar gradient penalty term as adopted in IRM, we define Norm as:
Norm (Ø, Alf) = ΣΣ
VA=A, R ( A, f1)
(8)
IET eEE

---

As we assume ft is fixed for invariance regularization term C--TRM- we only calculate gradient and optimize for & and A, but not updating ft. This could avoid the over peer-parametizedd predictor fr finding a trivial solution to achieve zero gradients instead of learning the correct causal correlation. Similar trick is utilized in (Ahmed et al., 2021). Note that the gradient w.r.t each graph weight means whether module could help reduce the risk for this task. Therefore, by penalizing the invariance regularization, the modules containing non-causal factors will be assigned zero weights.
In the experiments, we observe that at the early optimization stage, the model has non-zero gradients for all parameters, including the graph weights, thus directly regularizing the gradient norm might influence the optimization. Therefore, we propose modified version of gradient regularization CVar that penalizes the variance of the task module graph's gradient on different environments: G-IRM
Var CG-TRM (9, - ΣE LET CEE
V (, Avge (VA=A, 12; Ill
(9)
E
By minimizing CG.TRM: we force all the learned modules to have similar gradients across different environments, and not overfit only some of the environments. still allows some modules to have non-zero gradients as long as it's the same across environments, and relies on loss term C to update these weights, while CG-TRM forces all gradient to be zero. Therefore, LG.TRM is a loose regularization that not influences the overall optimization too much. shares similar intuition of REx (Krueger et al., 2021) that penalizes risk variance, while C VVar G-IRM penalize gradient variance. We provide pseudo-code of MT-CRL framework in Appendix C.
4 Experiment
In this section, we evaluate whether MT-CRL could benefit the performance of MTL models on existing benchmark datasets, and study whether it could indeed alleviate spurious correlation.
Experimental Setup. One key ingredient of our MT-CRL is achieve the optimality of causal graph over different distributions. However, we might not access multiple environmental labels in multi-task learning datasets. Therefore, we adopt a more realistic setup, such that we only assume to have a single validation set that contains unknown distribution shifts (i.e. change of confounder CMTTL) compared to the training dataset. We thus could utilize training and valid sets as two environments to calculate invariance regularization, while we only utilize the training set to caleulate task loss to avoid the task predictor overfits. Note that in this way, our method could get access to the label information in the validation set. To avoid the possibility that the performance improvement is brought by additional label, for all the other baseline methods, we also add the validation data into the training set to calculate task loss and learn MTL model.
most
Dataset. We choose five widely-used real-world MTL benchmark datasets, i.e., Multi-MNIST (Sun, 2019), Movielens (Harper & Konstan, 2016), Tasknomy (Zamir et al., 2018). NYUv2 (Silberman al., 2012) and CityScape (Cordts et al., 2016), and try to determine train/valid/test split such that there exist distribution shifts between these sets. Dataset details are in Appendix D.2. Note that except NYUv2, our data split the same as the default split settings of these datasets, which also try to test model's capacity to generalize across domains.
Baselines. As MT-CRL is regularization framework built upon modular MTL architecture (in this paper we choose MMoE as instantiation, but it can be applied to other modular networks), we mainly compare with two gradient- gradient-based multi-task optimization baselines: PCGrad (Yu al., 2020) and Grad Vac (Wang et al., 2021). We also compare with two domain generalization basclines: IRM (Ahuja et al., 2020) and DANN (Ganin et al., 2016). For IRM adopt different per-task predictors instead of all-one vector to adapt MTL setup, and calculate penalty via Eq. (6).
Hyper-Parameter Selection. For a fair comparison, all methods are based on the same MMoE architecture. Our methods contain a lot of hyper-paramette including some model specific ones such as number of modules (K) and regularization specific ones. To avoid the case that performance improvement is caused by extensive
tuning, we mainly search optimal model hyper- parameter on Vanilla MTL setting, and use for all baselines. For regularization specific parameters, we take Multi-MNIST, the simplest dataset among the testbed, to find optimal combination, and use for all other datasets. Detailed selection procedure and results are shown in Appendix H.
7

---

Methods
Multi-MNIST
MovieLens
Taskonomy
CityScape
NYUv2
Avg.
Vanilla MTL
baseline to calculate relative improvement
Single-Tase Task Learning
+3.3%
+0.2%
-2.5%
-2.4%
-12.2%
-2.7%
MTL PCGrad
+4.5%
+0.2%
+3.1%
+2.1%
+7.4%
+3.5%
MTL GradVVc
+4.6%
+0.3%
+3.5%
+2.1%
47.2%
+3.5%
MTL + DANN
+4.1%
+0.4%
+1.2%
+0.3%
-0.4%
+1.1%
MTL + IRM
+5.0%
+0.4%
+1.1%
+0.6%
-0.1%
+ +1.4%
MT-CRL w/o CC-IRM
+5.9%
+0.2%
+3.2%
+1.5%
+4.3%
+3.0%
MT-CRL with CNorm
+7.8%
+1.0%
+6.5%
+2.9%
+8.0%
+5.2%
MT-CRL with LG-TRM
+8.1 %
+1.1% %
+7.1%
+2.8%
+8.2%
+5.5%
Table Relative Performance improvement of different multi-task learning (MTL) strategies com- pared to vanilla MTL baseline. Detailed results for each task are shown in Table 6-10 in Appendix E.
Trin Caloel Cred
fvaid Thiiti Cauoal Cred
Disentangled Reg.
Graph Reg.
Multi-MNIST Accuracy
deccor
CS-VAE
Cxps
Lea!
/
/
0.915 + 0.018
/
/
/
0.896 + 0.024
/
0.882 + 0.020 /
x
0.891 = 0.016
Too
'baby
1995,
V 0.903 + 0.017
good, club, young, stri,, die lop children' words with MT-CHL: pink, parenthood, Alice jungle
"Strip tcase 1996', Gamedy|Grime 'All Strippers must die', HorrorCCrime 'hangmen also die, "Drema/War
x 0.908 + 0.021
Table 3: Ablation Studies disentangled and Graph regularization components in MT-CRL, evaluated on Multi-MNIST dataset.
Figure 4: Task-to-Module gradients of model without MT-CRL show Module is spurious. MT- CRL could help alleviate spurious correlation.
4.1 Experiment Results
As each task has a different evaluation metric and cannot be directly compared, we calculate the relative performance improvement of each method compared vanilla MTL, and then average the relative improvement for all tasks of each dataset. As summarized in Table 2, the average improvement of MT-CRL with LG-/RM 5.5%, significantly higher than all other baseline methods. The most critical step of MT-CRL is to learn correct causal graph. We therefore report MT-CRL is shown in the last block, C achieve better results
with different invariance regularization. for most datasets than Norm while removing the invariance regularization could significantly drop the relative performance. Compared to IRM which calculate gradient and update per-task predictors, MT-CRL uses disentangled modules and G-IRM to avoid overfitting to achieve invariance, Results show that for datasets with large amount of tasks, e.g., Taskonomy and NYUv2, MT-CRI. significantly outperform IRM, showing the modification more suitable for MTI. setup.
Ablation Studies. We then study the effectiveness of the other two components in MT-CRL, i.e., disentangled and graph regularization. We mainly report the ablation studies on Multi-MNIST in table 3 as it's relatively small that we could quickly get the results of all combinations.
For disentangled regularization, after removing Ldecor, the performance drops from 0.915 to 0.882, which fits our discussion that we cannot conduct causal lcarning over entangled modules. We also explore one classical generative disentangled representation method, i.c., 3-VAE. As shown in the table, the results of using 3-VAE are 0.896, lower than our utilized decorrelation regularization. We hypothesize that this is probably because not all generative factors are useful for downstream tasks. Generative objectives might compete for the model capacity and in addition, the unused factors could be potentially spurious.
Another key component is graph regularization. After removing both Caps and Loat, the performance drops to 0.891. This show that even invariance regularization could penalize non-causal modules, would be better to force their weights to be zero via sparsity regularization, and to be non-degenerate

---

Left Task
05
04
Right Task
-07
-11
Figure 5: (valid-train) Yask-to-Modull gradients of model with MT-CRL on Multi-
via balance regularization. We also conduct ablation studies to remove either Leps or Lbat, and results show both important, and combining the could help achieve the best results.
Case Study. To show that real-world MTL problem indeed have spurious correlation problem and our MT-CRL could alleciate it, we take MovieLens as an example to conduct case study. Each task is for different movie types, and bag-of-word of movie title is of the features. We calculate the mask-to-module gradients @((((=))))) of the vanilla MMoE model without MT-CRI. We then visualize 'train' gradients, which shows how much each module is utilized to fit the training set, and 'valid-train' gradients, which shows how generalizable each module is. We find that module is utilized for children movie, but harmful in valid set, indicating it is spurious feature. We then use Grad-CAM to show that top words of module include strip and die, which not relevant to children movies. One possible reason is that some children movies contain the words club, which often co-occurred with strip and die in crime and war movies. After adding our MT-CRL, the module assigned to 'children' movie attends Pink, Parenthood, Alice and Jungle.
8F
We then show the (valid-train) Taskuto-Module gradients over Multi-MNIST datasets. With MT-CRL, in Figure 14, each module's saliency map only focus on one side of pixels. By looking each task output's 's saliency map, which help model focus only on causal part, compared with Figure 3(b) that have high weights on both. We also show the detailed gradient saliency map and induced task similarity graph of MovieLens, Taskonomy in Appendix G. All these case studies show MT-CRL could indeed alleviate spurious correlation in real MTL problems.
Related Work
Multi-Task Generalization. A deep neural model often requires a large number of training samples to generalize well (Arora et al., 2019; Cao & Gu, 2019). To alleviate the sample sparsity problem, MTL could leverage more labeled data from multiple tasks (Zhang & Yang, 2018). Most works studying multi-task generalization are based on a core assumption that the tasks are correlated. Earlier research directly define the task relatedness with statistical assumption (Baxter, 2000; Ben-David & Borbely, 2008; Lampinen & Ganguli, 2019). With the increasing focus on deep learning models, recent research decompose ground-trui MTL models into shared representation and different task- specific layers from a hypothesis family (Maurer et al., 2016). With such decomposition. Tripuraneni al. (2020) and Du et al. (2021) prove that a diverse set of tasks could help learn more generalizable representation. Wu et al. (2020) study how covariate shifts influence MTL generalization. Despite these findings, the core assumption of task relatedness might not be satisfied in many real-world applications (Parisotto et al., 2016; Zhang et al., 2021), in which tasks could even conflict with each other to compete model capacity, and the generalization performance MTL could be worse than single-task training.
To solve the task conflict problem, number of MTL model architectures have utilized modular (Misra et al., 2016; Lu et al., 2017; Rosenbaum et al., 2018: 8; Ma et al., 2018; Guo et al., 2020) or attention- based (Liu et al., 2019; Maninis al., 2019) design to enlarge model capacity while preserving information sharing. Our work is model-agnostic and could be applied to existing architectures to further solve the spurious feature problem. Another line of research alleviate task conflict during optimization. Some propose to balance the task weight via uncertainty estimation (Kendall et al., 2018), gradient norm (Chen et al., 2018), convergence rate (Liu et al., 2019), or pareto optimal- ity (Sener& & Koltun, 2018). Others directly modulate task gradients via dropping part of the conflict

---

gradient (Chen et al., 2020) or project task's s gradient onto other tasks' gradient surface (Yu et al., 2020; Wang et al., 2021). Though these works successfully facilitate MTL model to converge easier, our analysis show that with spurious correlation, the MTI. model with low training loss could still generalize bad. Therefore, our proposed MT-CRL that alleviates spurious correlation orthogonal to these prior works, and could be combined to further improve overall performance.
Spurious Correlation Problem. Due to the selection bias (Torralba & Efros, 201 I; Gururangan et al., 2018) or unobserved confounding factors (Lopez-Paz, 2016), training datasets always contain spurious correlations between non-causal features and task labels, with which trained models often leverage non-causal knowledge and may fail to generalize oat-Of-Distriuttion (OOD) when such correlation changes (Nagarajan et al., 2021). To solve the spurious correlation problem, some fairmess research pre-define set of non-causal features (e.g., gender and underrepresenteed identity) and then explicitly remove them from the learned representation (Zemel et al., 2013; Ganin et al., 2016; Wang et al., 2019). Another line of robust machine learning research does not assume to know spurious features, but regularize the model to perform equally well under different distribution. Distributionally Robust Optimization (DRO) optimizes worst-case risk (Sagawa et al., 2020). Invariant Causal Prediction (ICP) learns causal relations via invariance testing (Peters et al., 2016). Invariant Risk Minimization (IRM) forces the final predictor be optimal across different domains (Arjovsky et al., 2019). Risk Extrapolation (REx) directly penalizes the variance of training risk in different domains (Krueger et al.. 2021). Another line of work aim at learning causal representation (Schölkopf et al., 2021), i.e., high-level variables representing different aspect of knowledge from raw data input. Most of these works try to recover disentangled causal generative mechanisms (Parascandolo et al., 2018; Bengio et al., 2020; Liu et al., 2020; Mitrovic et al., 2021). Despite the extensive study of spurious correlation single-task setting, few work discuss for MTL models. This paper is the first to point out the unique challenges of spurious correlation in MTL setup.
6 Conclusion
In this paper, we study spurious correlation problem in the Multi-Tase Learning (MTL) setting. We theoretically and experimentally shows that task correlation can introduce special type of spurious correlation in MTL, and the model trained by MTL is more prone to leverage non-causal knowledge from other tasks than single-task learning. To solve the problem, we propose Multi-Task Causal Representation Learning (MT-CRL) which consists of: 1) decorrelation regularizer to learn disen- tangled modules; 2) a graph regularizer to learn sparse and non-degenerate mas-tto-mduule graph; 3) G-IRM invariant regularizer. We show MT-CRL could improve performance of MTL models on benchmark datasets and could alleviate spurious correlation.
Limitation Statement. Our analysis is based label-label confounders. However, existing MTL datasets don't 't provide exact confounder changes to study spurious correlation problem. As mitigation, in analysis part, we create two synthetic datasets, and in experiment part, we adopt train/valid/test split with several attribution differences to mimic confounder changes. To further study spurious correlation in MTL, in the future, we'd like to construct benchmark MTL datasets with known confounder changes (or analyze how some key attribute changes lead to spurious correlation problem), build mathematical model based it, and also explore and visualize which part of knowledge in real-world MTL datasets (e.g. Taskonomy) could be spuriously correlated to other tasks.
Acknowledgement. We sincerely thanks anoymoused NeurIPS reviewers for their constructive comments and suggestions to improve this paper. We thank Huan Gui, Kang Lee, Alexander D' Amour, Xuezhi Wang, Jilin Chen and Minmin Chen for insightful discussion and suggestion for this work. We also thank Ang Li for technical support for running experiments on Taskonomy, and Thanh Vu for running CityScape and NYUv2 experiments.
References
Ahmed, F., Bengio, Y., van Seijen, H., and Courville, A. C. Systematic generalisation with group invariant predictions. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: httttpps:///prreeviiii ew. net www.ormiid--boiiimeoee ZFJ.
10

---

Ahuja, K., Shanmugam, K., Varshney, K. R., and Dhurandhar, A. Invariant risk minimization games. CoRR, abs/2002.04692, 2020. URL https: htRp:h/t:://aatt /ttps///aaxvvvvio wwwplavo.r//aaae wwtpp/s/////aaata.o.......o..aaa /2002. 69....
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. Invariant risk minimization. CoRR,
abs/1907.02880 020999.0.................. 2019. URL http: //rpiv//axxv...///////aa/ .rgg httgg//bs/110000 02893.
Arora, S., Du, S. S., Hu, W., Li, Z.,, and Wang, R. Fine-grained analysis of optimization and gener- alization for overparameterized two-layer neural networks. Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learn- ing Research, pp. 322-332. PMLR, 2019. URL http: http::t/ppccceedninggg .ar.ggsmmm....om www.:saww........ee..........eee p297 aroral aaaral9a.1. html.
Badrinarayanan, V., Kendall, A., and Cipolla, Segnet: deep convolutional encoder-decoder architecture for image segmentation. CoRR, abs/1511.005611 2015. URL org/abs 11111 00561.
Baksalary, J. K. and Baksalary, O. M. Particular formulae for the moore-penrose inverse of a column- wise partitioned matrix. Linear Algebra and its Applications, 421(1):16-3,, 2007. ISSN 0024-3795. doi: https://doi.org// 10006//.....20000.....33... URL https: /www. sci erienneddieeee rectt com/ Special Issue devoted to the 12th ILAS
accianee/eriiieee////////2270000 Conference.
Balaji, Y., Farajtabar, M., Yin, D., Mott, A., and Li, A. The effectiveness of memory replay in large scale continual learning. CoRR, abs/2010.02418, 2020. URL httttttt//axxiv..ooom abs/ 2010. 02418.
Baxter, J. model of inductive bias learning. Journal of artificial intelligence research, 12:149-198. 2000.
Beery, S., Horn, G. V., and Perona, P. Recognition terra incognita. In Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XVI, volume 11220 Lecture Notes in Computer Science, pp. 472-489, Springer, 2018. doi: 10. https:: http:/doo.oo www.i.org/aao 10. 1007 0-077-8--00001700---01
28. URL
Ben-David, S. and Borbely, R. S. A notion of task relatedness yielding provable multiple-task learning guarantees. Mach. Learn., 73(3):273-287, 2008. doi: 10. 10.100771110944400--554533- URL www.ooo.o/........ hitpss://do wwa.ddi..oogg/11
Bengio, Y., Deleu, T., Rahaman, N., Ke, N. R., Lachapelle, S., Bilaniuk, O., Goyal, A., and Pal, C. J. A meta-transfer objective for learning to disentangle causal mechanisms. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababu, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:t/////ttt http:///eeeevvvvveee / net./ forumm (
Cao, Y. and Gu, Q. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché-Buc, F., Fox, E. B., and Gamnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 20/9, Vancouver, BC, Canada, pp. 10835- 10845, 2019. URL https: httpp:::ppooccceenneeee/oo neurips. wttpiiss..ccppppeeeee cc/paper/ 2019 9 has99/hahhccccc caaahh/cf9cc54eee htt.aasceee...eaaaaaeea.aaaaaaa.
html.
Caruana, R. Multitask learning. Machine learning, 28(1):41-75, 1997.
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. Info- gan: Interpretable representation learning by information maximizing generative adversar- ial nets. In Lee, D. D., Sugiyama, M., von Luxburg, U. Guyon, I., and Gamett, R. (eds.), Advances in Neural Information Processing Systems 29; Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 6 hash/ 2172-2180, 2016. URL wtt.thtttts///oeeeeeeeeeeeee ngs neurips new.uur wttps/urrr..c..pprrr/////./ (esd00l1E6aaabaateccaaacioi.. 9ebb- Abst aab. html.

---

Chen, Z., Badrinarayanan, V., Lee, C., and Rabinovich. A. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Dy, J. G. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, Stockholmsmüsssnn Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 793 PMLR, 2018. URL http: http:::://pcceeddrgge ml www.merss.rrr..e.mmmmmmm.mm.mm.e (tress/v8 //c0//cheen/// 8a. html.
Chen, Z.,, Ngiam, J., Huang, Y., Luong, T., Kretzschmar, H.,, Chai, Y., and Anguelov, D. Just pick sign: Optimizing deep multitask models with gradient sign dropout. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL hash/ 200266602222 7 55a 94 94a44499cc3444ebbrr26-----2-68 .ttttratt.............
/ttp::e:s////ccpeeepeeee ngs. neur i ps cc htc//apprr/2000
Cheung, B., Livezey, A.. Bansal, A. K., and Olshausen, B. A. Discovering hidden factors of variation in deep networks. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL http: //a//aiiivi////////////eea///aaa org/ abs / 14 412.1444 5583.
Cogswell, M., Ahmed, F., Girshick, R. B., Zitnick, L., and Batra, D. Reducing overfitting in deep networks by decorrelating representations. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL TtLhttp:// httaaaxivvv... http://axiiv.....aaaa...
11111.1 06068.
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. The cityscapes dataset for semantic urban scene understanding. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 3213-3223, IEEE Computer Society, 2016. doi: 10.1109/CVR..2066.335 URL https: ctwoioorrg/00 111111111 /9/CCPP.CC... 201 6. 350.
Du, S. S., Hu, W., Kakade, S. M., Lee, D., and Lei, Q. Few-shot learning via learning the representation, provably. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: //pp://ooeeeeieeeeeee net/forumts forum? 2===W222xxxwTT
Fifty, C., Amid, E., Zhao, Z., Yu, T., Anil, R., and Finn, C. Efficiently identifying task groupings for multi-task learning. CoRR, hass2209.046111 2021. URL https: aaaaaaixvvvvv/./.a//a/aa org. hatp/bb/221000 04617.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. S. Domain-adversarial training of neural networks. J. Mach. Learn, Res., 17: 59:1-59:35, 2016. URL http: /mmr.mmmr.../// wwwoorgg//o.... wwapg//aper///// /tress/v1/// /111/v1111111/1 21111/151191..111 html.
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W. Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: hectps://openrvvv/s id=Bygh 900KXx
net ffoum?
Geirhos, R., Jacobsen, J., Michaelis, C., Zemel, R. S., Brendel, W., Bethge, M., and Wichmann, F, A. Shortcut learning in deep neural networks. Nat. Mach. Intell., 2(11):665-673, 2020. doi: 1031/ 020-00257-22 URL https: :/tp::::///..oooe org/ 103 8 s4225 0000--000000-2--
Guo, P., Lee, C., and Ulbricht, D. Learning branch for multi-task learning. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3854-3863. PMLR, 2020. URL http:/ / ppwppeeiingggs.cepee mlr wwwpppeessseveee. vvllvvll http//uoeeeeeeeeeo html.
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S. R., and Smith, N. A. Anno- tation artifacts in natural language inference data. In Walker, M. A., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Compu- tutional Linguistics: Human Language Technologies, NAACL-HLTT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pp. 107-112, Association for Computational Linguistics, 2018. doi: 10. 18653/v 1am11820111
URL https: / do org /0. w0//////////nn/
12

---

Hainmueller, J. Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political analysis, 20(1):25 46, 2012.
Harper, F. M. and Konstan, A. The moviclens datasets: History and context. ACM Trans. Interact. Intell. Syst., 19:1:19:,,, 2016. doi: 10.1145/2827872. URL setpps://dot otg/:.ooo....ooooo 1145/2827872.
Higgins, I., Matthey, L.,, Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL net/ Eorum? SyyruumuUgSy
Kendall, A., Gal, Y., and Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In 2018 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 7482-7491. IEEE Computer Society, 2018. doi: 2011099VVR.2201.000777 (et__20188hhttt (tttmmm/endal... 2eFer__nn___RR_21____ watoor1ppppppp___eee__ pappp.atm...... html.
URL http: http:///ppaacccssseeee thecvf. wwt.coo/ccco//ooo http://cnneetttcceeeeee Multi Multi-Task-
Khani, F. and Liang, P. Removing spurious features can hurt accuracy and affect groups dis- proportionately. In Elish, M. C., Isaac, W., and Zemel, R. S. (eds.), FAceT '21: 2021 ACM Conference on Fairness, Accountubility, and Transparency, Virtual Event / Toronto, Canada, March 3- 2021, pp. 196-205. ACM, 2021. doi: 10.1 10.1145/3422188.4455588 URL https: //doi. www.o//////oo 11t.///..11111/..1.. 42188.288... 34115333442888888888.83 3445883.
Koyama, M. and Yamaguchi, S. When is invariance useful in an out-of-distribution generalization problem? CoRR, abs/2008.0180 01883, 2021. URL https: htt///xxiiv/////////////////.avl www.wwwwwww..aaa..aoooooerrere.e httpg//bs/20000 /200 01 188..
Krueger, D., Caballero, E., Jacobsen, J., Zhang, A., Binas, J., Zhang, D., Priol, R. L., and Courville, A. C. Out-of-distribution generalization via risk extrapolation (rex). In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5815-5826. PMLR, 2021. URL http: http:///poceeeneeeeeeeee ml mllr.press/v19/ krueger21a html.
Lachapelle, S., Brouillard. P., Deleu, T., and Lacoste-Julien. S. Gradient based neural DAG learning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL http::::///ppeevviiieee net. / forum? ?imidd= rk1bKA4YDS.
Lachapelle, S., López, P. R., Sharma, Y., Everett, K., Priol, R. L., Lacoste, A., and Lacoste-Julien, S. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. arXiv preprint ar 1rXiv:2107.
2021.
Lampinen, A. K. and Ganguli, S. An analytic theory of generalization dynamics and transfer learning deep linear networks. In 7th International Conference on Leurning Representations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019. OpenReview.net, 2019. URL https: // ///ppeneevew..ner..... fort/ffoum iidrrdddyy AddrryMMoCqte
Liu, C., Sun, X., Wang, J., Li, T., Qin, T., Chen, W., and Liu, T. Learning causal semantic representation for duitt--iiitiibuutinn prediction. CoRR, abs/201 681, 2020. URL https: //aaxxivorrg/aaaaaee.e.a abs 2011. 01681.
Liu, S., Johns, E.,, and Davison, A. J. End-to-end multi-task learning with attention. In IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 20/9, Long Beach, CA, USA, June 16-20, 2019, pp. 1871-1880. Computer Vision Foundation IEEE, 2019. doi: 10.1 10.1109//VP A.R.2019.0019,, URL http: optp/cccaaccss...ooo/o thecvf. com cowm//oonee_ccc/___/0.. Task Tatt End-To-Ener htg.aaaeeee...aii.ooooooaaaaaatn html.
/html/ /Liv_
wi
papeer
13

---

Locatello, F., Bauer, S., Lucic, M., Rätsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions the unsupervised learning of disentangled representations. In Chaudhuri. K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume of Proceedings of Machine Learning Research, pp. 4114-4124 4124. PMLR, 2019, URL http:::/ppocceeennggee/o/o/ html.
alrrpprss. v www.telllll.....
Lopez-Paz, D. From dependence to causation. arXiv: Machine Learning, 2016.
Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., and Feris, R. S. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. 20/7 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1131-1140. IEEE Computer Society, 2017. doi: N0.1109/CVPR.2017..222 URL https: / /wo..ii..///////// wwwgooogg/o.ooooooooo 10 1109 /C//RCC/R../// 2017 126..126.
Ma, J., Zhao, Z., Yi, X., Chen, J., Hong, L., and Chi, E. H. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts In Guo, Y. and Farooq, (eds.). Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018. pp. 1930-1939. ACM, 2018. doi: 011145/321981..2220000 URL https: ///://do //oi.ii..// org 10. 1145/1 14445/2115 /2////219199.1 3220007.
Maninis, K., Radosavovic. I., and Kokkinos, tiple tasks. In IEEE Conference CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 1851-1860. Com- puter Vision Foundation / IEEE, 2019. doi: 10.1 10.1109/CVPR.2019.000
Attentive single-tasking of mul- Computer Vision and Pattern Recognition,
URL
http:/ httt:://opeaaceess http/ppppcccssseeeeeeeee thecvf. com wtt.eee.coo/cooteeettcomm /wtp//entcvp_r2OOO000C 9/html1mmlmmmmm /Maninis.
201 p_paper htmll.
Maurer, A., Pontil, M., and Romera-Paredes, B. The benefit of multitask representation learning. J. Mach. Learn. Res., 17:81:1-81:32, 2016. URL htr.lttt//////rrrg//////// http://iww...rr.ppppppp///////// html.
Misra, I., Shrivastava, A., Gupta, A., and Hebert, M. Cross-stitch networks for multi-task learning. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 3994-4003. IEEE Computer Society, 2016. doi: 10. 10.1109/CVPR. 2016.433. URL https: /t///oooooooooooooooo wwwoooog/oooooooooooooooo/...1 11 109 CVCR. 2016 ..33.
Mitrovic, J., McWilliams, B., Walker, J. C., Buesing. L. H., and Blundell, C. Representation learning via invariant causal mechanisms. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:/ openreview. net/ forum
Nagarajan, V., Andreassen, A., and Neyshabur, B. Understanding the failure modes of out- of-distribution generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: //openreview net. /ett/forum forum? Sorumiid==STDGNETTT
Ng, I., Fang, Z., Zhu, S., and Chen, Z. Masked gradient- based causal structure learning. CoRR, abs/1910.08527, 2019. 2019, URL http: /ttp://aaxiivvv/ org/ abs 910. 08527.
Parascandolo, G., Kilbertus, N., Rojas-Carulla, M., and Schölkopf, B. Learning independent causal mechanisms. Dy, J. G. and Krause, A. (eds.). Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4033 4041. PMLR, 2018. URL http:/ / pwwpppeeoenng...eppepppppp mmr. http::/ss.....p.p...e.....m.eeee
pitess/v80... htt////0/paaaaaaaa/a 188a 88a. html.
Parisotto, E., Ba, J., and Salakhundinov, R. Actor-mimic: Deep multitask and transfer reinforcement learning. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference Learning Represen- tations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
URL WtRLhtt:://axiit
abs/ /1ss..11.... 01111.63122121111 06342.
Peters, J., Bühlmann, P., and Meinshausen, N. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society. Series B (Statistical Methodology), pp. 941-1012, 2016.
14

---

Rosenbaum, C., Klinger, T., and Riemer, M. Routing networks: Adaptive selection of non non-linear functions for multi-task learning. In 6th International Conference on Learning Representations. ICLR 2018, Vancouver, BC, Camada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https::/p//.... net./ htt.ps:/opeeereeiiiieeeeee h////pppnrvvveeeee
Rosenfeld, E., Ravikumar, P. K., and Risteski, A. The risks of invariant risk minimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://apenreviee. net forum?id=
BENIBVE J- 42.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL attps://openretteee net. IyxGuJIFvS.
net tete/fformmmt forum?id-
Schölkopf. B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio. Toward causal representation learning. Proc. IEEE, 109(5);612-334, 2021. doi: 10.1 10.1109/IPROC. 2021.3058954. URL https: //oii.oooo//// .arooi.orr//oo 0. 1109/JPROOO 2021 33058954.
Sener, O. and Koltun, V. Multi-task learning multi-objective optimization. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa- Dasa-Bianchi, N., and Garnett, R. (eds.), Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Infor- mation Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 525 536, 2018 URL https: htttttt::// pttpe//eoccceeeeeeeeeeeeeoo.s.se neurips ct/ htte/paper22210 //ah//h/hhhhh//// 33wee345553333333333333333 f Absereetbbtraattttt... httl..
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, V., Hinton, G. E., and Dean, J. Outra- geously large neural networks: The sparsely- mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Confer- ence Track Proceedings. OpenReview.net, 2017. URL https: http:///ppeeeeeeeeeeeeeee. net forum? id=BlckMDq1g.
Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. Indoor segmentation and support inference from RGBD images. In Fitzgibbon. A. W., Lazebnik, S., Perona, P., Sato, Y., and Schmid, C. (eds.), Computer Vision ECCV 2012 12th European Conference Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V, volume 7576 Lecture Notes in Computer Science, pp. 746-760. Springer, 2012. doi: 10. 001100779788---6433331554- 000000077778---4-- 0---5442-331----5555
URL
https: /ttp://ooiiio.oooo http///i.oggg//.//
Simonyan, K., Vedaldi, A., and Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Bengio, Y. and LeCun, Y. (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014. URL http: awwia.iiivv.... abs//b///11 6034.
Standley, T., Zamir, A. R., Chen, D., Guibas, L. J., Malik, J., and Savarese, S. Which tasks should be learned together in multi-task learning? In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 9120-9132. PMLR, 2020. URL http: // ixproceedinger wng.mnlmmmm............mm....... ..resss///////e / 19/99///t/ stssnlaenyye html.
Sun, S.-H. Multi-digit mnist for few-shot learning, 2019, URL https:///ithub. 2hachaal11//Matttiiiiiiioioo
Suter, R., Miladinovic, D., Schölkopf, B., and Bauer, S. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6056-6065. PMLR, 2019. URL http: http:://ppocceeanngeeeeoo//////e mlr. Sel.L.prees/vv1 suterl 19t.hmml.
Torralba, A. and Efros, A. A. Unbiased look at dataset bias. The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 201 1, Colorado Springs, CO, USA, 20-25 June 2011, pp. 1521-1528. TEEE Computer Society, 2011. doi: 10.1109/CVPR.201 5,99537777 URI. https: / /doi..../////// org/ 110.0..19///0001 2011. 5995347.
15

---

Tripuraneni, N.. Jordan, M. L, and Jin, C. On the theory of transfer learning: The im- portance of task diversity. In Larochelle. H., Ranzato, M., Hadsell, R., Balcan, M.. and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL http:////oocoedeeeeeeee neurips ccc httpppppp/222///aaaa 5958 wtf.ffccce...ffee322211111533333
Abstract html.
Wang, T., Zhao, J., Yatskar, M., Chang, K., and Ordonez, V. Balanced datasets are enough: Estimating and mitigating gender bias in deep image representations. In 2019 IEEE/CVF In- ternational Conference
Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 5309-5318. IEEE, 2019. doi: 10.1109//CCV22010000044 URL
https: //toi//doii//oo org/ 10. 110111199//0 ICCV 2019.00000 00541.
Wang. Z., Tsvetkov, Y., Firat, O., and Cao, Y. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In 9th International Conference on Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: http://pppeeeeeeeeeeee nwt. forum? dndiiviFFiiiii. _H_
Wu, S., Zhang, H. R., and Ré, C. Understanding and improving information transfer in multi-task learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: http://wppneeee.eeeeee n Corum?id-20 12hkBtDB.
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL http:t http::/ppp///.o.eeegnnngge//// (esaaffffda99de55305 -ttees5s00.0222-0ooottttttt....d Abstract.
httpps.ccpppppppp pttp//pper/22//////e/// 2ttp//222///hh////// hash//
7amir, A. R., Sax, A., Shen, W. B., Guibas, I.. J., Malik, J., and Savarese, S. Taskonomy: Disentangling task transfer learning. In 2018 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 3712- 3722. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109/CVPR. 2018.00391. URL http: /opp//cceecccseseeeeeeeeesssssse theevf. com/ wattm//ooneetttt______e Gamir_Taskonea www..aacooooooi.iieetangnnniinia Ttwpiisettaagningnasa__.cccc_ 2at 201.ppprrtt......m.mmm
Zemel, R. S., Wu, Y., Swersky, K., Pitassi, T., and Dwork, C. Learning fair representations. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 325-333. JMLR.org, 2013. URL http: htt.:::/pppoceeinngeee wwr.nnsmmmrr...m www.s...ss..p..................e /vress/v23/ v2 zeme 11 13. html.
Zhang, W., Deng, L.. Zhang, L., and Wu, D. A survey on negative transfer. IEEE TRANSACTIONS
ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2021.
Zhang. Y. and Yang, Q. An overview of multi-task learning, National Science Review, 5(1):30-43, 2018.
Zheng, X., Aragam, B., Ravikumar, P., and Xing, E. P. Dags with NO TEARS: continuous optimiza- tion for structure learning. Bengio, S., Wallach, M., Larochelle, H., Grauman, K., Cesa- Bianchi. N., and Gamett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrlPS 2018, December 3-8, 2018, Montréal, Canada, pp.
2018 URL https: httLLhhttss::/pooeeeeeeee http::::ss////pcpeepeeeoo www.ips.......... cc/ hptp::/AAAhhnnnnaaa..aaaaaaaeee2 httm.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The main contribution of this paper is to point out the unique challenges of spurious correlation problem MTL setup, and propose a workable solution. This has been discussed in abstract and intro.
16

---

(b) Did you describe the limitations of your work? [Yes] We've clearly stated the limitations and future directions to improve this work in Conclusion section.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss how MTL could help solve label scarcity problem in related work. Thus, for some task with evil purpose, MTL could still help their performance, which has negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] I've carefully read the guidances and ensured our paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of theoretical results? [Yes] We state the assumption of proposition 1. Specifically, we assume if MTL has spurious task correlation, then the model could utilize non-causal factors.
(b) Did you include complete proofs of all theoretical results? [Yes] The proof for Theo- rem 1 is described in Appendix A.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] They are all included in the supplemental material.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] They are specified in Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] The detailed results with error bar is listed in Table 6-10 in Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] They are specified Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes, we' we've properly cite all used code and data in Appendix.
(b) Did you mention the license of the assets? [Yes] I cite the github link with license.
(c) Did you include any new assets either in the supplemental material as URL? [Yes] submit our own code in supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We do not use any personal data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We do not use any personal data.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "r_zrnPdTvZW", "reviewer": "Reviewer_ZKY5", "summary": "The submission points out the risk of including non-causal features for one task when learning representations in the multi-task settings, accompanied by both a theoretical proof and an empirical analysis.\nIt then proposes a graph-disentanglement-based method, which is inspired by the disentangled causal mechanism and the invariant risk minimization, to address the problem.", "questions": "Please see the Weaknesses listed above.", "limitations": "The paper has provided a clear statement on its limitations after the conclusion.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n- It points out a novel yet important research problem -- the risk of including non-causal features for one task when learning representations in the multi-task settings, as shown in Figure 2.\n\n- It also gives theoretical proof as well as an empirical experiment on synthetic data to show that the problem does exist. \n\n- The proposed causal-graph-based method outperforms previous multi-task methods such as PCGrad and GradVec as well as single-task causal methods such as IRM on multiple multitask datasets that contain unknown distribution shifts.\n\nCons:\n\n- The disentanglement regularization shown in Eq. (1) assumes **linear correlation**, which will fail miserably with a deep feature extractor. It is highly possible that, with the regularization enforced, the multiple neural modules can still possess the same information even if they are linearly decorrelated (in other words, they can be **non-linearly correlated**).\n\n- As pointed out in Line 225, ``if the complexity of a task-predictor is much larger than the number of environments, it could learn an over-fitted solution that makes gradient zero but does not achieve invariance.'' It seems like the proposed alternative (Eq 8 and Eq 9) can still face the same issue. Have I misunderstood anything?\n\n- It contains a vast number of hyperparameters. It seems possible that the performance gain may come from hyperparameter tuning rather than from removing the spurious correlation. Is there any analysis of hyperparameter sensitivity? Does the proposed method underperform the baselines if the hyperparameters are not appropriately set?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nCjItRNZ4sh", "reviewer": "Reviewer_nWqE", "summary": "This work propose Multi-Task Causal Representation Learning framework to alleviate potential spurious correlation in Multi-task learning. Authors use three regularization methods to obtain the correct disentangled factors for experts, the sparsity for the Task-to-Module graph and the invariant causal representation respectively, which based on the Multi-gate Mixture-of-Experts architecture. Experiments on various Multi-task benchmarks and sufficient ablation studies have empirically proved the effectiveness of the method.", "questions": "How to determine and select the hyperparameters is reasonable when introducing so many hyperparameters in the method?\nCould authors give a more explicit discussion about the conflict between distinctive and complex task-predictors and the invariant?", "limitations": "Not relying on the MMOE architecture may have more impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: Authors study the problem of spurious correlation existing in multitask learning which has been noticed but rarely addressed directly to the best of my knowledge. Authors combine many methods to implement their claim. While neither their method is novel, their combination is novel and purposeful. And they clearly identify related work and place it in context in their paper.\n\nClarity: The paper is very well written; clear, precise, and easy to understand. I was able to follow the main contributions without issues. The paper contains many various and multi-level experiments which well prove the effectiveness of their method. \n\nSignificance: Alleviating spurious correlation is an interesting direction to multitask learning and may be the key to solving negative transfer. Experimental results show that this method has a general improvement over all datasets, but not significant improvement in natural CV datasets. \n\nAuthors discuss the hyperparameters tuning in the appendix. Among all the hyperparameters, I care about the number of experts $K$ that determines how many factors the feature extractor learned, on the one hand, determines the difficulty of training. Authors just use $K = 8$ by default and don’t tune it. I think it is beneficial and important to further discuss whether imposing disentangle regularization with fixed $K$ will reduce the capacity of the model, and whether $K$ should be adjusted according to the complexity of the dataset. Adding the empirical risks of methods on training datasets maybe helpful for comparing the ability of the fitting ability. And some hyperparameters $\\lambda_{decor},\\lambda_{sps},\\lambda_{bal}$ are not discussed, did I miss anything?\n\nAnd I have an issue on $\\mathcal{L}^{Var}_{G-IRM}$. The intuition behind this penalty is in conflict with the fact that distinctive and complex task-predictors are necessary for multitask learning. I'm not an expert in this area so please let me know if I've misunderstood something.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Vf3pMSzPmMn", "reviewer": "Reviewer_kRci", "summary": "This paper studies the problem of the lack of the generalizability of multi-task learning due to the spurious (no causal relationships) correlations between tasks. Such spurious correlations may wrongly be used during the learning process leading to bad generalization. Furthermore, similar spurious correlations on the label side are also explored. In the setting of label-label confounders, this paper shows that multi-task learning is prone to exploiting the undesired non-causal knowledge from other tasks. The argued problem of spurious correlations is tackled by using the proposed multi-task causal representation (MT-CRL) framework, MT-CRL represents multitask knowledge in a disentangled fashion and exploits the MTL-specific invariant regularization to learn the causality. The paper conducts experiments on multiple benchmark papers to demonstrate the effectiveness of the proposed method.", "questions": "Most of my questions are answered in the supplementary material. I am wondering if the task similarly graph of Figure 9 (in supplementary material) for Taskonomy datasets looks similar to the Taskonomy datasets or not?\n", "limitations": "There is no obvious severe limitation of this paper besides what the authors have already mentioned in the paper.\nI can say nothing regarding the negative social impact of this paper from my reading.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n* The problem studied in the paper, although not very original, is meaningful in the context of multi-task learning.\n\n* The proposed method, especially causal learning by graph invariant regularization is interesting. The practical consideration on optimizing the intractable bi-level optimization using the graph-invariant risk minimization is a valuable technical contribution.\n\n* The paper is well written and easy to follow due to its structure, examples, and interesting discussion.\n\n* The experimental results validate the effectiveness of the proposed method over the established baseline. Experiments on five benchmark datasets make the experimental section solid.\n\n* Supplementary material provides additional analysis, discussion, and results that further support the contributions.\n\nWeakness:\n\n* This paper has no serious weakness except the missing evaluation in the case of known confounder changes. However, this weakness has clearly been discussed in the limitations.\n\n* The contribution of the term L_G_{IRM}^var is rather marginal in Table 2.\n\n* Only the relative improvements (Table 2) and losses (Table 8, supplementary material) are reported on Taskonomy dataset. The performance evaluation on each task and the improvements would be interesting. Furthermore, only a subset of the available tasks is used.\n\n* Ablation results reported in Table 3 appear to be incomplete. Only a subset of cases is reported.\n\n* Minor: equations are not punctuated. Typo in L201.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7X7WkbHPpaT", "reviewer": "Reviewer_vBSD", "summary": "The submission discusses a way in which multi-task learning can potentially promote the use of features that might generalize poorly in OOD settings. This can happen when labels for different tasks (involving different features) are correlated in the training set but not outside of it.\n\nA method is proposed to improve generalization in such situations by imposing (soft) modularity on the predictive format. Specifically, a (soft) sparse connectivity matrix is learned connecting sets of encodings (modules) to tasks, with training losses to encourage de-correlation among the modules, sparse module-usage, and discourage degenerate solutions such as dead modules. Finally, an invariance penalty inspired from IRMv1 [1] and V-REx [2] is applied on the connectivity matrix to promote the same weighting of modules across different environments, with the assumption that spurious label-label correlations across MTL tasks are likely to be unstable across different environments.\n\nExperiments on a range of synthetic and realistic datasets (with synthetic splitting in all cases to simulate IID-OOD settings) showcase improvements over baselines.\n\n\n[1] Invariant risk minimization, Arjovsky et al., 2019.\n\n[2] OOD generalization via risk extrapolation, Krueger et al., 2020.", "questions": "1. The most major question I have is: how is hyper-parameter selection being performed? The precise method is unclear to me from my reading, but my current understanding is that since training and validation splits correspond to two environments with the invariance penalty applied for these two sets, non-invariance hyper-parameters are tuned on the validation set. If this is the case, how is the invariance penalty hyper-parameter tuned? Is this done at a second stage, with non-invariance hyper-parameters fixed from the previous stage? A related question: for every baseline method, are all hyper-parameter tuning-choices made on the same OOD validation set as for G-IRM?\n\n\n2. For the modular architecture, was the feature dimension scaled up by K, or divided into K parts from the same dimensionality as in baselines? This could imply different choices for baseline comparisons.\n", "limitations": "The authors have provided a limitation-statement.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The discussion of the possibility of a multi-task learning setup inadvertently promoting the learning of less-stable features due to label correlations among the tasks is original, to my knowledge. The robustness-method is largely derived from existing works for the most part, but there are some elements of interest. For example, fixing f_t in the invariance penalty is an interesting trick (although something similar was done in [3]). The case study with MovieLens seems like a nice example of how such task-label correlations could arise.\n\nI find the overall quality of the paper to be good. There are a number of relatively minor typos/writing issues I’ve detailed below, but I found the presentation reasonably clear on the whole.\n\nMiscellaneous thoughts:\n\n — In L126, it might make more sense to rephrase “cannot generalize” as “perform relatively less effectively”, since presumably the correlation between the label and the causal factors is much stronger (for example, Table 1 only shows 2-3% drops and not utter failure). This also suggests a natural baseline can often simply be ERM training drawn out over more epochs/iterations, since with longer training, gradient descent can often be expected to unlearn any high-bias but non-stable features (the “noisy correlations” in L218-L220). In reality, for neural network training, other aspects to do with training dynamics crop up which might well lead to more drastic failure, but for the purposes of the theoretical arguments being made here, the outlook is less gloomy.\n\n — As acknowledged in the submission, invariance across a sampling of environments is not guaranteed to capture causality, and comes with several assumptions. Perhaps it is more appropriate to not imply that works such as these achieve causal discovery or even approximate it, but rather that they simply learn robust features, which are more likely to be stable in OOD settings. Also, since A is a soft-matrix (even with the sparsity penalty), it’s not clear to me if it makes sense to refer to it as a causal “graph”.\n\n — I’m not sure if label-label correlations are primarily the issue; the real problem seems to be the feature-feature correlations, if I’ve understood the setup correctly (and this boils down to the usual feature-label correlations). The two seem equivalent, but consider introducing label-noise: now the label-label correlation rate can drop, and depending on the strengths of the correlation-rate and the label-noise, the label-label correlation can drop to negligible rates while the feature-feature correlation rate stays where it was. I understand the point in the submission that the underlying reason behind the correlation is that the MTL setup promotes the correlation, but I was wondering if a more direct way to view this problem would be that the features corresponding to the two tasks are correlated, rather than the labels. Just a suggestion for the authors to consider.\n\n\n[3] Systematic generalization with group-invariant predictions, Ahmed et al., 2021.\n\nTypos:\n\n — Fig 2 caption should say F_b instead of F_a both times\n\n — Missing period at end of title for Section 2.1\n\n — L133: “exploit” —> “exploits”\n\n — L143: “The” —> “the”\n\n — L170: “extracts” —> “extract”\n\n — L177: “prediction” —> “predictions”\n\n — L183: “disentangle” —> “disentangled”?\n\n — L186: Missing period at end of sentence.\n\n — L186-L187: Cheung 2015 and Cogswell 2016 are described as involving “generative disentangled representation works”, but I believe both of these works deal with disentangling representations in classification settings, not generative.\n\n — L211: the per-task predictor was defined as f_t, not g.\n\n — L215: “exist” —> “exhibit”?\n\n (Few others, running spell-check should fix)\n\n==================================\nPost-rebuttal: I've read the rebuttal, and it answers my questions adequately on the whole. I'm raising my score to 7.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept