Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of the lack of the generalizability of multi-task learning due to the spurious (no causal relationships) correlations between tasks. Such spurious correlations may wrongly be used during the learning process leading to bad generalization. Furthermore, similar spurious correlations on the label side are also explored. In the setting of label-label confounders, this paper shows that multi-task learning is prone to exploiting the undesired non-causal knowledge from other tasks. The argued problem of spurious correlations is tackled by using the proposed multi-task causal representation (MT-CRL) framework, MT-CRL represents multitask knowledge in a disentangled fashion and exploits the MTL-specific invariant regularization to learn the causality. The paper conducts experiments on multiple benchmark papers to demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: Strength:

* The problem studied in the paper, although not very original, is meaningful in the context of multi-task learning.

* The proposed method, especially causal learning by graph invariant regularization is interesting. The practical consideration on optimizing the intractable bi-level optimization using the graph-invariant risk minimization is a valuable technical contribution.

* The paper is well written and easy to follow due to its structure, examples, and interesting discussion.

* The experimental results validate the effectiveness of the proposed method over the established baseline. Experiments on five benchmark datasets make the experimental section solid.

* Supplementary material provides additional analysis, discussion, and results that further support the contributions.

Weakness:

* This paper has no serious weakness except the missing evaluation in the case of known confounder changes. However, this weakness has clearly been discussed in the limitations.

* The contribution of the term L_G_{IRM}^var is rather marginal in Table 2.

* Only the relative improvements (Table 2) and losses (Table 8, supplementary material) are reported on Taskonomy dataset. The performance evaluation on each task and the improvements would be interesting. Furthermore, only a subset of the available tasks is used.

* Ablation results reported in Table 3 appear to be incomplete. Only a subset of cases is reported.

* Minor: equations are not punctuated. Typo in L201.

questions: Most of my questions are answered in the supplementary material. I am wondering if the task similarly graph of Figure 9 (in supplementary material) for Taskonomy datasets looks similar to the Taskonomy datasets or not?

limitations: There is no obvious severe limitation of this paper besides what the authors have already mentioned in the paper.
I can say nothing regarding the negative social impact of this paper from my reading.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The submission discusses a way in which multi-task learning can potentially promote the use of features that might generalize poorly in OOD settings. This can happen when labels for different tasks (involving different features) are correlated in the training set but not outside of it.

A method is proposed to improve generalization in such situations by imposing (soft) modularity on the predictive format. Specifically, a (soft) sparse connectivity matrix is learned connecting sets of encodings (modules) to tasks, with training losses to encourage de-correlation among the modules, sparse module-usage, and discourage degenerate solutions such as dead modules. Finally, an invariance penalty inspired from IRMv1 [1] and V-REx [2] is applied on the connectivity matrix to promote the same weighting of modules across different environments, with the assumption that spurious label-label correlations across MTL tasks are likely to be unstable across different environments.

Experiments on a range of synthetic and realistic datasets (with synthetic splitting in all cases to simulate IID-OOD settings) showcase improvements over baselines.


[1] Invariant risk minimization, Arjovsky et al., 2019.

[2] OOD generalization via risk extrapolation, Krueger et al., 2020.
strengths_and_weaknesses: The discussion of the possibility of a multi-task learning setup inadvertently promoting the learning of less-stable features due to label correlations among the tasks is original, to my knowledge. The robustness-method is largely derived from existing works for the most part, but there are some elements of interest. For example, fixing f_t in the invariance penalty is an interesting trick (although something similar was done in [3]). The case study with MovieLens seems like a nice example of how such task-label correlations could arise.

I find the overall quality of the paper to be good. There are a number of relatively minor typos/writing issues I’ve detailed below, but I found the presentation reasonably clear on the whole.

Miscellaneous thoughts:

 — In L126, it might make more sense to rephrase “cannot generalize” as “perform relatively less effectively”, since presumably the correlation between the label and the causal factors is much stronger (for example, Table 1 only shows 2-3% drops and not utter failure). This also suggests a natural baseline can often simply be ERM training drawn out over more epochs/iterations, since with longer training, gradient descent can often be expected to unlearn any high-bias but non-stable features (the “noisy correlations” in L218-L220). In reality, for neural network training, other aspects to do with training dynamics crop up which might well lead to more drastic failure, but for the purposes of the theoretical arguments being made here, the outlook is less gloomy.

 — As acknowledged in the submission, invariance across a sampling of environments is not guaranteed to capture causality, and comes with several assumptions. Perhaps it is more appropriate to not imply that works such as these achieve causal discovery or even approximate it, but rather that they simply learn robust features, which are more likely to be stable in OOD settings. Also, since A is a soft-matrix (even with the sparsity penalty), it’s not clear to me if it makes sense to refer to it as a causal “graph”.

 — I’m not sure if label-label correlations are primarily the issue; the real problem seems to be the feature-feature correlations, if I’ve understood the setup correctly (and this boils down to the usual feature-label correlations). The two seem equivalent, but consider introducing label-noise: now the label-label correlation rate can drop, and depending on the strengths of the correlation-rate and the label-noise, the label-label correlation can drop to negligible rates while the feature-feature correlation rate stays where it was. I understand the point in the submission that the underlying reason behind the correlation is that the MTL setup promotes the correlation, but I was wondering if a more direct way to view this problem would be that the features corresponding to the two tasks are correlated, rather than the labels. Just a suggestion for the authors to consider.


[3] Systematic generalization with group-invariant predictions, Ahmed et al., 2021.

Typos:

 — Fig 2 caption should say F_b instead of F_a both times

 — Missing period at end of title for Section 2.1

 — L133: “exploit” —> “exploits”

 — L143: “The” —> “the”

 — L170: “extracts” —> “extract”

 — L177: “prediction” —> “predictions”

 — L183: “disentangle” —> “disentangled”?

 — L186: Missing period at end of sentence.

 — L186-L187: Cheung 2015 and Cogswell 2016 are described as involving “generative disentangled representation works”, but I believe both of these works deal with disentangling representations in classification settings, not generative.

 — L211: the per-task predictor was defined as f_t, not g.

 — L215: “exist” —> “exhibit”?

 (Few others, running spell-check should fix)

==================================
Post-rebuttal: I've read the rebuttal, and it answers my questions adequately on the whole. I'm raising my score to 7.
questions: 1. The most major question I have is: how is hyper-parameter selection being performed? The precise method is unclear to me from my reading, but my current understanding is that since training and validation splits correspond to two environments with the invariance penalty applied for these two sets, non-invariance hyper-parameters are tuned on the validation set. If this is the case, how is the invariance penalty hyper-parameter tuned? Is this done at a second stage, with non-invariance hyper-parameters fixed from the previous stage? A related question: for every baseline method, are all hyper-parameter tuning-choices made on the same OOD validation set as for G-IRM?


2. For the modular architecture, was the feature dimension scaled up by K, or divided into K parts from the same dimensionality as in baselines? This could imply different choices for baseline comparisons.

limitations: The authors have provided a limitation-statement.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The submission points out the risk of including non-causal features for one task when learning representations in the multi-task settings, accompanied by both a theoretical proof and an empirical analysis.
It then proposes a graph-disentanglement-based method, which is inspired by the disentangled causal mechanism and the invariant risk minimization, to address the problem.
strengths_and_weaknesses: Pros:

- It points out a novel yet important research problem -- the risk of including non-causal features for one task when learning representations in the multi-task settings, as shown in Figure 2.

- It also gives theoretical proof as well as an empirical experiment on synthetic data to show that the problem does exist. 

- The proposed causal-graph-based method outperforms previous multi-task methods such as PCGrad and GradVec as well as single-task causal methods such as IRM on multiple multitask datasets that contain unknown distribution shifts.

Cons:

- The disentanglement regularization shown in Eq. (1) assumes **linear correlation**, which will fail miserably with a deep feature extractor. It is highly possible that, with the regularization enforced, the multiple neural modules can still possess the same information even if they are linearly decorrelated (in other words, they can be **non-linearly correlated**).

- As pointed out in Line 225, ``if the complexity of a task-predictor is much larger than the number of environments, it could learn an over-fitted solution that makes gradient zero but does not achieve invariance.'' It seems like the proposed alternative (Eq 8 and Eq 9) can still face the same issue. Have I misunderstood anything?

- It contains a vast number of hyperparameters. It seems possible that the performance gain may come from hyperparameter tuning rather than from removing the spurious correlation. Is there any analysis of hyperparameter sensitivity? Does the proposed method underperform the baselines if the hyperparameters are not appropriately set?
questions: Please see the Weaknesses listed above.
limitations: The paper has provided a clear statement on its limitations after the conclusion.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work propose Multi-Task Causal Representation Learning framework to alleviate potential spurious correlation in Multi-task learning. Authors use three regularization methods to obtain the correct disentangled factors for experts, the sparsity for the Task-to-Module graph and the invariant causal representation respectively, which based on the Multi-gate Mixture-of-Experts architecture. Experiments on various Multi-task benchmarks and sufficient ablation studies have empirically proved the effectiveness of the method.
strengths_and_weaknesses: Originality: Authors study the problem of spurious correlation existing in multitask learning which has been noticed but rarely addressed directly to the best of my knowledge. Authors combine many methods to implement their claim. While neither their method is novel, their combination is novel and purposeful. And they clearly identify related work and place it in context in their paper.

Clarity: The paper is very well written; clear, precise, and easy to understand. I was able to follow the main contributions without issues. The paper contains many various and multi-level experiments which well prove the effectiveness of their method. 

Significance: Alleviating spurious correlation is an interesting direction to multitask learning and may be the key to solving negative transfer. Experimental results show that this method has a general improvement over all datasets, but not significant improvement in natural CV datasets. 

Authors discuss the hyperparameters tuning in the appendix. Among all the hyperparameters, I care about the number of experts $K$ that determines how many factors the feature extractor learned, on the one hand, determines the difficulty of training. Authors just use $K = 8$ by default and don’t tune it. I think it is beneficial and important to further discuss whether imposing disentangle regularization with fixed $K$ will reduce the capacity of the model, and whether $K$ should be adjusted according to the complexity of the dataset. Adding the empirical risks of methods on training datasets maybe helpful for comparing the ability of the fitting ability. And some hyperparameters $\lambda_{decor},\lambda_{sps},\lambda_{bal}$ are not discussed, did I miss anything?

And I have an issue on $\mathcal{L}^{Var}_{G-IRM}$. The intuition behind this penalty is in conflict with the fact that distinctive and complex task-predictors are necessary for multitask learning. I'm not an expert in this area so please let me know if I've misunderstood something.
questions: How to determine and select the hyperparameters is reasonable when introducing so many hyperparameters in the method?
Could authors give a more explicit discussion about the conflict between distinctive and complex task-predictors and the invariant?
limitations: Not relying on the MMOE architecture may have more impact.
review_title: null

Meta-review: metareview: This work studies how to alleviate potential spurious correlation in multi-task learning, and proposes a Multi-Task Causal Representation Learning (MT-CRL) framework, which aims to represent multi-task knowledge via disentangled neural modules, and learns which module is causally related to each task via MTL-specific invariant regularization

The proposed method sounds reasonable, the results are solid, and the paper is well written. Overall it is a good work. 

confidence: Certain
Decision: Accept