Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose a method for improving model robustness to target shift (i.e., shift in prior probability of labels) in multitask learning (MTL). Specifically, the authors consider a different hypothetical data generating process (DGP) than common MTL formulations: the authors assume that the observed data is generated causally from the targets, and that the targets are subject to unobserved confounding. This differs from the common assumption that tasks $Y$ and $Y'$ are conditionally independent given the data $X$. The authors use this DGP to show that modeling the the likelihood (i.e., $p(x|y,y')$) is invariant to shifts in the unobserved confounder, and thus is invariant to target shifts. Using Bayes rule and relaxing to factorize by task, the proposed method (GMTL) amounts to a modification of "standard" discriminative MTL to account for possible changes to the label distribution. Using synthetic and reweighted (to simulate target shifts) data, the authors show that with appropriate tuning the proposed method yields more robustness to target shifts that standard MTL which ignores the possibility of target shifts.
strengths_and_weaknesses: The idea in this paper is interesting: the authors propose to use knowledge about the data generating process (DGP) to improve the robustness of MTL under target shift. The premise of the approach (that a model of the data likelihood will be invariant to the shift) is sound and sensible, and the implementation is convenient (builds upon discriminate formulations to MTL). However, I think the framing of the target-causing confounding problem misses some context, and I have some questions/concerns about some choices/relaxations the authors make in implementing the proposed approach. In light of this, I think the paper is missing some more in-depth analysis of the performance expectations of the proposed approach.

**Re: Framing**

I have a few comments about the way the authors frame their method, as the framing seems to be presented as one of the primary contributions of the paper. In the introduction, the authors contrast the proposed approach (GMTL) with existing discriminative MTL (DMTL) approaches by saying that "DMTL is flawed under an alternative set of assumptions." I just want to point out that in Fig 1, from a statistical perspective, the graphs in Fig1a and Fig1b (DMTL under the different assumption sets) are incompatible. This is because the DMTL assumption that $Y \perp Y' | X$ holds in Fig1a but does not hold in Fig1b. Importantly, this is a *testable* assumption---because $X$, $Y$, and $Y'$ are observed in the training data, we can perform conditional independence tests to measure the compatibility of this assumption with the available data. Thus, in theory, it seems a user should be able to determine if the DMTL assumption is reasonable for their problem.

Second, while to my knowledge the GMTL formulation is novel for multitask learning, I would like to note that in supervised learning the DAG in Fig 1c has been considered in [1] (see Fig 1 in [1]). In one experiment they consider the target shift scenario (Section 5.1) and their approach ends up modeling (the equivalent of) $p(x|y,y')$. Given the similarities in the resulting objectives, I think some discussion of this work and its related threads (i.e., causal graphical approaches to stability/invariance to distribution shifts) is likely warranted.

I also think this connection to causal graphical approaches opens up additional methodological possibilities for the authors' line of work. For example, in Section 3, the authors note that the density estimation problem for $p(x | y, y')$ will, in general, be quite difficult and instead use equation (5) as their target of inference (and further relax it in eq (6)). However, by noting possible connections to the causal approaches, there may be alternative approaches to fitting the model such as inverse propensity weights (IPW). The authors would need to double check me, but I think reweighting the data by (something like) $\frac{1}{p(y, y')}$ during the training of a discriminative model would be an alternative approach for achieving the desired invariance (see, e.g., the weighting scheme in [2]).

[1] Subbaswamy, A., Schulam, P., & Saria, S. Preventing failures due to dataset shift: Learning predictive models that transport. AISTATS, 2019.

[2] Makar, M., Packer, B., Moldovan, D., Blalock, D., Halpern, Y., & D’Amour, A. Causally motivated shortcut removal using auxiliary labels. AISTATS, 2022.

**Re: relaxing the GMTL objective**

In Section 3.1, the authors "allow $p(y, y'|x)$ to factorize... purely out of convenience." While I understand why the authors want to do this (it reduces the multitask network part of GMTL to DMTL), it is clearly the case that $Y \not \perp Y' | X$ in Fig 1c (even if there were no unobserved confounder $U$ !). The authors assert that "we can allow this factorization as long as we capture the dependency between the targets when estimating $p(y, y')$", but this is not obvious to me. I would have expected to see either a theoretical justification or some experiments showing the effect of this relaxation by comparing the approach in eq (6) to one which does not assume that $p(y, y'|x)$ factorizes.

(As a side note, do cross-stitch networks (CSN) count as modeling $p(y, y'|x)$ jointly? If so, it might be possible to use CSN to perform the above comparison, but an approach that more explicitly jointly models $p(y, y'|x)$ might be better.)

**Re: interpretation of $\alpha$**

In equation (6), the authors also introduce a new hyperparameter $\alpha$ which interpolates between DMTL and GMTL. The motivation is to allow for solutions which generalize better under small or moderate shifts to the joint label distribution. However, it wasn't clear to me why this would be the correct or optimal approach for handling an uncertainty set of test distributions under target shift. In the simulated shifts considered in Section 5.4, it seems the authors allow for essentially arbitrary shifts to $p(y,y')$. This will include shifted label distributions that cannot be modeled in the form $p(y, y')^{1-\alpha}$, since the $\alpha$-based transformation will never change the mode (or relative order) of the probability mass function (except when $\alpha = 1$ in which case the label distribution becomes uniform).

The authors select an "optimal" $\alpha$ by finding the value that performs best on the shifted data, but why is the $\alpha$ based interpolation the correct way to implement GMTL given that the uncertainty set is more general? Can a given $\alpha$ value be framed as the solution to, e.g., a distributionally robust optimization problem for a certain uncertainty set? What can we say about the worst-case, average, or some other performance measure across the uncertainty set for GMTL? For $\alpha=1$, I see from eq (5) that predictions should be invariant to target shifts. What, though, can be said about model accuracy/performance? For $0 < \alpha < 1$ what can be said about performance?


**Minor notes**
* This is a very minor note, but isn't it the case that when interpreting $\alpha$ the test target distribution is *proportional* to $p(y,y')^{1-\alpha}$? If $p$ is a valid pmf, then $q=p^{1-\alpha}$ generally won't sum to 1 and needs to be renormalized.
questions: Could the authors please address my questions about:
* What is the effect of allowing $p(y, y'|x)$ to factorize (when it does not under GMTL assumptions)? (See discussion above)
* Why is the $\alpha$ based interpolation the correct way to implement GMTL given that the uncertainty set is more general? What can be said about the model's performance across the uncertainty set? (See discussion above)
limitations: Yes, the authors adequately address the limitations and potential negative impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a generative multitask learning (GMTL) method to handle the existence of target-causing confounding from a generative model perspective. It tries to find the invariant information under different interventions (or in target-shift case) with the help of an auxiliary target. Experimental results show the robustness of the target shift.
strengths_and_weaknesses: Strengths:
1. This paper introduces the idea of anticausal learning for multitask learning. With the causal factorization of the joint distribution, the invariant information is used for tackling the target shift problem.
2. The accuracy of the experimental results demonstrates the effectiveness of the proposed methods. 

Weaknesses:
1. Why p(y, y’|x) in Eq. (5) can be transferred into the product of p(y|x) and p(y’|x) (in Eq. (6)). According to Figure 1(c), in GMTL, y and y’ are not independent conditional on x, but are independent with each other conditional on latent variable U.
2. Is there any effective method to determine the $\alpha$ when there is no prior information?
questions: See my comments above.
limitations: Not available.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers a more relaxed assumption in multitask learning, compared to previous works: targets are not conditionally independent given the input, but rather caused by a hidden confounding factor. Therefore, the authors propose generative multitask learning (GMTL), where the confounding factor is addressed in loss function. The GMTL loss can be linearly interpolated together with traditional multitask (denoted as DMTL) loss, with a tunable interpolation factor. Experiments are run by augmenting different multitask learning methods with the new loss on different datasets, and result shows change in accuracy is positively correlated to the weight of interpolation factor on GMTL loss.

strengths_and_weaknesses: Strengths: 
(1) This seems to be the first paper addressing confounding factors that affect multitask targets. 
(2) The GMTL objective is formulated with clear logic. 
(3) Experiment results support the claims.
Weaknesses: 
(1) It is unclear why the experiments satisfy the assumption, where some confounding factor u cause y and y’. 
(2) The authors mention that they use two targets only “without loss of generality”, but it is unclear how to extend this method to multiple targets - do we assume a common confounding factor for all targets, or one confounding factor per pair of targets, or something else?

questions: (1) Under what conditions can we assume the existence of such a confounding factor, and why the experiments satisfy these conditions? 
(2) How to extend this two-target system to multiple targets?
limitations: Limitations of this method are not addressed in the paper. One limitation can be it is hard to infer the linear interpolation factor in general tasks.
review_title: null

Meta-review: metareview: The decision is to accept the paper.

The paper presents an interesting perspective on confounding in the marginal distribution of task targets in multi-task learning settings, under an anti-causal prediction and no-unobserved-confounders assumption. The paper suggests an inference time strategy for eliminating some of the influence of the marginal correlation between task labels by subtracting off their joint log-probability from an inference-time objective. For convenience (and usability with current MTL training pipelines), the strategy employs an additional factorization in the posterior over tasks. The authors demonstrate effectiveness on several datasets.

The reviewers agreed that this was a fresh perspective on this problem, but also noted some limitations that could deserve more discussion in the paper. In particular, it would be useful for the authors to give an example (even a toy example) where the factorization really does hurt compared to an oracle that had access to the joint p(y, y' | x), which would motivate more fundamental research on the MTL side, and give end-users a sense of whether the method was appropriate for their problem.

Less pressingly, additional commentary on the particular modeling assumption about downstream domains that is being made with the alpha parameterization of the objective would also be useful; here, too, it would be useful to have a toy example where this simplification might also yield sub-optimal results, or at least a note that there may be other interpolation schemes between DMTL and GMTL that would work better. Basically, framing this as _a_ sensitive solution rather than _the_ solution would be helpful to point other researchers toward directions for future work.

Despite some of these reviewer concerns, I think there is enough here that the community would find this paper stimulating and useful.
confidence: Certain
Decision: Accept