Generative multitask learning mitigates target-causing confounding
Authors: Taro Makino
Krzysztof J. Geras
Kyunghyun Cho
Venue: NeurIPS 2022 Accept (2022)
Keywords: causal representation learning
causal machine learning
multitask learning
causality
robustness
out-of-distribution generalization

Abstract: We propose generative multitask learning (GMTL), a simple and scalable approach to causal machine learning in the multitask setting. Our approach makes a minor change to the conventional multitask inference objective, and improves robustness to target shift. Since GMTL only modifies the inference objective, it can be used with existing multitask learning methods without requiring additional training. The improvement in robustness comes from mitigating unobserved confounders that cause the targets, but not the input. We refer to them as \emph{target-causing confounders}. These confounders induce spurious dependencies between the input and targets. This poses a problem for conventional multitask learning, due to its assumption that the targets are conditionally independent given the input. GMTL mitigates target-causing confounding at inference time, by removing the influence of the joint target distribution, and predicting all targets jointly. This removes the spurious dependencies between the input and targets, where the degree of removal is adjustable via a single hyperparameter. This flexibility is useful for managing the trade-off between in- and out-of-distribution generalization. Our results on the Attributes of People and Taskonomy datasets reflect an improved robustness to target shift across four multitask learning methods.

Generative multitask learning mitigates targef-causing confounding
Taro Makino'
Krzysztof J. Geras
Kyunghyun Cho 133,4
NYU Center for Data Science
NYU Grossman School of Medicine
3 Genentech ACIFAR LMB taro@nyu.edi
Abstract
We propose generative multitask learning (GMTL), simple and scalable approach to causal machine learning in the multitask setting. Our approach makes minor change to the conventional multitask inference objective, and improves robustness to target shift. Since GMTL only modifies the inference objective, it can used with existing multitask learning methods without requiring additional training. The improvement in robustness comes from mitigating unobserved confounders that cause the targets, but not the input. We refer them as larget-causing confounders. These confounders induce spurious dependencies between the input and targets. This poses problem for conventional multitask learning, due its assumption that the targets are conditionally independent given the input. GMTL mitigates target-causing confounding inference time, by removing the influence of the joint target distribution, and predicting all targets jointly. This removes the spurious dependencies between the input and targets, where the degree of removal is adjustable via single hyperparameter. This flexibility is useful for managing the trade-off between in- and out-of-distribution generalization. Our results on the Attributes of People and Taskonomy datasets reflect improved robustness to target shift across four multitask learning methods.
Introduction
Deep neural networks (DNNs) excel at extracting patterns from unstructured data, but these patterns can fail to generalize outside of the training distribution. These failures are often attributed to DNNs learning statistical associations rather than causal relations (Ribeiro al. 2016 Jo and Bengio 2017 Geirhos et al. 2020]. Therefore, there is a concerted effort to make patterns learned by DNNs satisfy certain properties of causal relations, such as invariance and modularity (Schölkopi 2019 Schölkopf et al., 2021 Wang and Jordan 2021]. This research direction is called causal machine learning (ML). Existing approaches to causal ML often require additional information in the form of labeled
environments (Arjovsky et al. 2019 Lu et al. 2021] or labeled confounders (Puli et al. 2021] These requirements are restrictive, and it is therefore beneficial to identify more natural settings where existing information can be used for causal MI..
We consider the setting of multitask learning (MTL) [Caruanal 1997 Ruder 2017 Zhang and Yang 2017]. where there is an input x E X' and multiple targets. Without loss of generality, we consider two targets y y' and y' E )" As is common in the MTL literature, main target, and y' as the auxiliary target. Both targets are used during training, but only the main target used during evaluation. In the conventional approach to MTL, which we call discriminative
distinguish y as being the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Y
Y
X
X
(a) DMTL
(b) DMTL (altemative assumptions)
(c) GMTL
Figure 1: (a) The conventional approach to MTL, which we call discriminative multitask learning (DMTL), assumes the targets y and y' are conditionally independent given the input x. Due this assumption, the inference objective for predicting the main target y is argmax, loy x). (b) DMTL is flawed under an alternative set of assumptions, where the targets cause the input, and there exists an unobserved confounder u that causes the targets, but not the input. We call u larget-causing confounder. Since y' is unobserved when predicting y with DMTL, this unblocks the backdoor path y + + y' - X and makes the input and targets spuriously dependent. That is, p(y x) shifts when p(u) shifts. (c) We propose generative multitask learning (GMTL., where the inference objective is argmaxy, y.y' log p(x y, y'). Unlike DMTI,, this objective conditions all targets, which d-separates x and u, and removes the spurious dependencies between the input and targets.
multitask learning (DMTL), the inference objective is
argiaxx ply, y') x) = argmax logp(y y.y'
+ log ply' x).
y.y'
This factorization corresponds to assuming the targets are conditionally independent given the input. This is convenient assumption to make, since it prevents Vx y from growing exponentially with the number of tasks. Since only the main target y is used during evaluation, the inference objective
argmaxlog p(y x).
(2)
y
DMTL does not make causal assumptions explicitly, but its conditional independence assumption is consistent with the causal graph in Fig. la We argue that DMTL is flawed under an alternative set of assumptions.
First, we assume the targets cause the input, which implies we are predicting the cause from the effect. This is called anticausal learning, and considered to describe many common problems such as image classification [Schölkopf et al. 2012). For example, in object recognition, the object category causes the pixel representation of that object. Second, we assume there exists variable u that causes the targets, but not the input. We call target-causing confounder.
The problem with DMTL under these alternative assumptions is that the auxiliary target y' is unobserved, as illustrated in Fig. [b] This is problematic because it unblocks the backdoor path y + > y' - x, and makes x and y spuriously dependent. Since the backdoor path passes through u, DMTL is sensitive to shifts in p(u). When p(u) shifts, the relationship between x and y changes, and DMTL fails to generalize. This is the weakness DMTL that aim improve. Since u causes the targets, shift in p(u) leads to shift in p(y,y' y'), which is often called target shift [Zhang et al. 2013). The targets are observable, unlike u1 so we use target shift as an observable proxy for shifts in p(u).
We propose generative multitask learning (GMTL) as a way to mitigate target-causing confounding. GMTL is based on the following idea. If we write the causal factorization of the joint distribution of the observed variables as
p(x.y.y') = p(x v.y'))) v.y')'yy,') = p(x lxx))) / ply u)ply' uu)(u)du,
this shows that p(x y, y') is invariant shifts in p(u). Therefore, if
argmax log p(x y,y')
(3)
y.y'

---

as the inference objective, we become invariant to shifts in p(u). Importantly, unlike Eq. 1 the inference objective in no longer factorizes over the targets. This means that even if we only care about evaluating the main target we also need to consider y'. Conditioning on all targets during inference closes the backdoor paths between the inputs and targets. This d-separates x and u, and removes the spurious dependencies between the input and targets.
In practice, we implement GMTL as an approximation of Eq. 3 As we discuss in Section 3 this allows us to formulate GMTL as a minor change to the inference objective of DMTL. This makes GMTL practical, since it can be used with existing MTL methods without requiring additional training. We also introduce a single hyperparameter to adjust the degree to which we remove spurious dependencies between the input and targets. This makes GMTL include DMTL as a special case, where no spurious dependencies are removed. This flexibility is useful for managing the trade-off between in-distribution (ID) and out-of-distribution (OOD) generalization, since spurious dependencies are predictive in the former setting, but not the latter.
In order to empirically validate GMTL, we perform experiments on two datasets called Attributes of People [Bourdev et al. 2011] and Taskonomy (Zamir et al. 2018). Our results show that GMTL improves robustness to target shift across four MTL methods. We attribute this improvement to GMTL's ability to mitigate target-causing confounding. This serves as evidence that causal ML shows promise for MTL.
2 Background
2.1 Out-of-distribution generalization
OOD generalization refers to the capability of ML. systems generalize on data outside of the training distribution. [engio et al.] [202T] describe it as one of the greatest challenges in ML, and one that cannot be solved solely by scaling up datasets and computation. The failure of ML systems to generalize OOD is general problem that has been reported across a wide range of problems (D' Amour al. 2020). The core issue is that it often easier for ML systems learn patterns that are dataset-specific, rather than those that hold universally. and there is often no incentive to learn the latter. These -ptaset-specffic patterns have many names, such as surface statistical regularities nJo and Bengio 2017). nonrobust features [Tippaa et al., 2019), and shortcuts [Geirhos) al. 2020). We take a causal perspective and adopt the term spurious dependencies, which are defined as statistical dependencies not supported by causal links [Paal] 2009
2.2 Dataset biases induce spurious dependencies
Spurious dependencies can arise from design choices in dataset creation. Recht et al.) [2019] demon- strated that state-of-the-ar image classifiers trained on ImageNet (Deng et al. 2009] failed to generalize replicated test set designed mirror the original test set distribution. |nnsstrmm et al. [2020] offered an explanation for this phenomenon. The authors reported that the dataset replication procedure relied on estimating human-in -irtan----thht-ooo metric called selection frequency. and that bias in estimating this statistic led to undesirable variation across datasets. Selection frequency is defined for an input-target pair, and measures the proportion of crowdsourced annotators who deem the pair correctly labeled. |Recht et al. [2019) performed statistic matching on this metric, which corresponds to conditioning on it. Since selection frequency is caused by the input and target, this corresponds to conditioning on collider, which induces a spurious dependency between the input and target. Such dependency would fail to generalize OOD, which may cxplain why predictive performance degraded on the replicated test set.
In this work, we address a different source of dataset bias that we call target-causing confounding. A variable is target-causing confounder if it is causes the targets, but not the input. For a simple example, suppose we have images of people in indoor scenes, and the targets are clothing- rellted attributes such as hat and scarf. The season causes the targets, since people pair various types of clothing depending on the season. The season does not cause the image, since the images are of indoor scenes. Therefore, the season is target-causing confounder that induces spurious dependencies between the input and targets. Relying on such dependencies for prediction can be problematic if the training and test sets represent different seasons. Although this is simple and contrived example
3

---

for the purpose of illustration, dataset biases can occur in real-life settings like medicine, and have serious societal impacts (Mehrabi et al. 2021
2.3 Causal machine learning
In Section 2.1 we made the distinction between patterns that are dataset-specific, and those that hold universally. The field of causality formalizes the notion patterns that hold universally, calling them causal relations [Pearl 2009]. Causal ML aims to use ideas from causality to improve machine
learning methods, particularly in the area of OOD generalization Zhang et al. [2013] Arjovsky et al.,
2019 Schölkopf 2019, Wang and Blei 2019 Puli et al. [022] Schölkopf et al. a [202]] Our work falls under this category, since GMTL improves robustness to target shift by using an intervention distribution for prediction. This idea is also shared by (Subbaswamy et al. [2019), who propose
a general framework for estimating intervention distributions for robust prediction. GMTL is a
practical method for achieving this high-dimensional datasets. Relatedly, [Makar et al. [0022] use
importance reweighting to learn distributions that are invariant to intervention on an auxiliary target.
Both Subbaswamy al. (2019) and Makar al. 2022 assume the same causal graph GMTL.
2.4 Multitask learning
MTL can reduce computational cost through parameter sharing, and improve predictive performance
over training on each task individually (Standley et al. [020)] Its use is widespread, spanning domains
such as scene understanding [Zamir et al. [018 Chowdhuri et al. 2019 Zamir et al. [020]. . medical
diagnosis [Kyono al. [202]], natural language processing McCann et al. [018 Liu al. 2019
Radford et al. 2019 Wang et al. 2019 Worsham and Kalita 2020, Aghajanyan et al.) 2021] recom-
mender systems Covington et al. [0166] Ma et al. a [008]] and reintorcement learning Kalashnikov
et al. 2021 Distinguishing between main and auxi auxiliary targets is common [Ruder] [017 Libbel and
Körner 2018 Vafaeikia et al.j 2020]. but not universal, and is primarily important for concerns such
as which loss to use for early stopping [Caruana 1997]. We make this distinction throughout this paper, denoting the main target as y, and the auxiliary target as y'.
3 Generative multitask learning
In GMTL, we condition on all targets in order to d-separate x and u, and remove spurious depen- dencies between the input and targets. During inference, we rely solely on p(x y,y"), since is invariant to shifts in p(u). Approaching this naively by estimating p(x y, y')') directly can be difficult, especially when x is high-dimensional. We can get around this by using Bayes" rule to write
argmax p(x yyy') argmax log ply.y' x)p(x) y.y'
ply,y') y")
y.y' argmaxlogp(y.. x) logp(y.y').
y.y
This shows that the argmax over the targets results in the difficult-ooessimmaee p(x) term to drop out, meaning we only need estimate p(y.y' x) and y'y.y').
In order to make GMTL practical, we replace Eq. with an approximation, i.c.
argmax log p(y x) + log ply'((y' x) a logply,y'), y.y'
a E [0.
(5)
This introduces two changes. The first that we allow ply. y' DMTL. This allows us to use conventional approaches to estimate ply, x), which is beneficial since training is notoriously difficult in MTL (Standley et al.) 2020]. To be clear, do not allow this factorization because we assume the targets are conditionally independent given the input, as this would violate our causal graph in Fig. Tc We do so purely out of convenience, so that GMTL can leverage existing approaches for estimating p(y. ' x). We can allow this factorization as long as we capture the dependency between the targets when estimating p(y,y'). This prevents Eq. from factorizing over the targets, and satisfies our requirement of conditioning on all targets in order to remove spurious dependencies between the input and targets. We discuss some complications with estimating p(y, y') in Section 3.2) but in general, it can be done trivially for categorical targets.
x) to factorize, as was the case in

---

Therefore, GMTL only changes the inference objective of DMTL, and can be applied with existing multitask methods without requiring additional training.
The second change is the introduction of the hyperparameter a E [0, 1), which enables us to interpolate between DMTL (a 0) and the most extreme case of GMTL (a = 1). a controls the degree to which we remove spurious dependencies between the input and targets. The ability to adjust this effect important because if the training and test distributions are identical, then we should be willing to use any dependencies available, regardless of whether they are spurious or causal. However, it is more realistic to assume training and test distributions will be different, and under these circumstances, setting a >> O can improve robustness to target shift.
3.1 Interpreting a
The hyperparameter a represents uncertainty about the test target distribution. In the case of complete certainty, i.c. if we know that the test target distribution is identical to the empirical target distribution, we set uncertainty, we set a = 1 to completely remove the influence of the empirical distribution. This is equivalent to assuming the test target distribution is uniform. Generally, it can be shown that GMTL corresponds to replacing the empirical target distribution p(y. y') with a distribution proportional to ply,y')' The derivation is in the supplementary material. We perform this replacement during inference in order to keep training the same as DMTL. In principle, we could achieve the same invariance by reweighting examples by 1 1/p(y,y)) during training [Zhang et al. 2013 Makar et al. 2022].
0 to use the empirical distribution. In contrast, in the case of complete
This raises important point- represents assumption, and is not a hyperparameter that can be tuned in the conventional sense. Instead, tuning a is analogous to model selection w.r.t. an unknown distribution, which is a difficult open problem [Gulrajani and Lopez-Paz 2021]. Progress in this direction is likely to benefit GMTL, as well as other methods with similar hyperparameters [Wortsman et al. 2022]. In the remainder of this work, OOD test distribution. That is, our main focus is not on the difficulty of tuning a, which is separate research question. Instead, we study the efficacy of GMTL assuming have correctly assessed how similar the training and test target distributions are, i.e. knowing the optimal o.
assume oracle access to the optimal for given
Having said that, we provide a simple and effective heuristic for setting a even for an unknown test distribution, which makes GMTL practical. This is based on the fact that a represents a trade-off. Increasing improves target shift robustness at the expense of ID predictive performance. We show empirically in Section 6 that this relationship is very strong, and holds across two datasets, multiple pairs of tasks, and four MTL methods. Therefore, our heuristic is to set the maximum value within an acceptable loss in ID predictive performance, where the latter is measured using the validation set. This ensures that are robust as possible to target shift, while keeping ID predictive performance at an acceptable level. These results are in the supplementary material.
If we do have access to the test target distribution q(y.y), y'), then we can do away with and simply replace ply.y') with ('y,y)), which results in the following inference objective:
argmax log ply x) + log ply x) Mly,(y, y') log q(y.y').
y.y'
Alternatively, we can use g(y, y //((y,/())))))))) to correct for target shift during training (Zhang et al. 2013].
3.2 Estimating p(y.y')
GMTL requires estimating p(y.y') y'), which is trivial for categorical targets when yxy" is small. The maximum likelihood estimate of p(y,y') (') is obtained by counting the number of occurrences of the pair (y,y', and dividing by the total across all pairs. However, there is an important precaution to be made, since it is possible that certain pairs of tyyy') are never observed. This is problematic, since ply,y' y') results in taking a logarithm of zero Eq. 5 This can be addressed with additive smoothing, where a pseudocount > O is added to the count for each pair prior to normalization. The issue of sparsity in the target distribution is the primary challenge in scaling GMTL to a larger number of tasks. Another complication is that in MTL, it is not unusual for a subset of targets to be missing for a given example. When this the case, the counting approach only works with the subset of examples for which all targets are present, which can lead to poor estimates.
5

---

Toy problem
Before empirically validating GMTL on real datasets, we use synthetic data demonstrate how DMTL and GMTL behave differently in the presence of target-causing confounding, We construct data generating process (DGP) where a target-causing confounder induces spurious dependency between scalar input X and binary target Y.
Suppose there are two targets Y. (0.1) that are bivariate Bernoulli. This distribution can be = 0, P(Y" = 1) = 0', and Cov(Y, YJI We keep and 0' fixed, and vary the value of Cov(Y, Y1 in order to induce target shift. Cov(Y, Y"] a target-causing confounder in this context. X is generated by a mixture of Gaussians with latent variable = 2Y + Y', i.e. X ~ (z, 2).. We fix 0 0' 0.5. oj of 0.4, and o2 og = 0.6, and vary Cov[Y, Y"] from -022 to 0.2. In other words, the only property of the DGP that we vary is Cov[X,Y'].
specified in terms of P(Y
= P(Y=1)
Fig. shows how the decision boundaries of DMTL and GMTL with 1 are affected by variation in the target-causing confounder Cov|Y, Y The decision boundary of DMTL is the value of X for which the solution to argmax, log Ply æ) changes, where Y' has been marginalized out. In contrast, the decision boundary of GMTL is the value of X where the component of the solution argmax,,.' y log P(x: y, y.y') changes. The decision boundary for DMTL changes in response Cov Y" while it remains constant for GMTL. This is because GMTL only the Gaussian densities for prediction, which have no connection to Cov(Y, Y"J.
In this example, we have shown that GMTL and DMTL arrive at different solutions by varying only the dependency between and Y' induced by the target-causing confounder Cov Y'J. The key difference between the two methods is that DMTL uses the spurious dependency between X and Y, while GMTL does not. Which of the two methods achieves better predictive performance depends on the particular test distribution, since the spurious dependency will be predictive in ID settings, and not in OOD settings.
Decision boundary
1.48
1.46
1.44
DMTL GMTL
1.42
-02
-0.1
0.0
0.1
0.2
CovlY,
Figure 2: The decision boundary of DMTL changes w.r.t. Cov Y"), while GMTL invariant.
5 Experimental setup
We perform experiments to demonstrate that GMTL improves robustness target shift. We evaluate across wide range of target shifts ranging from mild to severe, and analyze how the optimal changes w.r.t. the severity of target shift. Recall from Section 3.1 that we assume oracle access to the optimal since are focusing selection w.r.t. unknown distributions, which is a separate open problem. The purpose of these experiments is to show that if we correctly assess how similar the training and test distributions are, we can benefit from using GMTL. order to draw robust conclusions, we perform our experiments using two datasets, multiple pairs of classification tasks, and four MTL methods. Here, we describe our experimental setup.
validating GMTL, not on whether
can perform model
5.1 Datasets
Attributes of People The Attributes of People dataset [Bourdev et al., 2011] consists of 8,035 images of people. There are 4,013 examples in training set, and 4,022 the test set. Since the authors did not specify validation set, we randomly sample 20% of the training set to use as the validation set. Fach image comes labeled with up nine binary attributes: male, long hair, glasses, hat, t-shirt, long sleeves, shorts, jeans, and long pants. We each attribute binary classification

---

task, and predict whether the attribute appears in the image. We drop the gender attribute, since it has the potential for negative downstream applications [Wang and Kosinski 2018]. We experiment with various pairs of tasks, taking turns specifying one as the main task, and the other as the auxiliary task. For each pair of tasks, we report the classification accuracy for the main task. Since not all attributes are labeled for each example, we consider the three pairs of tasks with the highest number of co-occurrences. These are: hat and long sleeves, long hair and hat, and glasses and hat.
Taskonomy The Taskonomy dataset (Zamir et al. 2018 is of a much larger scale than the first dataset, containing approximately four million images of indoor scenes. We use subset of the dataset that is provided by the authors for faster experimentation, with 548,785 examples in the training set, 121,974 in the validation set, and 90, 658 in the test set. Each image comes paired with 26 labels relevant to scene understanding, but since we are focusing on classification, we only use the object and scene annotations targets. We use the object annotations as a 100-class object classification task, and the scene annotations as a 64-class scene classification task. Between object and scene classification, we take turns specifying one as the main task, and the other as the auxiliary task. We report the top-1 accuracy for the main task.
5.2 Models
No Parameter Sharing (NPS) The first MTL method we use trivial combination of two single-task networks. That is, ply x) and p(y' x) are trained separately. We call this No Parameter Sharing (NPS). This is equivalent single-task learning during training. but becomes different during inference for > 0, since the single-task networks influence one another through the -a log ply, y') term. While NPS is primarily meant to be simple setting for comparing GMTL and DMTI., offers some practical utility well. MTL. methods are notoriously difficult to train, and it is often challenging to get them to perform better than the single-task baseline [Alonso and Plank] [017]. NPS combined with GMTL offers way to combine two single-task networks to reap the benefits of MTL, by estimating ply.'' y') and predicting jointly over the targets.
Shared Trunk Networks (STN) The second MTL method is a hard parameter sharing method that call Shared Trunk Networks (STN), where all weights except for the final classification layer shared. This is not an effective architecture for many of the tasks, since the predictive performance is worse than the single-task baseline. This is common for STN AAonso and Plank 2017), but include it for completeness because it is the canonical MTL architecture. The results comparing single-task and MTL methods the ID test set in the supplementary material.
Cross-stitch Networks (CSN) For the third MTL method, we use soft parameter sharing method called Cross-stitch Networks (CSN) Missa et al. 2016). CSN takes two trained single-task networks, and takes a linear combination of the activations each layer during the forward pass. Training CSN involves learning the coefficients of the linear combinations, as well as fine-tuning the single-task network weights. This network performs better than the single-task bascline across most datasets and tasks.
Full Parameter Sharing (FPS) For our experiments on Attributes of People, we model p(y, y' x) directly as a four-class classification, without assuming factorization. Since this a single network with all weights shared across both tasks, we call this Full Parameter Sharing (FPS). We include FPS to assess the appropriateness of letting p(y. y' x) factorize in the other MTL approaches, as we discussed in Section 3 We omit FPS Taskonomy due to the complications of y >" being large, and leave this to future work.
For all datasets and tasks, we use ResNet-50 He et al. 2016 pretrained on ImageNet Deng et al. 2009] for the single-task networks. We train using Adam (Kingma and Ba, 2015) with L2 regulariza- tion, and tune the learning rate and regularization multiplier. For data augmentation, during training we resize the images to 256 256, randomly crop them to 224x 224 and randomly horizontally flip them. During validation and testing. we resize the images to 256 x 226,, and center crop them to 224 2244. For all experiments, we train single-task networks from five random initializations, Details regarding other hyperparameters are included in the supplementary material.
7

---

5.3 Simulating target shift with importance sampling
In our experiments, we aim to evaluate across wide range of target shifts. Since it is impractical to collect many different test sets, we simulate target shifts using importance sampling. Suppose the original test distribution is p(x, y, y") = p(x y, y.y')(yy, y'), and the simulated test distribution is y(x)""") p(x y. y.y')((x y )-) p(x y. y') is invariant due to our assumption that it is causal relation. Therefore, the expected accuracy y')) under the simulated test distribution ((x, y, y') is given by
( = (o y. (xx, y, yn)
q(v,y')
- p(y,y")
g(y(m), ,
((((m) -.ار.. (yii (ugii
p(y!n), y)(c)))
g(y(m), yr(mn)
g(ylm)) y
1(x(m), y(()) yhm)).
p(y(n), y((nl)
p(y(m), yrimn)
m=1
Now that we can evaluate across a wide range of OOD y')s, we need a way to quantify the severity of the target shift between p(y,y' and y"y.
5.4 Measuring the severity of target shift between p(y, y") and g(y,y')
Our metric to measure the severity of target shift between ply.y') and y must be such that predictive performance degrades w.r.t. the severity of shift. Not all metrics satisfy this criteria. To build intuition, consider a Bernoulli distribution where P(Y = 1) = 0. If we use a norm-based metric such as total variation, this treats a change in from 0.6 to 0.8 and a change from 0.6 to 0.4 as being the same. However, from the point of view of prediction, the latter should be more detrimental, since it reverses the ranking of the classes.
Similarly, Kullback-Leibler (KL) divergence is a common choice for measuring the distance between distributions, but it suffers from related problem. Consider 0 from above changing from 0.9 to 0.9999, and from 0.9 to 0.4. Due to taking the logarithm of a small number, KI. divergence considers the first to be a more severe shift, This undesirable, because the change from 0.9 0.4 represents a reversal in the ranking of the classes.
These examples point to need for ranking- based metric, since a significant change to the ranking of classes is detrimental to predictive performance. Weighted rank correlation satisfies this. If the weighted rank correlation between y') and q(y. y') is positive with large magnitude, it means that there are no significant changes in the ranking of classes, and that the target shift is not severe. In contrast, if the weighted rank correlation is negative with large magnitude, it implies the rankings of classes has changed significantly. This constitutes a more severe target shift. We therefore use weighted Kendall's r (Shieh] 1998). which we henceforth refer of target shift between p(y. y and g(y, y'). We provide the definition of 7 in the supplementary material. It relies on a weighting function, for which we use 1/(r + 1), where is the ranking.
T, to measure the severity
5.5 Sampling g(y,y')
Our choice of distance metric informs how to sample gly, since our goal is to evaluate across a wide range of target shift severities. We designed a method which shuffles log p(y, y') and perturbs it with noise. To be precise, let f'y E (1,2, (1.2.... /)' x )"1) such that each (y.y') is assigned a unique integer index. We use Pf(yx') = p(y, Let = A P Y"I where x ~ (.....)) We randomly shuffle these indices with क (1,2......) 21,2,..... A x iJ. Then, we define y"),
by
q(y.y') y') x exp(logp(y, + iff(y, >i
(exp(log<<(fx>>")) +6) if Si.
where .N(0, and ~1(10-12,5)
8

---

Attributes of people
4 Accuracy
0.2
Optimal
0.5
0.0
0.0
(0 60.6.1.0)
(0.2.0.6)
(-0.2.0.) (-0.6.0.2 (.1.0.0.6)
0.6.1.0)
(0.2.0.6) (.0.2.0.)) (-0.6.00.)) (-1.0.0.6)
Taskonomy
0.8
0.10
Optimal
0.6
Accuracy
0.4
0.35
0.2
0.30
10.7.1.0)
(0 5.0.7
(0.3.0.5)
(0. (0.1.0 3) (-0.2.0
(0:
7.1.0.0
(0.5,6.7)
(0.3.0.5) (0.1.0.3) (.0.2.0.1
NPS
STN
CSN
FFS
Figure 3: order evaluate whether GMTL improves robustness to target shift, we estimate OOD accuracy across range of target shift severities and a range of a E [0, 1J. The results are grouped w.r.t. the severity of target shift т, and we report the optimal for each group. subfigures, as the target shift becomes more severe from left to right, the optimal a monotonically increases for both Attributes of People (top left) and Taskonomy (bottom left). This is because larger a removes more spurious dependencies induced by target-causing confounding, which is more beneficial when the severity of shift increases. the right subfigures, the vertical axis the difference in accuracy between GMTL and DMTL. The improvement in accuracy from using GMTL increases w.r.t. the severity of target shift for both Attributes of People (top right) and Taskonomy (bottom right). The reason is that the spurious dependencies induced by target-causing confounding become less predictive as the severity shift increases, and therefore removing them yields larger improvements in accuracy.
seen in the left
6 Results
We compute the OOD test accuracy across target shifts ranging from mild severe, where p(y, y') is the original test set distribution, and g(y,y' is a randomly sampled OOD distribution. First, we sample OOD g(y, y'), and compute 7 relative p(y, y"). represents the severity of target shift between gly, yy) and the original test distribution p(y, yy) We then use importance sampling to compute the OOD accuracy under each g(y,y' y') across range of [0.
The results are split into five groups that are equally spaced in terms of r. Each group represents the accuracies for range of under particular severity of target shift. For each group, compute the optimal value of a that attains the highest accuracy. We also compute the difference in accuracy between GMTL and DMTL the optimal value of a for each group.
We observe strikingly similar results for both datasets and all pairs of tasks. As the target shift becomes more severe, the optimal increases. A representative example for each dataset is shown in the left subfigures in Fig. 3 and the rest are in the supplementary material. This observation matches understanding of a from Section 3.1] Since
our degree of belief on how similar p(y. y') and q(y, y') are, the optimal a tends to be higher when the target shift is more severe. Increasing a removes more of the spurious dependencies induced by target-causing confounding, which is more beneficial for severe target shifts.
Also, as the severity of target shift increases, the difference in accuracy between GMTL and DMTL for the optimal a increases as well. This can be seen the right subfigures Fig. 3 and is also in-line with our expectations. As the target shift becomes more severe, the spurious dependencies induced by target-causing confounding become less predictive, and removing them yields larger improvements in predictive performance.

---

Both patterns hold very similarly for all MTL methods. This suggests that it is our formulation of GMTL that is significant, rather than the particular parameterization used to learn ply, y x). Both the optimal o, as well as the corresponding improvement in accuracy using GMTL, increase monotonically w.r.t. the severity of target shift. This is strong relation that holds across datasets, pairs of tasks, and MTL methods. To quantify the generality of this result, we aggregate all of our results, and compute the correlation between and the optimal a. Since T is used in the context of an interval, take the midpoint of each interval. This results in correlation of -0.847. This indicates a strong positive correlation between the severity of target shift and the optimal a. The generality of this relation is strong evidence that GMTL improves robustness target-causing confounding, and also that our interpretation of a is correct. Our experimental results using the a-selection heuristic described in Section 3.1 well other results showing the accuracy for range of a, not just the optimal one, are in the supplementary material.
7 Conclusion
We presented generative multitask learning (GMTL), an approach for causal ML in the multitask setting that only changes the inference objective of conventional MTL. Our approach mitigates the effect of target-causing confounders, which are variables that cause the targets, but not the input. This removes spurious dependencies between the input and targets, and improves robustness to target shift. Looking forward, we are excited by the new perspectives that causality can bring improve weaknesses in ML. Our work demonstrates the potential for this in the context of MTL. We plan to investigate what other weaknesses ML can be improved by adopting a causal perspective.
Acknowledgements
This work was supported by grants from the National Institutes of Health (P41EB017183) the National Science Foundation (HDR-1922638) the Gordon and Betty Moore Foundation (9683), and Samsung Advanced Institute of Technology (Next Generation Deep Learning: From Pattern Recognition to AD).
References
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In EMNLP, 2021.
Héctor Martínez Alonso and Barbara Plank. When is multitask learning effective? semantic sequence prediction under varying data conditions. In EACL, 2017.
Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv, 1907.02893, 2019.
Yoshua Bengio, Yann LeCun, and Geoffrey E. Hinton. Deep learning for Al. Commun. ACM, 64(7): 58-65, 2021.
Lubomir D. Bourdev, Subhransu Maji, and Jitendra Malik. Describing people: A poselet-based approach attribute classification. In ICCV, 2011.
Rich Caruana. Multitask learning. PhD thesis, Cameeiie Mellon University, 1997.
Sauhaarda Chowdhuri, Tushar Pankaj, and Karl Zipser. Multinet: Multi-modal multi-task learning for autonomous driving. In WACV, 2019.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In RecSys, 2016.
Alexander D' Amour, Katherine A. Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby. Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yi-An Ma, Cory Y. McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natara- jan, Christopher Nielson, Thomas F. Osbome, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica
10

---

Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Un- derspecification presents challenges for credibility in modern machine learning. arXiv, 201 2011..00 1.03955, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras. Jacob Steinhardt, and Alek- sander Madry. Identifying statistical bias in dataset replication. In ICML, 2020.
Robert
Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nat. Mach. Intell., 2(1 (11))665-673, 2020.
Geirhos,
Ishaan Gulrajani and David Lopez-Pazz In search of lost domain generalization. In ICLR, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Jason Jo and Yoshua Bengio. Mcasuring the tendency of cnns to learn surface statistical regularities.
arXiv, 1711.11561, 2017.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic rein- forcement learning at scale. arXiv, 2104.08212, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Trent Kyono, Fiona J Gilbert, and Mihaela Van Der Schaar. Triage of 2d mammographic images using multi-view multi-task convolutional neural networks. ACM Transactions on Computing for Healthcare, 2(3):1-24, 2021.
Lukas Liebel and Marco Körner. Auxiliary tasks in multi-task learning. arXiv, 1805.06334, 2018.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and lianfeng Gao. Multi-task deep neural networks for natural language understanding. In ACL, 2019.
Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lohard and Bernhard Schölkopf. Nonlinear invariant risk minimization: causal approach. arXiv, 2102. 2021.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In KDD, 2018.
Maggie Makar, Ben Packer, Dan Moldovan, Davis Blalock, Yoni Halpern, and Alexander D' Amour. Causally motivated shorteut removal using auxiliary labels. In AISTATS, 2022.
Bryan McCann. Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv, 1806.08730, 2018.
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Compur. Surv,,
2021.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In CVPR, 2016.
Judea Pearl. Causality. Cambridge university press, 2009.
Aahlad Manas Puli, Lily H. Zhang, Erie K. Oermann, and Rajesh Ranganath. Predictive modeling in the presence of nuisance-induced spurious correlations. arXiv, 2107.00520, 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.

---

Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In SIGKDD, 2016.
Sebastian Ruder. An overview of multi-task learning deep neural networks. arXiv, 1706.05098. 2017.
Bernhard Schölkopf. Causality for machine learning. arXiv, 1911. 2019.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris M. Mooij. On causal and anticausal learning. In ICMI., 2012.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. arXiv, 2102.11107, 2021.
Grace S Shieh. A weighted kendall's tau statistic. Statistics & probability letters, 39(1):17-24, 1998.
Trevor Standley, Amir Roshan Zamir, Dawn Chen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In ICML., 2020.
Adarsh Subbaswamy, Peter Schulam, and Suchi Saria. Preventing failures due to dataset shift: Learning predictive models that transport. In AISTATS, 2019.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Tumer, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, 2019.
Partoo Vafaeikia, Khashayar Namdar, and Farzad Khalvati. A brief review of deep multi-task learning and auxiliary task learning. arXiv, 2007.01 01 126, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
Yilun Wang and Michal Kosinski. Deep neural networks are more accurate than humans at detecting sexual orientation from facial images. Journal of personality and social psychology. 114(2):246, 2018.
Yixin Wang and David M Blei The blessings of multiple causes. Journal of the American Statistical Association, 1174529) 2019.
Yixin Wang and Michael Jordan. Desiderata for representation learning: A causal perspective. arXiv,, 2109.03795, 2021.
Joseph Worsham and Jugal Kalita. Multi-task learning for natural language processing in the 2020s: where are we going? Pattern Recognition Letters, 136: 126,16, 2020.
Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi. Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In CVPR, 2022.
Amir Roshan Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.
Amir Roshan Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao. Jitendra Malik, and Leonidas J. Guibas. Robust learning through cross-task consistency. In CVPR, 2020.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In ICML, 2013.
Yu Zhang and Qiang Yang. A survey multi-task learning. arXiv, 1707.08 000708110 14, 2017.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
12

---

(b) Did you describe the limitations of your work? [Yes] See Section 3.1
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof' of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen-
tary material
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section Jndd the supplementary material
(c) Did you report error bars (c.g., with respect to the random seed after running ex- periments multiple times)? [Yes] Our main results in Section 6 do not allow error bars, but they represent the combined results of five random seeds. Our results in the supplementary material do have error bars w.r.t. the random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplementary material
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section S we use two public datasets: Attributes of People and Taskonomy.
(b) Did you mention the license of the assets? [Yes] We mention the license of the two public datasets in the supplementary material.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We include our code in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A] No need to, the datasets are public.
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [Yes] See Section 5 the Attributes of People dataset has gender label that we drop, since predicting gender can have negative consequences.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "yQ071W9vUvl", "reviewer": "Reviewer_yUFE", "summary": "In this paper, the authors propose a method for improving model robustness to target shift (i.e., shift in prior probability of labels) in multitask learning (MTL). Specifically, the authors consider a different hypothetical data generating process (DGP) than common MTL formulations: the authors assume that the observed data is generated causally from the targets, and that the targets are subject to unobserved confounding. This differs from the common assumption that tasks $Y$ and $Y'$ are conditionally independent given the data $X$. The authors use this DGP to show that modeling the the likelihood (i.e., $p(x|y,y')$) is invariant to shifts in the unobserved confounder, and thus is invariant to target shifts. Using Bayes rule and relaxing to factorize by task, the proposed method (GMTL) amounts to a modification of \"standard\" discriminative MTL to account for possible changes to the label distribution. Using synthetic and reweighted (to simulate target shifts) data, the authors show that with appropriate tuning the proposed method yields more robustness to target shifts that standard MTL which ignores the possibility of target shifts.", "questions": "Could the authors please address my questions about:\n* What is the effect of allowing $p(y, y'|x)$ to factorize (when it does not under GMTL assumptions)? (See discussion above)\n* Why is the $\\alpha$ based interpolation the correct way to implement GMTL given that the uncertainty set is more general? What can be said about the model's performance across the uncertainty set? (See discussion above)", "limitations": "Yes, the authors adequately address the limitations and potential negative impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea in this paper is interesting: the authors propose to use knowledge about the data generating process (DGP) to improve the robustness of MTL under target shift. The premise of the approach (that a model of the data likelihood will be invariant to the shift) is sound and sensible, and the implementation is convenient (builds upon discriminate formulations to MTL). However, I think the framing of the target-causing confounding problem misses some context, and I have some questions/concerns about some choices/relaxations the authors make in implementing the proposed approach. In light of this, I think the paper is missing some more in-depth analysis of the performance expectations of the proposed approach.\n\n**Re: Framing**\n\nI have a few comments about the way the authors frame their method, as the framing seems to be presented as one of the primary contributions of the paper. In the introduction, the authors contrast the proposed approach (GMTL) with existing discriminative MTL (DMTL) approaches by saying that \"DMTL is flawed under an alternative set of assumptions.\" I just want to point out that in Fig 1, from a statistical perspective, the graphs in Fig1a and Fig1b (DMTL under the different assumption sets) are incompatible. This is because the DMTL assumption that $Y \\perp Y' | X$ holds in Fig1a but does not hold in Fig1b. Importantly, this is a *testable* assumption---because $X$, $Y$, and $Y'$ are observed in the training data, we can perform conditional independence tests to measure the compatibility of this assumption with the available data. Thus, in theory, it seems a user should be able to determine if the DMTL assumption is reasonable for their problem.\n\nSecond, while to my knowledge the GMTL formulation is novel for multitask learning, I would like to note that in supervised learning the DAG in Fig 1c has been considered in [1] (see Fig 1 in [1]). In one experiment they consider the target shift scenario (Section 5.1) and their approach ends up modeling (the equivalent of) $p(x|y,y')$. Given the similarities in the resulting objectives, I think some discussion of this work and its related threads (i.e., causal graphical approaches to stability/invariance to distribution shifts) is likely warranted.\n\nI also think this connection to causal graphical approaches opens up additional methodological possibilities for the authors' line of work. For example, in Section 3, the authors note that the density estimation problem for $p(x | y, y')$ will, in general, be quite difficult and instead use equation (5) as their target of inference (and further relax it in eq (6)). However, by noting possible connections to the causal approaches, there may be alternative approaches to fitting the model such as inverse propensity weights (IPW). The authors would need to double check me, but I think reweighting the data by (something like) $\\frac{1}{p(y, y')}$ during the training of a discriminative model would be an alternative approach for achieving the desired invariance (see, e.g., the weighting scheme in [2]).\n\n[1] Subbaswamy, A., Schulam, P., & Saria, S. Preventing failures due to dataset shift: Learning predictive models that transport. AISTATS, 2019.\n\n[2] Makar, M., Packer, B., Moldovan, D., Blalock, D., Halpern, Y., & D’Amour, A. Causally motivated shortcut removal using auxiliary labels. AISTATS, 2022.\n\n**Re: relaxing the GMTL objective**\n\nIn Section 3.1, the authors \"allow $p(y, y'|x)$ to factorize... purely out of convenience.\" While I understand why the authors want to do this (it reduces the multitask network part of GMTL to DMTL), it is clearly the case that $Y \\not \\perp Y' | X$ in Fig 1c (even if there were no unobserved confounder $U$ !). The authors assert that \"we can allow this factorization as long as we capture the dependency between the targets when estimating $p(y, y')$\", but this is not obvious to me. I would have expected to see either a theoretical justification or some experiments showing the effect of this relaxation by comparing the approach in eq (6) to one which does not assume that $p(y, y'|x)$ factorizes.\n\n(As a side note, do cross-stitch networks (CSN) count as modeling $p(y, y'|x)$ jointly? If so, it might be possible to use CSN to perform the above comparison, but an approach that more explicitly jointly models $p(y, y'|x)$ might be better.)\n\n**Re: interpretation of $\\alpha$**\n\nIn equation (6), the authors also introduce a new hyperparameter $\\alpha$ which interpolates between DMTL and GMTL. The motivation is to allow for solutions which generalize better under small or moderate shifts to the joint label distribution. However, it wasn't clear to me why this would be the correct or optimal approach for handling an uncertainty set of test distributions under target shift. In the simulated shifts considered in Section 5.4, it seems the authors allow for essentially arbitrary shifts to $p(y,y')$. This will include shifted label distributions that cannot be modeled in the form $p(y, y')^{1-\\alpha}$, since the $\\alpha$-based transformation will never change the mode (or relative order) of the probability mass function (except when $\\alpha = 1$ in which case the label distribution becomes uniform).\n\nThe authors select an \"optimal\" $\\alpha$ by finding the value that performs best on the shifted data, but why is the $\\alpha$ based interpolation the correct way to implement GMTL given that the uncertainty set is more general? Can a given $\\alpha$ value be framed as the solution to, e.g., a distributionally robust optimization problem for a certain uncertainty set? What can we say about the worst-case, average, or some other performance measure across the uncertainty set for GMTL? For $\\alpha=1$, I see from eq (5) that predictions should be invariant to target shifts. What, though, can be said about model accuracy/performance? For $0 < \\alpha < 1$ what can be said about performance?\n\n\n**Minor notes**\n* This is a very minor note, but isn't it the case that when interpreting $\\alpha$ the test target distribution is *proportional* to $p(y,y')^{1-\\alpha}$? If $p$ is a valid pmf, then $q=p^{1-\\alpha}$ generally won't sum to 1 and needs to be renormalized.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "q_Q1-vRu9TE", "reviewer": "Reviewer_q5bZ", "summary": "This paper proposes a generative multitask learning (GMTL) method to handle the existence of target-causing confounding from a generative model perspective. It tries to find the invariant information under different interventions (or in target-shift case) with the help of an auxiliary target. Experimental results show the robustness of the target shift.", "questions": "See my comments above.", "limitations": "Not available.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper introduces the idea of anticausal learning for multitask learning. With the causal factorization of the joint distribution, the invariant information is used for tackling the target shift problem.\n2. The accuracy of the experimental results demonstrates the effectiveness of the proposed methods. \n\nWeaknesses:\n1. Why p(y, y’|x) in Eq. (5) can be transferred into the product of p(y|x) and p(y’|x) (in Eq. (6)). According to Figure 1(c), in GMTL, y and y’ are not independent conditional on x, but are independent with each other conditional on latent variable U.\n2. Is there any effective method to determine the $\\alpha$ when there is no prior information?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "g0iIa1L1LTn", "reviewer": "Reviewer_yfCS", "summary": "This paper considers a more relaxed assumption in multitask learning, compared to previous works: targets are not conditionally independent given the input, but rather caused by a hidden confounding factor. Therefore, the authors propose generative multitask learning (GMTL), where the confounding factor is addressed in loss function. The GMTL loss can be linearly interpolated together with traditional multitask (denoted as DMTL) loss, with a tunable interpolation factor. Experiments are run by augmenting different multitask learning methods with the new loss on different datasets, and result shows change in accuracy is positively correlated to the weight of interpolation factor on GMTL loss.\n", "questions": "(1) Under what conditions can we assume the existence of such a confounding factor, and why the experiments satisfy these conditions? \n(2) How to extend this two-target system to multiple targets?", "limitations": "Limitations of this method are not addressed in the paper. One limitation can be it is hard to infer the linear interpolation factor in general tasks.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n(1) This seems to be the first paper addressing confounding factors that affect multitask targets. \n(2) The GMTL objective is formulated with clear logic. \n(3) Experiment results support the claims.\nWeaknesses: \n(1) It is unclear why the experiments satisfy the assumption, where some confounding factor u cause y and y’. \n(2) The authors mention that they use two targets only “without loss of generality”, but it is unclear how to extend this method to multiple targets - do we assume a common confounding factor for all targets, or one confounding factor per pair of targets, or something else?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept