Multitask Learning Can Improve Worst-Group Outcomes
Authors: Atharva Kulkarni
Lucio M. Dery
Amrith Setlur
Aditi Raghunathan
Ameet Talwalkar
Graham Neubig
Venue: Accepted by TMLR ()

Abstract: In order to create machine learning systems that serve a variety of users well, it is vital to not only achieve high average performance but also ensure equitable outcomes across diverse groups. However, most machine learning methods are designed to improve a model's average performance on a chosen end task without consideration for their impact on worst group error. Multitask learning (MTL) is one such widely used technique. In this paper, we seek not only to understand the impact of MTL on worst-group accuracy but also to explore its potential as a tool to address the challenge of group-wise fairness. We primarily consider the standard setting of fine-tuning a pre-trained model, where, following recent work \citep{gururangan2020don, dery2023aang}, we multitask the end task with the pre-training objective constructed from the end task data itself. In settings with few or no group annotations, we find that multitasking often, but not consistently, achieves better worst-group accuracy than Just-Train-Twice (JTT; \citet{pmlr-v139-liu21f}) -- a representative distributionally robust optimization (DRO) method. Leveraging insights from synthetic data experiments, we propose to modify standard MTL by regularizing the joint multitask representation space. We run a large number of fine-tuning experiments across computer vision and natural language processing datasets and find that our regularized MTL approach \emph{consistently} outperforms JTT on both average and worst-group outcomes. Our official code can be found here: \href{https://github.com/atharvajk98/MTL-group-robustness.git}{\url{https://github.com/atharvajk98/MTL-group-robustness}}.

Published in Transactions on Machine Learning Research (02/2024)
Multitask Learning Can Improve Worst-Group Outcomes
Atharva Kulkarni'
atharvak@ts.com wwt.essssss................... edu
Language Technologies Institute, School of Computer Science Carnegie Mellon University
Lucio M. Dery*
Idery@cs. cmvu. edu
Computer Science Department, School of Computer Science Carnegie Mellon University
Amrith Setlur
asetlur@cs.cmu..dt edu
Machine Learning Department, School of Computer Science
Carnegie Mellon University
Aditi Raghunathan
raditiinss.c crau. edu
Computer Science Department, School of Computer Science
Carnegie Mellon University
Ameet Talwalkar
alalwalk@cs.cm cow.auaalkessccmu..co crvu, edu
Machine Learning Department, School of Computer Science
Carnegie Mellon University
Graham Neubig
gneubig@es.cmul
Language Technologies Institute, School of Computer Science
Carnegie Mellon Universily
Reviewed on OpenReview: https: http:///ppereeeeeee/mm net/ forun? id= sPLhAIGmm
Abstract
In order to create machine learning systems that serve variety of users well, it is vital to not only achieve high average performance but also ensure equitable outcomes across diverse groups. However, most machine learning methods are designed to improve a model's average performance on chosen end task without consideration for their impact on worst group error. Multitask learning (MTL) is one such widely used technique. In this paper, seek not only to understand the impact of MTT on worst-group accuracy but also to explore its potential as tool to address the challenge of group-wise fairness. We primarily consider the standard setting of fine-tuning pre-trained model, where, following recent work (Gururangan el al. 2020 Dery et al. 2023). we multitask the end task with the pre- training objective constructed from the end task data itself. In settings with few or no group annotations, we find that multitasking often, but not consistently, achieves better worst-group accuracy than Just-Train Train-Twice (JTT: Liu et al. (2021)) representative distribuutonally robust optimization (DRO) method. Leveraging insights from synthetic data experiments, propose to modify standard MTL by regularizing the joint multitask representation space. We run large number of fine-tuning experiments across computer vision and natural language processing datasets and find that our regularized MTL approach consistently outperforms JTT on both average and worst-group outcomes. Our official code can be found here: tttps://githul www/coo///taa/o http///attaraaaoo///L------
Equal contribution.

---

Published in Transactions on Machine Learning Research (02/2024)
1 Introduction
As machine learning systems exert ever-increasing influence on the real world, it is paramount that they not only perform well on aggregate but also exhibit equitable outcomes across diverse subgroups characterized
by attributes like race (Buolamwini & Gebru 2018 Liang et al.] 2021). gender (Buolamwini & Gebru, 2018:
Srinivasan & Bisk, 2022) and goographic location (Jurgens et al. 2017 De Vries et al.] 2019 Ayush et al.
2021j. Therefore, it is important to understand the impact of widely-nsed machine learning techniques with
respect to these desiderata. Multitask learning (MTT.) (Caruana 1997 Baxter 2000 Ruder el. al. 2019 Dery et al. 2021a) is one example of such technique that features prominently in machine learning practitioners' toolbox for improving& model's aggregate performance. However, the effect of multitask learning worst group outcomes is underexplored. In this paper, we both study the impact of MTL, as is, worst group error and also consider whether modifications can be made to improve its effect on worst group outcomes.
Traditionally, the problem of worst-group generalization has been tackled explicitly via methods such as distributionally robust optimization (DRO) (Ben-Tal Sen-Tales et al. 2013 Duchi & Namkoong 2018 Hashimoto et al. 2018 Sagawa et al., 2020a). In contrast to traditional empirical risk minimization, DRO aims to minimize the worst-case risk over a predefined set of distributions (the uncertoinly set). Defining the uncertainty set usually (but not always) requires access to group annotations. Since average performance based approaches like MTL are typically designed without consideration of group annotations, our focus will be settings with limited-to-no group annotations. In these settings, there exist number of generalized Reweighting
(GRW) algorithms for distributional robustness (Nam el al. 2020 Liu el al. 2021 Zhang el. al. 2022 Nam
iet al. 2022 Qiu et al., 2023 Zhai et al.) 2023 Izmailov et al.] 2022]. These approaches minimize the weighted average risk based on the weight assigned to each example. One such widely used method is the Just-Train: Twice (JTT) algorithm (Liu et al. (021). which performs two model training runs: one identify poorly performing examples and another run that upweights these examples. For our empirical explorations, we take JTT representative DRO method and use it to provide reference performance to situate our study of multitask learning.
We focus our investigations of multitask learning on the ubiquitous setting of fine-tuning a pre-trained model. Here, a common way to improve end task average performance to multitask the end task with the pre- training objective constructed over the task data itsclf (Gururangan et al. 2020 Dery et al.) 2021b). We intuit that this multitasking approach could improve robustness to worst group outcomes since previous work like Hendrycks et al. (2019 2020): Mao et al. (2020) has established a favorable connection between pre- training and robustness (both adversarial and out-of- distribution). We test our intuition by conducting preliminary experiments across pair of computer vision and natural language tasks in two settings: one with limited group annotations and the other with none. Initial results (Table [] reveal that multitasking shows promise in that it can improve worst group outcomes over ERM and JTT/ However, these improvements are not consistent. Therefore, we are spurred to consider modifications to make it a more competitive tool against worst group outcomes.
Table 1: Standard multitasking improves worst group outcomes over ERM and JTT but not consistently. Experimental details can be found in Section A.I
No Group Labels
Val Group Labels
Dataset
Method
Koost Ace
Woest-Ghoup Ace
ERM
80.1gg
S5.4,4
Waterbirds
JTT
82.
85.92s
(MTL) ERM-MIM
80.1.
65.324
ERM
51.8:4
87. Ae.
CivilSmmall
JTT
62.6.3
88 Dy
(MTL.) MRM
58.3.
08.50
In order to build intuition about how to adapt MTL to target the worst group error, we conduct controlled experiments on two-layer linear models trained from scratch on synthetic data. We borrow the synthetic data setup introduced by Sagawa et al. (2020b) in which training data consists of two majority groups, where spurious features (features that are not required to robustly solve the end task) are predictive of the end task oulput, and two minority groups, where spurious features are uncorrelated with the output. [Sagawa el al. (2020b) demonstrated that under certain conditions on the generative distribution of the input features, lincar models trained on such data would probably rely on spurious features and thus, have poor worst group error. Working with this simplified setup where worst-group outcomes are easily inducible allows us to study MTL's effects more incisively. We instantiate reconstruction from noised input, as our auxiliary
2

---

Published in Transactions on Machine Learning Research (02/2024)
task to perform multitask learning in the synthetic setup. This choice is partially informed by the fact that many pre-training ovjectives like masked language modeling (MLM) (Devlin et al. 2018) and masked image modeling (He et al.] 2022 Tong et al. 2022) are based on imput reconstruction, When training solely on this auxiliary task, we uncover that regularizing the pro-output layer of the model is critical for ensuring that the model upweights the core features (features required to robustly solve the end task) over the spurious ones. This leads us to the following recipe for improving worst-group error: regularized multitask learning of the end task with the (appropriately chosen) auxiliary objective.
Through battery of experiments across natural language processing (NLP) and computer vision (CV) datasets, we demonstrate that multitasking the end task with the pre-training objective along with & regularization on the shared, pre-prediction layer activations is competitive when pitted against state- of-the-art DRO approaches like JTT (Liu et al. 2021) and Bitrate-Con Constrained DRO (Sethur et al. 2023). Specifically,
settings where only validation group annotations are available, regularized MTT. outperforms JTT and BR-DRO on 3/3 and 2/3 datasets, respectively. Our approach improves worst-group 4%) and JTT (by 1%) in settings where group annotations
accuracy over ERM (by as much are completely unavailable. Moreover, regularized MTL consistently outperforms both ERM and JTT average performance, regardless of whether group annotations are available or not. Thus, within the prevailing framework of utilizing pre-trained models for downstream fine-tuning, our results demonstrate that regularized multitask learning can be a simple yet versatile and robust tool for improving both average and worst-group outcomes.
2 Informal motivation for our regularized MTL method
Problem Sctup Preliminarics: Let cach input example x E have classification label y E Y and demographic attribute E S. Each group g v) G is defined by the label y and the altribute 8, such that spuriously correlates with y. Thus, the sample space of G is the Cartesian product of Y and S. Our goal is to learn model that minimizes the worst-group error. We evaluate the models based on their worst group accuracy (WGA), i.e., the minimum predictive accuracies of our model across all groups. We are interested in the setting where the spurious attribute 8, and consequently, the group identity are unavailable (or available to a very limited degree) at training time, as annotating spurious attributes is typically expensive.
Why would expect multitask learning to help mitigate worst group outcomes? It would be naive to assume that multitasking the end task with any auxiliary task would prevent poor group outcomes. In order to better understand intuitively which auxiliary tasks may be helpful, we first provide an example using the data generation process and linear model setup presented in Sagawa et al. (2020b)'s work on the effect of spurious and core features on worst-group accuracy.
When do models incur high worst group error? Sagawa et al. (020b) describe a simple data- generating distribution that defines, for each example, label y E {-1, 1]. and features The features described as either core features Foore if they are associated with the label y, or spurious features Tspur if they are associated with the spurious attribute
spurious attribute s E [[1,1],
Feore y No (yl, coor Ja.) Tspur
(1)
8 ~ N (sl, 2apur Jd.)
= [Fcore: Pspur] E Rd and d = (de + de)
x
We can then define linear model, parameterized by w, that predicts the label given the features.
(0) w (0)
(2)
Note that because the core features Toore are the ones associated with the label to be predicted, they are the ones that the model should use in order to attain high predictive accuracy.
We will delve deeper into other motivations in Section
3

---

Published in Transactions on Machine Learning Research (02/2024)
The cross-product of the space of possible labels y #1 and spurious attributes s= =1 divides the samples generated from this distribution into four groups. When some groups are more frequent than others in the training data, a correlation between 5} is created. Further still, in the presence of the above correlation, if the spurions features have lower variance with respect to the data generating process (Equation ! i.c., pepur S core: linear models will tend to rely more on (assign higher weight to) the spurious features over the core ones (Sagawa et al.j 2020b). This learned reliance on the spurious features instead of the core features that are truly predictive of the label results in poor worst-group error.
Why does reconstruction help? Considering the above, for auxiliary task to be helpful, it should discourage the model from using spurious features by showing stronger preference for core features in exactly the case when o S ooore In this paper, we argue that one class of tasks that fulfills this criterion are reconstruction tasks, where we predict original input features from noised versions.
For instance, the example above, we add noise with a constant variance of goise over each dimension, it results in noised imputs that have variances 2 spur (o + moisee I &eore (o + o moise) per spurious and core feature dimension, respectively. Under the assumption that both true labels y = and y -1 are equally probable, and in the simplest case where are reconstructing features independently of each other with a linear predictor f wyd; (where I is the noised input), the Bayes optimal weight on feature would be (see Appendix A.3 for the full proof):
of + +05 (1/0=1 + 12 y=
W.
bayes
(3)
of + 0.5
11/0=1 + u2 1R + nolae
where
2 Mily=11 are the per-dimension means from Eqn 1
Note that w. bayes is larger for dimensions with higher variances of assuming 1 are symmetric across core and spurious features (i.e all i). Thus, this reconstruction task places more weight on the core features in exactly the setting where a linear predictor for the end task would prefer to use the spurious features. Note that for this preference of the core features be effectively realized, the auxiliary task needs to be sufficiently up-weighted, but not so much so that the end task not learned at all.
Why is regularization necessary? Given an auxiliary task with the above property of preferring core to spurious features (under o sppur doore), model with sufficient capacity can still rely spurious features for solving the end lask. We can incentivize the model to mostly use core features by applying sulficient regularization to the parts of the model that are shared between the two tasks (such as shared feature extractors). The restricted capacity encourages the model to rely on features that would cause it to do well on both lasks, which would be the core features.
Based on the intuition established in this section, we propose a simple yet effective method for improving worst-group outcomes: Multitasking the end task with the pre-training objective which tends to be reconstruction lasks while regularizing the shared (pre-prediccton)) layer. In Section we will test this intuition through synthetic data experiments, and in Sections 4 and we demonstrate its empirical efficacy through natural data experiments.
3 Synthetic Data Experiments
We initiate our investigation with an empirical study in a simplified context, involving the training of a two-layer linear model on synthetic data. This exploration is designed to substantiate the informal intuition introduced in Section
through concrete empirical findings.
3.1 Data Generating Distribution
We base our experiment on the data generation distribution from Equation into core and spurious ones.
I where features are divided
4

---

Published in Transactions on Machine Learning Research (02/2024)
Tra Training Data
As instantiation, we consider end the task data defined by Tend (()):N)))iEN where we have N total samples. Here de = =; d. = = d 2. The data is dominated by samples where {s = y) and thus, we have two majority groups G Gz=y=1 and Gz=y=-1, each with Rmaa samples. The two minority groups are when {s =yy each with Pmin
Spurious Feature
iHaeerty Grous) Sise
0.0 -05-
Diiner ty Grousd Class m
Dagurity Greusl Class
2
piiner by Ground Cless
G5=-y=1, G.m--=-1- Due the fact that 7majj > Timin the attribute is highly correlated with the label y in the training data and thus, is spurious feature when considering the true data generation distribution. The end task is to predict the true label from the given input data Zi.
Figure shows data sampled from the generative process described in Equation 1 We produce 1000 samples in R² with core 0.6 and 2 spur 0.1. A'min 100 and 7max 900, making the spurious feature highly correlated with the true label.
Figure training data (1000 points).
Visualization of synthetic
3.2 Training on the end task only
Task Only |a|| = 0.10
End-Task Only |al| = 10.00
Acc(Worst) = 64.15% Acc(Avg) 83.16%
Acc(Worst) = 48.30% Acc(Avg) = 75.21%
1.5
1.5
1.0
1.0
Spurious Feature
Spurious Feature
0.5
[Majority Group] Class
0.5
[Majority Group] Class
x
[Majority Group] Class 0
X
[Majority Group] Class 0
0.0
[Minorily Group] Class
0.0
[Minority Group] Class 1
[Minority Group]
[Minority Group]
-1.5
-1.5
-2
2
-2
Core Feature
Core Feature
Figure/ 2: Predictors learned when we train on the end task only. Examples visualized are the balanced test samples created from Equation
We train on Tend only to confirm that the resulting model has poor worst-group outcomes. Since we will eventually be do multitasking, we use a two layer linear model where the first layer is a linear featurizer that will eventually be shared between all lasks being multitasked and the second layer is a prediction head dedicated the end task. This shared featurizer but separate head architecture is common in modern multitask learning YYu et al. 2020 Michel et al., 2021 Dery al.] 2021a).
For simplicity, the featurizer layer f (-) is diagonal linear function parameterized by
f R Rdd [(o) (diag(a))
(4)
And the final put prediction layer is given by
end Upred
md) f(x))= (w (dia(aa) (wnond) T
Note that we have effectively parameterized a linear model with decompased formulation which will be helpful once we proceed to multitasking, The end task loss is binary cross-entropy with l regularization on
the diagonal parameterization allows easily read off much weight assigned features versus spurious
5

---

Published in Transactions on Machine Learning Research (02/2024)
Auxiliary Task only
feature weights
Median
21
6
Reconstruction Head
fa(
Shared Intermediate Layor
log
0-
3F
fa(z)
ag4tz
Prediction Head
-2
(2 md
fu(x)
8
0.1
10
ab
Figure 4:
Multitask learning architecture used in
Figure 3: The ratio log( Supur)
acore for 2 (extreme)
Section 3.4 We use shared intermediate layer and
choices of |a) across 4 hyperparameter settings (learning rate X batch size).
two separate prediction heads for Taux and Tend
wend is given by the following equation where be the sigmoid function:
Lend (wodd a)
1
Σ (2..).)
log (ar (upred) + (1 - v:) -log (1 - a (Ufred))
A +
Il 1|
(5)
N
We fit the model solely the end lask by rumning batched stochastic gradient descent on Lend- We use a batch size of 64, learning rate of 10-3, À 1, and 500 epochs. We use 100 generated points as validation data for model selection. As can be seen in Figure training on the end task only can result in predictor that achieves poor worst group error. This occurs even with varying the norm of the featurizer parameter
3.3 Training on auxiliary data only
As motivated in Section we proceed to introduce reconstruction based auxiliary task. The auxiliary task data is defined by Taux ((,,,,,) ii)i where we have M total samples. M unlabelled points (with respect to the end task) are taken from the distribution described by Equation 1 Noise of the form
Cnoise
N
x x
(6)
is applicd to each point. The task is reconstruct xs from T.. Reusing the featurizer from Equation define the following prediction model:
= (Waux) T f(a)(z)
aux
pred
W parameterizes the auxiliary prediction head which we regularize to [Waaxx Raxd | 1}. Finally, our reconstruction loss is given by:
Lreeon (Waux a)
Σ
| - (Waux)? f())
2M
Using the synthetic data instantiation in Figure I apply noise from N 12), i.e., oo on each of the 1000 training points to get the training data for the auxiliary task We fit the model solely to the auxiliary task by rumning batched stochastic gradient descent on Lrecon We use learning rates in the set 1,0-,1009) and batch sizes in the set (64, 2563.
We consider cases when the intermediate layer a has low versus high capacity, reflected in norm.. Low 4-norm |a|l1 |asparl |acorel 0.1 means restricted capacity since this constraint (along with |aux|| P 1) results in models that cannot fit the training data perfectly. High 4-rnrmm (aal)] 10)
Note that while we could generate more points the auxiliary task, would like mimic setting where refrain from introducing external data (data beyond end task training data) since methods like JTT and BR-DRO do utilize them.

---

Published in Transactions on Machine Learning Research (02/2024)
means that the model is expressive enough to perfectly fit the training data. Figure provides insight into the learned intermediate layer in either case. When the model has enough capacity, there is no competition between the core and spurious features. This means solutions where the spurious feature is weighted more than the core feature are feasible as long as the core feature weight is enough to reconstruct the the noised core features well.
However, under restricted capacity where the learned weight of the core and spurious features are in direct competition, the model has to put more weight the features in order to achieve lower auxiliary loss (as motivated in Section 2). This can be seen in Figure 3 Thus, for the auxiliary task to be effective forcing model to use core features over spurious ones, the model's capacity must be reasonably restricted.
3.4 Multitasking with regularization
Given the findings from Section 3.3 we proceed to multitask Lend and Lrocon along with regularization on the shared layer a. Let (T) = {a Rd
|a/1 = - a set of f--norm constrained vectors, we solve the following multitask optimization problem:
Wanx wend ā p A(r) froron (Waux, a) + Lend (w a)
We implement the multitask model illustrated in Figure 4 We use the same set of syper-paramotors used in Section and perform joint stochastic gradient descent on both T Tend and T Taux- When is chosen to be small enough, model capacity is restricted, and the model is forced to rely chiefly on the core features to do well on both the end and auxiliary tasks. Figure 5 evinces this. When the norm of the shared layer is high |a]]] 10, the end task can still predominantly rely on the spurious features leading to poor worst group error. On the other hand, when model capacity is reasonably restricted by setting ||a||] 0.1, we see from Figure 5 (left) that we can achieve improved worst group accuracy. Thus, we can effectively leverage the reconstruction auxiliary task in this simplified setting by applying sufficient regularization to ensure improved worst-group outcomes.
Summary of results from Sections Regularized multitasking leads to improved worst-group outcomes.
Table
3.3 and 3.4
a|l
a/|
Method
Worst. Group Ace
Worst Group Acc
End task only
64.15
48.30
Regularized MTL
94.02
0.0
Regularized MTL
Regularized MTL
|a|1 0.10 10.00
|a| 10.00 10.00
Acc(Worst) 94.02% Acc(Avg) 95.00%
Acc(Worst) 0.00% Acc(Avg) 50.00%
1.5
1.5
1.0
1.0
Spurious Feature
Spurious Feature
0.5
X
[Majority Group] Class
0.5
[Majority Group] Class
[Majority Group] Class
(Majority Group] Class
0.0
0.0
[Minority Group] Class
[Minority Group] Class
-0.5
[Minority Group] Class
-0.5
[Minority Group] Class
-1.5
-1.5
-3
-2
-3
-2
2
3
Core Feature
Core Feature
Depicted are the learned half-spaces for the multitask model under - = (0.1, 10) and a = 10. Restricting the capacity of the shared feature space critical for multitasking to be effective for improving worst group error. Examples visualized are 1000 balanced test examples sampled from Equation
Figure 5:

---

Published in Transactions on Machine Learning Research (02/2024)
4 Details For Natural Data Experiments
We have made a case for using regularized multitask learning to combat poor worst-group performance through empirical explorations in a simplifiod, synthetic setting. In this section, we review the experimental details for our investigations of tasks of more practical interest.
4.1 Datasets
We conduct experiments across three datasets. To relieve the burden of compute, we introduce a fourth dataset, a smaller, sub-sampled version of one of the original datasets for ablations.
1. Waterbirds:
This image classification dataset was introduced by Sagawa et al. (0200a). The task is distinguish between species of land and water birds. It consists of bird images sourced from the CUB dataset Wah et al. (2011) and superimposed on land or water backgrounds from the Places dataset Zhou ct al. (2018). The label (type of bird) is spuriously correlated with the background, resulting in 4 groups. Since this is a small dataset (4795 train examples), we also use it for ablations.
2. MultiNLI: This is a natural language inference dataset. The task is to classify whether the second sentence is entailed by, contradicts, or is neutral with respect to the first. sentence (Williams el al., 2018 Following TollowingSaagawa et al. (2020a), we utilize the presence of negation words as a spurious attribute, leading to the crcation of total of 6 groups.
3. Civilcomments: The Civilcomments dataset is a toxicity classification dataset that contains comments from online forums Borkan et al. (2019); Koh et al. (2021). Along with the toxicity label, each text is annotated with additional overlapping sub-group labels of 8 demographic identities: male, female, LGBTQ, Christian, Muslim, other religions, Black, and White. As per Koh et al. (2021) and Sagawa et. al. 2020a), we defines 16 overlapping groups by laking the Cartesian product of the binary toxicity label and each of the above demographic identities.
4. Civileomments-smalll As Civilcomments
large dataset of about 448000 datapoints, we create a sub-group stratified subset of 5% for conducting ablations and other detailed experiments. Our subset contains 13770, 2039, and 4866 datapoints in our train, validation, and test split, respectively.
4.2 Multitask Model and Training Details
We follow the parameter sharing paradigm [Ruder 2017 Sener & Koltun 2018 where both Tend and Taux share the same model body, parameterized by Orase. We instantiate task-specific heads, parameterized by the last layer activations immediately before
fend and aux respectively. We introduce & regularization the per-task prediction heads 0 Specifically, let hend haux Rdd be the output representations generated by the base model, which are fed into their respective task-specific heads. Our final multitask learning objective is expressed as follows:
Lrnall Lend + Claux Laux + Otyeg ([hend [|])
We cross-validate optimizing Linal with different weighting schemes. We choose Qaux and Oreg from the set
Note that whilst we optimize Linal we care only about improving worst-group error on T Taux- We use the pretrained BERT (Devlin et al. 2018) and ViT (Dosovitskiy al. (2200) as the shared base models for NLP and CV tasks, respectively. We leverage the base models' self-supervised pre- training objectives, namely, masked language modeling (MLM) and masked image modcling (MIM) for our auxiliary transfer task Taux as in Dery el al. (2021a). These auxiliary objectives are based on end-lask data itself (unless specified otherwise). We do this to maintain an apples-to-appler comparison with our chosen baselincs, which do not use external data. As Section 5 will show, we obtain performance improvements even in this setting. In Section 5.4] we show that our improvements when using task-only data are predicated on sufficient prior pre-training. More details on the multitask model and batching scheme are presented in A.1
"In contrast synthetic data experiments, where the norm coustraint is applied to pre-prediction layer weights, in this section directly apply the norm constraint to the features, indirectly constraining model weights. In the syuthetic experiment, core and spurious features have one-to-one mapping model weights, enabling direct regularization of the features, but that is longer possible in more complex models.
8

---

Published in Transactions on Machine Learning Research (02/2024)
For training, we vary the fine-tuning learning rate within the ranges of {10-3 1,00-,10-- 10104] } for Waterbirds, and (10-1. 10-55 for the text datasets. We experiment with batch sizes in the set (4,8, 16, 32). We use the same batch sizes for Tend and Taux. We train for 50 epoch for the NLP datasets and 200 epochs for Waterbirds, with an early stopping patience of 10, as per the check-pointing scheme explained in section
4.2 We use the Adam optimizer for NLP datasets with decoupled weight dlecay regularization of 10-2 [Loshchllov Hutter 2017). Consistent with the recent studies on ViT (Dosovitskiy al., 2020 Steiner jtt al., 2022), we use SGD with momentum of 0.9 (Sutskever et al., 2013) to fine-tune Waterbirds. We run each hyperparameter configuration across 5 seeds and report the averaged results. We report the ERM, JTT, and groupDRO results for Civilcomments and MultiNT.I from Tdrissi et al. (2022) as the authors conducted extensive hyperparameter tuning across all these methods. However, since Idrissi et al. (2022) report results on Waterbirds using a ResNet-50 model (He et al. 2016) and our experiments employ ViT, we re-run all baselines using VIT with a consistent set of hyperparameters, as mentioned above.
Evaluation Details We assess all methods and datasets using two model selection strategies:
1. Val-GP: This strategy requires group annotations in the validation data during training. Here, we select the model based on the maximum worst-group accuracy on the validation data.
2. No-GP: This strategy requires no access to any group annotations during training. We select the model based on the average validation accuracy.
Baseline Methods Since we evaluate our method based on its ability to generalize to worst performing groups, we benchmark it against three popular methods found in group generalization literature. These methods either directly or indirectly optimize for worst-group improvements.
1. Empirical Risk Minimization (ERM): This is the standard approach of minimizing the average loss over all the training data. No group information is used during training except when the Val-GP strategy is used for model selection.
2. Just Train Twice (JTT): It presents a two step approach for worst group generalization Liu et al. (2021). JTT first trains a standard ERM model for T epochs to identify misclassified datapoints. Then, second model is trained reweighted dataset constructed by upweighting the misclassified examples by aup. It does not group information during training except for the Val-GP strategy.
3. Bit-rate Constrained DRO (BR-DRO): Traditionally, in the two-player formulation of DRO, the adversary can use complex reweighting functions, resulting in overly pessimistic solutions. In contrast, BR-DRO Sethur et al. (2023) constrains the adversary's complexity based on information theory under inte-innependent prior. While BR-DRO offers weaker robustness without performance guarantees for arbitrary reweighting, is less pessimistic and suitable for simpler distribution shifts, characterized by a reweighting function contained in simpler complexity class. BR-DRO does not use group information during training except for the Val-GP setting.
4. Group-DRO: Group distributionally robust optimization minimizes the maximum loss across all the sub-groups Sagawa et al. (2020a). This optimization method incorporates group annolations during training. Similar to prior works Liu et al.] (2021): Tdrissi et al. (2022); Sethur et al. (2023). we treat it as an oracle, as this is the only method that uses group annotations.
5 Results And Discussion
In this section, we provide empirical evidence demonstrating the effectiveness our regularized MTL approach in mitigating worst-group error while maintaining average performance across different scenarios.
5.1 Multitasking is Competitive with Bespoke DRO Methods
We first compare our approach with previously proposed methods for tackling worst-group accuracy. Table 3 details the performance of various methods across the tasks of interest for the Val-GP sctting. As expected, groupDRO yields the highest worst-group accuracy as directly optimizes for it. Our MTL approach outperforms JTT and BR-DRO on two datasets (MNLI and Waterbirds) while performing comparatively
9

---

Published in Transactions on Machine Learning Research (02/2024)
Table 3: Mean and standard deviations of the test worst-group accuracies across all the methods under consideration. Regularized MTL consistently reduces the gap between ERM and groupDRO when considering worst-group accuracy.
Method
Group Labels Civilcomments
MNLI
Waterbirds
ERM JTT
Val Only
61.32.0
67.61.2
85.41.4
Val Ouly
67.81.6
67.51.9
85.92.5
BR-DRO ERM + MT L1
Val Ouly
68.9p.7
68.50.s
86.71.3
Val Only
68.23.2
69.71.5
87.52.7
groupDRO (Upper Bound)
Train and Val
69.91.2
78.0p.7
93.9p.7
with BR-DRO on the CivilComments dataset. Given the competitive results in Table 3 we argue that our regularized MTL formulation is an attractive option over JTT and BR-DRO. Multitasking already features prominently in many ML code bases. Thus, introducing our simple regularization modification existing MTL implementations represents smaller technical overhead compared to introducing JTT BR-DRO to target worst-group error. Also, capable of improving both worst-group and average accuracy.
will see in Section 5.2 below, regularized MTL is single approach
5.2
Multitasking Improves both Average and worst-group Performance Even in the Absence Group Annotations
Waterbirds (no validation group annotations)
Waterbirds
Waterbirds
Waterbirds (with validation group annotations)
(with validation group annotations)
(no validation group annotations)
$8.0
90.0
87.5
87.5
Average Accuracy
.... Mean
85.0
65.0
82.5
82.5
86.0
96.0
80.0
83.0
85.0
95.0
75.8
Median
66.0
22.5
Mean
93.0
$3.0
Civicommennt--me
Civilromments small
Clvilcommants-smm group annotations)
Clvlkcomment--nnce
(no validation group annotations)
(with validation group annotations)
group
Accuracy
70.0
70.0
Median
Average Accuracy
64.0
83.0
60.0
60.0
82.0
12.0
50.0
50.0
Median
Mean
ERM
JIT
ERM MLM
L1
ERM
JIT
ERM MLM
ERM
JTT
ERM MIM
ERM
JIT Methods
ERM MLM
Methods
Methods
Methods
Figure 6: Comparison the performance of different approaches with respect to average and worst-group accuracy on the Waterbirds dataset under val-GP and no-GP settings. Regularized MTL improves both average and worst-group accuracy even without group annotations. All methods enjoy lift in when validation group annotations are available.
Though previous works typically assume that practitioners have access to the group annotations the validation set (Liu et al. 2021 Kirichenko et al. 2022). we are interosted in settings where no such annotations are available. This covers many tasks of practical interest since, in some cases, it may be prohibitively cost-intensive (financially and terms human labor) to acquire group annotations even for the smaller validation set (Paranjape al., (2023). Consequently, we present comparative performance analysis Figure 6 encompassing settings with and without access to group annotations. With respect to worst-group accuracy, regularized MTL approach outperforms JTT and achieves 2% lift over ERM when group
10

---

Published in Transactions on Machine Learning Research (02/2024)
amnotations absent, trend consistent across both Waterbirds and Civilcomments-small datasets. While this lift. of R 2% remains when validation group annotations are introduced, the benefit from group-labeled data is more pronounced (+s 5%%.5%). This boost can be worthwhile to practitioners who have the resources to obtain some group annotations. Moreover, it bocomes evident from Figure that our method not only yields superior worst-group performance but also improves average performance.
5.3 Are both Regularization and Multitasking Jointly necessary?
Table 4: Discntangling the impact of L1 regularization and SSL objective on wost-group accuracy. We see that regularized multitasking is necessary for gains in both average and worst-group performance.
No Group Annotations
Val Group Annotations
Datasct
Method
Avg Ace
WG Ace
Avg Ace
WG Acc
JTT
95.6g.3
82.11.2
94.00.5
85.92.
Waterbirds
ERM
95.5p.2
80.1
94.1a.7
85.4
LI
95.60.3
82.05.4
94.70.9
86.4
+ MIM
95.3g.4
80.14.6
95 95.0p.6
85.32.4
+ MIM + L1
95.8p.3
83.33.4
95.4p.4
87.52.7
JTT
83.3g.2
52.5p.2
81.3u.s
68.01.8
Civilcomments-Smal
ERM
83.9g.4
51.6p.6
81.41.a
67.42.1
+ LI
83. 83.7a.4
51.6G.0
80.3u.7
66.31.6
+ MLM
83.91.2
58.36.6
80.30.7
68.50.4
MLM
L.
84.4p.4
53.74.3
82.0p.5
69.41.7
In this section, we conduct an ablation to verify if both multitask learning and regularizing the final layer joint embedding space necessary to improve average and worst-group performance. Our results are captured in Table When assessing the worst-group accuracy, we find that regularizing the final embedding space during ERM can, at times, result in worse performance compared to training via standard ERM (66.31. 6 vs 67.42. on CivilComment--small with validation group labels). On the othe hand, multitasking without regularization can fail to improve over ERM, as evinced by the lack of improvement on Waterbirds. The regularized MTL approach is the only setting consistently improving on both datasets with and without validation group annotations. In line with these findings, we observe that joint L1 regularization and multitask learning setup yields the highest average accuracy in most cases.
5.4 Impact of Pre- Training
Table Waterbirds: Impact pre-training on average and worst-group accuracy.
No Group Annotations
Val Group Annotations
Pretrianed
Method
Avg Acc
WG Ace
Avg Ace
WG Acc
ERM
65.
4.51.6
53.3n.7
10.12.
No
JTT
67.0s.3
10.8
56.22.1
49.94.0
ERM + MTM + L1
67.02.3
1.65p.7
53.52.
12.0g.2
ERM
95.502
80.1
94.1o.7
85.41.4
yes
JTT
95.6p.3
82.1
94.0n.5
85.92.5
ERM + MTM + L1
95.80.3
83.33.4
95.4p.4
87.52.7
Fine-tuning pre-trained models is arguably the de-facto paradigm in machinc learning (Devlin et al. 2018 Dosovitskiy et al. 2020 Dery et al. 2021b). Consequently, our experiments so far have exclusively focused on pre-trained models. In this section, we wish to understand the effect of deviating from this paradigm on our MTL approach. Thus, we compare against JTT and ERM when the model is trained from scratch instead of starting with a pro-trained model. Tables and depict our results on Waterbirds and Civilcomments. small, respectively. Our results show that pre-training is critical for setting up regularized MTL viable
11

---

Published in Transactions on Machine Learning Research (02/2024)
Table 6: Civilcomments--mmee
Impact of pre-training on average and worst-grout accuracy.
No Group Annotations
Val Group Annotations
Pretrianed
Method
Avg Ace
WG Acc
Avg Ace
WG Acc
FRM
80.70.s
31.17.2
74.40.9
54.0a.7
No
JTT
79.60.s
34.9g.9
74.31.2
58.71.3 1.3
80.7p.6
31.37.6
71.20.3
56.2p.9
ERM
83.90.4
51.65.6
81.41.0
67.42.1
Yes
JTT
83.3u.2
52.55.2
81.30.8
68.0
MRM+LL
84.4p.4
53.74.3
82.0p.6
69.41.7
remedy against poor worst-group outcomes. We posit the following explanation for this outcome. Note that our informal motivation in Section 2 presupposes an ability to solve the auxiliary task to a reasonable degree. Solving the MLM and MIM tasks effectively from scratch with only the inputs of the relatively small supervised dataset difficult. This poor performance on the auxiliary task translates to an inability to constrain the use of the spurious features on the end-task. Consistent with prior works (Tu et al. 2020; Wiles et al. 2022). our recommendation to practitioners is to use our approach during the fine-tuning of pre-trained models to be maximally effective.
Another consequence of our findings is that caution is warranted in interpreting the results of previous work on DRO in light of the new paradigm of mostly using pre-trained models. Most previous results on DRO have examined the setting of training from scratch, and as Tables f and 5 DRO methods significantly outperform competitors in that setting. However, the originally outsized gains in worst-group error significantly shrink when we move pre-trained models whilst method shows superior performance.
6 Related Work
Multitask Learning. Multitask learning is a common stratergy for ML practitioners to improve the
average performance of their models [Ruder] 2017 Ruder et al.) 2019 Liu et al.) 2019). While work like Hendrycks et al. (2019 2020) Mao al. (2020) have shown that multitasking can improve the adversarial and out-of-distributtio robustness of models, the impact of multitasking on worst-group outcomes has been relatively unexplored. Makino et al. 2022 propose generative multitask learning (GMTL), a method that bolsters robustness to larget shift by conditioning the input on all available targels, thereby addressing challenges associated with target-causing confounders and spurious dependencies between input and targets. However, it is important to note that their approach necessitates all target annotations during training, a requirement we do not assume in our scenario. Our work is inspired by Gururangan et al.. (2020). who introduce constructing auxiliary objectives directly from end-task data. for continued pre- training (they dub this Task Adaptive Pre-training TAPT). Following Dery et al.) (2021b) 2023), we multitask this auxiliary task with the end-task. However, unlike these studies, the worst-case group accuracy of the final model. The work by Michel et al. (202T) explores the balancing of worst and average performance in multitask learning. In their study, they focus on a set of equally important end (asks, striving for proficient model performance across all of them. In contrast, our work delves into the asymmetrical multitask setting, where the presence of the auxiliary task is determined by its contribution to enhancing our target metric the end task.
focus is on improving
Robustness using group demographics. Our multitask learning approach is primarily designed for settings with limited-to-no group annotations. However, many DRO approaches assume the presence of group annotations for all training points. Among the approaches that leverage group information, Group Distributionally Robust Optimization Sagawa et al. (2020a.) is the most popular technique that trics to minimize the maximum loss over the sub-groups. Goel et al.] (2021) presented Model Patching, a data augmentation method designed to enhance the representation of minority groups. FISH, proposed by [Shil ct al. (2022) focuses on domain generalization via inter-domain gradient matching. In the settings where group annotations are expensive (financially or in terms of human resources) procure, these methods are not viable options.
12

---

Published in Transactions on Machine Learning Research (02/2024)
Robustness without group demographies. Extensive research has been dedicated to addressing the challenges of worst-group generalizat ion in the more realistic scenario where access to group annotations during training is unavailable. GEORGE Sohoni et al. (2020) adopts a muastering-asser methodology to unveil latent groups within the datasct and subsequently employs groupDRO for improved robustness. Learning from Failure (I.fF) Nam et al. (2020) introduces a two-stage strategy. In the first stage, an intentionally biased model aims to identify minority instances where spurious correlations do not apply. In the second stage, the identified examples are given increased weight during the training of second model. Just Thain Twice (JTT) method Liu et al. (2021) follows similar principle by training model that minimizes loss over reweighted dataset. This dataset is constructed by up-weighting training examples misclassified during the initial few epochs. Our regularized MTL approach has several advantages over these methods, even though they are all deployed in the same limited-to-no group annotations settings. As we have demonstrated, our approach can improve both worst-group and average performance, unlike the other approaches targeted against worst-group crror only. Secondly, due to the widespread usage of multitask learning by many ML practitioners, implementing our modification represents minimal overhead instead of introducing one of the above bespoke approaches.
7 Conclusion
In this work, we presented an empirical investigation of the impact of multitasking on worst-group outcomes. We found that deploying multitasking, as is, does not consistently improve upon worst performing groups. We have shown that while DRO methods, like JTT, display superior performance when models are trained from scratch, this is not the case in the currently more widespread setting of fine-tuning pre-trained model. Specifically, when fine-tuning, our method regularized multitasking of the end-task with the pre-training objective constructed over end-task data- leads to improvements in worst-case group accuracy over JTT. Our work has demonstrated that is possible to design a single, simple method that improves both worst- case group accuracy average accuracy regardless of the availability of group annotations. Since multitask learning is already standard part of many practitioner's toolbox, and our modification adapt it against worst-group accuracy is simple, our approach requires minimal overhead to integrate into existing systems compared to bespoke DRO approaches. We thus encourage practitioners to introduce our modification to their MTL pipelines as an essentially free way of improving worst- performance without sacrificing gains in average performance.
In order to keep an apples-to-apples comparison with DRO approaches, we have primarily focused on multitasking with auxiliary objectives based on end-task data only. For future work, would be interesting to explore the impact of multitasking with auxiliary objectives based external data more deeply. It would also be interesting to leverage meta-learning to dynamically adapt the auxiliary tasks towards improving worst-case group outcomes (Dery el al. 2021b 2023). Additionally, we leave the study on the generalizability of our method to adversarial robustness, domain shift, and label shift to future work.
8 Acknowledgements
This work was also supported in part by the Tang AI Innovation Fund, Defence Science and Technology Agency Singapore, National Science Foundation grants IIS170512,, IIS1838017, IIS2046613, IIS2112471, and funding from Meta, Morgan Stanley, Amazon, Google, Schmidt Early Career Fellowship and Apple. Any opinions, findings, conclusions, or recommendations expressed this material are those of the author(s) and do not necessarily reflect the views of any of these funding agencies.
13

---

Published in Transactions on Machine Learning Research (02/2024)
References
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7799-5811, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 18.13//11 http1/////eer/nee mennlp-main. 2021. emnlp-main. main. 468
468. URL aattp://aacaatthologg org/2021
Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon. Geography-aware self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 10181 10190, 2021.
Jonathan Baxter, A model of inductive bias learning. Journal of artificial intelligence research, 12:149 198, 2000.
Aharon Ben-Tal, Dick Den Hertog, Anja De Wacgenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341 357, 2013.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, WWW '19, pp. 491-500, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450366755. doi:
URL https://doi.org:
10. 0t.1145/330856.3331111
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accomnlabilily and transparency, pp. 77-91. PMLR, 2018.
Rich Caruana. Multitask learning, Machine learning, 28:41 75, 1997.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive lcarning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. Does object recognition work for everyone? In Proceedings of the JEEE/CVI CVF conference on compuler vision and pallern recognition workshops, pp. 52 52-59, 2019.
Lucio M Dery, Yann Dauphin, and David Grangier. Auxiliary task update decomposition: The good, the bad and the neutral. arXiv preprint orXiv:210s. 2021a.
Lucio M Dery, Paul Michel, Ameet Talwalkar, and Graham Neubig. Should we be pre-training? an argument for end-task aware training as an alternative. arXiv preprint 0Xix:2109.07439 2021b.
Lucio M. Dery, Paul Michel, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. AANG Automating
auxiliary learning. In The Eleventh International Conference on Learning Representations, 2023. URL
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint Xii::1810004805, 2018
Alexey Dosovitskly, Lucas Beyer, Alexander Kolesníkov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dohghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint orXio:2010.11999 2020.
John Duchi and Hongscok Namkoong. Learning models with uniform performance via distributionally robust optimization. arXiv preprint arXim:1810.0877 181000880, 2018
Karan Goel, Albert Gu, Yixuan Li, and Christopher Re. Model patching: Closing the subgroup performance gap with data augmentation. In International Conference on Learning Representations, 2021. URL
14

---

Published in Transactions on Machine Learning Research (02/2024)
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 19338 19347, 2023.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.1090 2020.
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1929-1938, PMLR, 10-15 Jul 2018. URL https://procoooiings. wtt.ingssmmr...sss///. 180/hashimotol8a.F html
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Kaiming He, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and paltern recognition, pp. 15000016009) 2022.
Dan Hendryeks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International conference on machine learning, pp. 2712-2721. PMLR, 2019.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. arXiv preprint ar arXi::2004.011 061000 2020.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-accuracy In Bernhard Schölkopf, Caroline Uhler, and Kun Zhang (eds.), Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, pp. 336-351. PMLR, 11-13 Apr 2022. URL https://procceddings mlr. press/v177/idriasi222 html
Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson. On feature learning in the presence of spurions correlations. Advances in Newral Information Processing Systems, 35:38516 -33553,, 2022.
David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. Incorporating dialectal variability for socially equitable language identification. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 51-57, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10. N..1865//v1/P172200 URL https: //aclanthology... org.g//-or/P1/20099
Maurice Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81 93, 1938.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient. for robustness to spurious correlations. ar preprint Std:2224.029337 2022.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xic, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Trena Gao, Tony Lee, Elienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5637-5664. PMT.R, 18 24 Jul 2021. URL https://proccedingg
mlr ptppps/ve///oo/ie/a/eo http///a////o2aa.hmm
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, pp. 6565-6574 PMLR, 2021.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghumathan, Pang Wei Koh, Shiori Sagawa, Percy Liang; and Chelsea Finn. Just train twice: Improving group robusiness without training group information. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6781-6792. PMLR, 18-24 Jul 2021. URL
.lr.......pppp..... pttsspprrsvvv11111l1ss .html
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4487-4496, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1..11665111119110 URL htttps://cllntthooogs http:/ccaaatttoooooooo/.////1110
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXie preprint arXi:: 2017.
Taro Makino, Krzysztof Geras, and Kyunghyun Cho. Generative multitask lcarning mitigates target- causing confounding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 36546 36558. Curran Associates, Inc., 2022. URL https://proccedng...orre.e http://g....iii..c...eeeeee cc/paper httpe/aaprr__eeeelleeeeeeeeeeeee ftt.es/ppppr//022//2//.p
Chengzhi Mao, Amogh Gupta, Vikram Nitin, Baishakhi Ray, Shuran Song, Junfeng Yang, and Carl Vondrick. Multitask learning strengthens adversarial robustness. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 158-174. Springer, 2020.
Paul Michel, Sebastian Ruder, and Dani Yogatama. Balancing average and worst-case accuracy in multitask learning. arXiv preprint arXiv:21 10.05380 2021.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De- biasing classifier from biased classifier. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volune 33, pp. 20673 20684. Curran Associates, Inc., 2020. URL Pap.saa.3-pppppppp.. pdf
cctpspprrpppppppppppeeeeeeeeee files//aperr/2000
Junhyun Nam, Jaehyung Kim, Jaeho Lee, and Jinwoo Shin. Spread spurious attribute: Improving International Conference on Learning
worst-group accuracy with spurious altribute estimation. Representations, 2022. URL ltttp:://pperrevier http:ppprrrreeeeeeteooooorrrraaa net/forumidd= Ftt.rrrniid_FFFpprrr2
Bhargavi Paranjape, Pradeep Dasigi, Vivek Srikumar, Lake Zettlemoyer, and Hannaneh Hajishirzi. AGRO: Adversarial discovery of error-prone groups for robust optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevieer n
Shikai Qiu, Andres Potapczynski, Pavel Izmailov, and Andrew Gordon Wilson. Simple and fast group robustness by automatic feature reweighting. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28448- 28467. PMLR, 23 29 Jul 2023. URL https::/prroceednngss... .mlr. prress/v202/qiu22e3 html
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv:1706. 1.X.::1006.05098, 2017.
arXie preprint
Scbastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 conference of the North American chapler of the association for computational linguistics: Tutorials, pp. 15 18, 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020a. URL https://openrervier. met//forummidexxxxxuJJF...
16

---

Published in Transactions on Machine Learning Research (02/2024)
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In Hal Daumé III and Aarti Singh (eds.); Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of mlr.
Machine Learning Research, pp. 8346-8356. PMLR, 13-18 Jul 2020b. URL press/vv199saagaaa22a httml
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL http/s////prooeddnggg
Amrith Setlur, Don Dennis, Benjamin Eysenbach, Aditi Raghunathan, Chelsea Finn, Virginia Smith, and Sergey Levine. Bitrate-constraanct DRO: Beyond worst case robustness to unknown group shifts. In The Eleventh International Conference Learning Representations, 2023. URL //operrv//per..iee.... net/
forum? forum?1d=2Q2MuuRline
Yuge Shi, Jeffrey Seely, Philip Torr, Siddharth N, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In International Conference on Learning Representations, 2022. URL
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification probloms. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 19339-19352. Curran Associates, Inc., 2020. URL
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Yin Liu. MASS: Masked sequence sequence pre- training for language generation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5926 5936. PMLR, 09 15 Jun 2019. html
mlr, press/ veresa/v9/// v97/ v97/song19d www.:mlww.....................e.
Tejas Srinivasan and Yonatan Bisk. Worst of both worlds: Biases compound in pro-trained vision-and- language models. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp. 77-85, Seattle, Washington, July 2022. Association for Computational Linguistics. doi: 10.18653/v1 /1t///2222222222221
URL https:///cclanthologgy http://ooooooooo//2.geeo.ellooee
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL http:://oppprrreeeeeeeeeererrrrs 4nPswr1KcP
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1139-1147, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL |ttp:: mlr mwt.press.///p////mm.mm.e.. v28 sttsssusrss1.... html
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervisod vidco pre-training. Advances in neural information processing systems, 35: 10078 10093, 2022.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness to spurious correlations using pro-trained language models. Transactions of the Association for Computational Linguistics, 8:621-633, 2020. doi: 10.1162/taci 00335. URL https://adlattiooogg..ogg//// tac1-1.40
17

---

Published in Transactions on Machine Learning Research (02/2024)
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-uesd birds- 200-2011 dataset. 2011.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebufi, Ira. Ktena, Krishnamurthy Dj Dvijotham, and Ali Taylan Cemgil. A fine-grained analysis distribution shift. In International Conference Learning Representations, 2022. URL
Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Associalion for Compulational Länguistics: Human Language Technologgie, Volume 1 (Long Papers); pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 18653/v1/N18-1101. URL httpa://aclaattooooo.oorr////
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsca Finn. Gradient surgery for multi-task learning. Advances in Newral Information Processing Systems, 33:5824 5836, 2020.
Runtian Zhai, Chen Dan, J Zico Kolter, and Pradeep Kumar Ravikumar. Understanding why generalized reweighting does not improve over ERM. In The Eleventh International Conference on Learning Representations, 2023. URL https:://ppenreview set//forum?d=aasPee W8F-
Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Re. Correct-n- contrast: a contrastive approach for improving robustness to spurious correlations. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), ) Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 26484 26516. PMLR, 17 23 Jul 2022. URL bttps:s:s//proeeeiiii
mlr www.prreeevl.. hww.1//2/hanngzz...m html.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6): 1452 1464, 2018. doi: 10.100//PAMM:2211:27730000
18

---

Published in Transactions on Machine Learning Research (02/2024)
A Appendix
A.1 Training Details
For T prim, employ task-spocific classification head of single-layer multi-layer perceptron (MLP). For Taux: we leverage the pre-trained MLM and MIM heads from BERT and ViT, respectively. We utilize the embedding of the [CL.]] token from the base model through this MLP for classification. To facilitate effective multitask training, we adopt a uesroeeeero batching scheme Aghajanyan et al. (2021). This facilitates the accumulation of gradients across tasks prior to each parameter update, contributing to improvod training efficiency and convergence. Lastly, ensure proper scaling, the L1 loss normalized by the number of parameters in the shared representation.
A.2 Going Beyond the Pre-training Objective
Table 7: Impact of different pre-training objectives average and worst-group accuracy. Not all auxiliary lasks can help improve worst-group performance.
No Group Annotations
Val Group Annotations
Dataset
Method
Avg Acc
WG Ace
Avg Are
WG Acc
ERM
95. 95.30.2
80.14s
94.1a.
85.41.4
Waterbirds
+ MIM + L1
95.8o.s
83.33,4
95.44.4
87.52.7
+ SimCLR + L1
96.1g.3
84.05.4
95.5p.r
87.2
ERM
83.90.4
51.65.a
81.41.
67.42.1
Civilcommentt-Smalll
+ MLM
84.4g.4
53.74.3
82.00.5
69.41.7
+ CLM + L1
83.30.7
50.94.9
81.1g.9
67.31.4
Previous works in mult mlasking with self-supervised objectives suggest that different auxiliary objectives have disparate impacts on end task performance (Dery et al. 2023). Out of curiosity about the impact of the choice of auxiliary objective, we explore the impact of going beyond the model's original pre-training objective. For the Waterbirds dataset, we experiment with SimCLR a constrastive prediction task based on determining whether two distinct augmented images originate from the same base image (Chen et al. 2020). For BERT experiments on Civilcommente-small. we substitute the standard masked language modeling (MLM) task with causal language modeling (CLM) the auxiliary task. From the results in Table 7 we observe that SimCLR's performance closely resembles that of the MIM pre-training objective, whereas CLM shows relatively inferior results compared to MLM. We hypothesize that BERT's intrinsic bidirectional attention mechanism and non-autoregressive nature are not ideally suited for causal language modeling (Song et al. 2019 resulting in the model underperforming in our multitask setup. Given the model performance's sensitivity to the replacement. objective's 's choice, we proffer a practical rocommendation to practitioners: use the pre-training objective the auxiliary task. This aligns with recent work best practices for fine-tuning pre-trained models Gooaa et al. 2023
A.3 Bayes optimal model for comensson-independdee reconstruction under noised inputs
{(w) E [xx: 3
E (wx)) 2wrx;ū + (wxz)))
2
al(ws)
E [1r00e] w,E [1:2)]]
Dw,
19

---

Published in Transactions on Machine Learning Research (02/2024)
The optimal weighting wi is achieved when 80(ws)
0. Ow;
"$ [
E (
E (24 + ci)
E (o: =
E [xxx)
+
27 E
E [(x; + 2E [2r]] E
+ E (E
note E [c:] = 0 E [(xg) [ = of + 0.5 (1, (1/ + p9(0=-1)
of + 0.5 (nă 1 + Mijy=-1
of 0.5 ( 1 +
+ o
noise
B Broader Impact Statement
In terms of broader impact, our work has beneficial implications for group fairness and mitigating demographii--asse bias in machine learning systems. Specifically, we present a simple approach to improving worst-case group error. This means that our work contribules positively to certain groups that would otherwise be impacted by the poor performance of ML models. We have also provided new lens from which to view the problem of the impacts of multitasking, resulting in showing that it is tool able against poor group-based outcomes. However, of multiple objectives. This results in increased power consumption and, thus, greenhouse emissions during the training of models. Nevertheless, introducing our simple regularization modification to existing MTL implementations represents a negligible technical overhead than introducing JTT or BR-DRO to target worst-group error.
method introduces compute overhead due to the optimization
Table 8: Sensitivity analysis of all the methods to different hyperparameters wrt worst-group accuracy.
Dataset
Method
Seed
Learning Rate
Batch Size
Up
Aauz
Aveg
ERM
0.0734
0.3717
-0.4381
JTT
0.0573
-0.380
0.0246
0.1862
Waterbirds
ERM MT LI
0.0314
0.0142
0.0943
0.4114
gRoupDRO
-1688
0.196
-0.1384
FRM
0.0617
-0.8
0.1029
JTT
-0,0315
0.2104
-0.0047
0.1111
Sivllolmeents-Sall
ERM MT LI
0.0677
0.0453
0.0601
-0.2955
DRO
-0.092
0.7332
-0.1726
Sensitivity to Hyperparameters: For evaluating the sensitivity of worst group accuracy to various hyperparameters across different methods, we employ Kendall's rank coefficient т Kendall [938). To adhere to our computation budget, we opt for the optimal learning rate utilized in ERM for both JTT and ERM+MT+L1. Our sensitivity analysis from Table R reveals distinct preferences in learning rates. The ViT model applied to waterbirds, demonstrating preference for higher learning rates. Conversely, when trained on the civilommentts-small dataset, BERT exhibits a preference for lower learning rales. The epoch at upsampling is also sensitive hyperparameter for JTT. The sensitivity to different seed values is negligible across all the methods. Notably, our method displays the most minor sensitivity to changes batch size. However, it is noteworthy that our method most responsive to variations the regularization parameter (Areg); with lower values resulting in higher worst group accuracy.
20


Reviews:
[{"review_id": "gS9gbcGWik", "reviewer": "Reviewer_tsay", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is the first to demonstrate that the subgroup shift robustness can be improved by masked image/language modeling as the auxiliary task, as far as I know. \n\n2. In the synthetic data experiment, the contrast between Fig. 4 and Fig. 5 looks compelling. \n\n\nWeaknesses:\n\n1. Section 3.3 uses the weight norm as the measure for model complexity, while Section 4.2 uses the last layer feature norm, which means there is a gap between the argument in the synthetic data and in the real data. Resolving the inconsistency is important to understand the effect the the regularization term, which is quite essential as Fig. 5 shows. \n\n2. The real data experiment on image data is only done on a small synthetic dataset, it is unclear whether the method will scale to larger datasets like WILDS-iWildCam [2]. \n\n3. Some papers on multi-task learning for out-of-distribution generalization might be relevant to the paper, e.g., [1].\n\n\n[1] Albuquerque, Isabela, et al. \"Improving out-of-distribution generalization via multi-task self-supervised pretraining.\" arXiv preprint arXiv:2003.13525 (2020).\n\n[2] https://wilds.stanford.edu/datasets/", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to use masked image/language modeling as an auxiliary task to improve the worse-group performance in datasets with subgroup shifts. On a synthetic dataset, the auxiliary task is shown to be quite effective when the model is trained with a proper regularizer. On 4 real-world datasets, the method achieves better worse-group accuracy than JTT and BR-DRO when there is no group labels.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Elaborate the gap between the model complexity measure in synthetic and real-world data, and provide some thoughts on why the feature regularization is so important for MLM to work well. \n\nTest if the method is scalable to large-scale vision datasets.", "broader_impact_concerns": "Not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "42LBYEnxP6", "reviewer": "Reviewer_wKjf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a strong paper that delivers a simple but meaningful message in a very clear way. Toy problems are used skillfully to present a narrative that is easy to follow, and left me believing that the method should work, even before seeing the empirical results. The empirical results are also strong, and support the authors' claims. I appreciated the level of detail the authors devoted in their discussion of model selection, which I believe is crucial for a paper about about robustness.\n\nThe only weakness I found is the authors' lack of discussion on their choice of L1 norm regularization (as opposed to other forms, such as L2 norm or dropout). The choice of regularization seems important given how critical it is to their proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose that multitask learning applied to a discriminative task and an input-reconstruction task can improve worst-case and average-case accuracy for the discriminative task. There exist two types of features: core and spurious. Solely training on the discriminative task can result in the core features being ignored. When the input-reconstruction task is simultaneously optimized with strong regularization, this ensures the core features are also learned, thus improving robustness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would add some discussion about the choice of regularization, and why you went with L1 norm.\n\nI also suggest discussing https://arxiv.org/abs/2202.04136 in the MTL section of the related work. It is about improving robustness to shifts in p(y, y') in multitask learning, where y and y' are the targets. I believe it's related to this work, since yours is about improving robustness to shifts in p(y, s), where s is the spurious attribute in Section 2. Both can be framed as improving robustness to unobserved confounding that is particular to MTL.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jIjbINsglL", "reviewer": "Reviewer_DGj8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper addresses an important and timely problem of group fairness in machine learning, especially in the context of pre-trained models and MTL.\n\nThe paper demonstrates the effectiveness of the proposed method on three natural datasets across computer vision and natural language processing tasks and shows that it consistently outperforms several state-of-the-art DRO methods on both worst and average group outcomes, regardless of the availability of group annotations.\n\nThe paper did several ablation experiments on the effect of their method and pertaining. \n\nWeaknesses:\n\nThe paper lacks a structure, as of this version, the introduction consists of methods and results as well. \n\nThe paper does not provide a clear and rigorous definition of the problem and the proposed method.\n\nThe paper does not provide a clear explanation of why pre-training is critical for the success of the proposed method, and how it affects the trade-off between core and spurious features.\n\nThe paper does not compare the proposed method with other MTL methods that use different auxiliary tasks or different regularization techniques, which could provide more insights into the effectiveness of the proposed method.\n\nThe paper needs to address the potential limitations or drawbacks of the proposed method, such as the computational cost, the sensitivity to pre-task, hyperparameters, or the generalization to other domains or tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates how multitask learning (MTL) affects worst-group outcomes, which are the performance of a model on the most disadvantaged subgroups of data.\n\nThe paper proposes regularized MTL, which combines the end task with a pre-training objective and applies ℓ1 regularization to the shared representation space, as a simple and effective method to improve worst-group outcomes.\n\nThe paper provides both synthetic and natural data experiments to show that regularized MTL can induce the model to rely more on the core features and achieve better worst-group accuracy than distributionally robust optimization (DRO) methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper should be restructured and define the problem first and then introduce the method, results and supporting evidence. It should improve the readability and clarity of some sections, such as the problem formulation, the synthetic data experiments, and the appendix. For example, the paper could define the notation and the terminology more explicitly, and provide more explanations and examples for the synthetic data experiments. The paper could also reorganize the appendix and provide more details and proofs for some of the claims and results.\n\nThe paper should mention the contribution and novelty of the work explicitly in the introduction. \nThe paper should compare the proposed method with other MTL methods that use different auxiliary tasks or different regularization techniques. This would help to understand the advantages and disadvantages of the proposed method, and how it relates to existing MTL literature.", "broader_impact_concerns": "The paper does not have a Broader Impact Statement section. The paper should add a Broader Impact Statement section that discusses the potential ethical, social, or environmental implications of the work, both positive and negative.\n\nSome possible broader impact concerns of the work are:\n\nThe paper could have a positive impact on improving safety and group fairness and reducing discrimination in machine learning applications, especially for those that use pre-trained models and MTL. The paper could also inspire more research on understanding and mitigating the effects of spurious correlations and feature selection on group outcomes.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to improve the worst-case performance of a given model across groups by introducing a masked reconstruction loss. After building intuition in toy settings, the proposed approach is compared to various baselines on standard benchmarks. Reviewers generally found the approach convincing and appreciated the clarity and structure of the paper, though one reviewer made suggestions about the presentation that should be considered. Otherwise, the various reviewers' concerns were addressed during the rebuttal.
Decision: Accept (poster)