Do Current Multi-Task Optimization Methods in Deep Learning Even Help?
Authors: Derrick Xin
Behrooz Ghorbani
Justin Gilmer
Ankush Garg
Orhan Firat
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-Task Optimization
Multi-Task Neural Networks
Task Interference
Reproducible Research

Abstract: Recent research has proposed a series of specialized optimization algorithms for deep multi-task models. It is often claimed that these multi-task optimization (MTO) methods yield solutions that are superior to the ones found by simply optimizing a weighted average of the task losses. In this paper, we perform large-scale experiments on a variety of language and vision tasks to examine the empirical validity of these claims. We show that, despite the added design and computational complexity of these algorithms, MTO methods do not yield any performance improvements beyond what is achievable via traditional optimization approaches. We highlight alternative strategies that consistently yield improvements to the performance profile and point out common training pitfalls that might cause suboptimal results. Finally, we outline challenges in reliably evaluating the performance of MTO algorithms and discuss potential solutions.

Do Current Multi-Tasi Optimization Methods in Deep Learning Even Help?
Derrick Xin" Google Research Mountain View, CA dxin@google.co com
Behrooz Ghorbani" Google Research Mountain View, CA
Ankush Garg Google Research Mountain View, CA ankugarg@google.com
poorbani@googler com
Orhan Firat Google Research Mountain View, CA orhanf@google com
Justin Gilmer Google Research Mountain View, CA gi gilmer@google.com com
Abstract
Recent research has proposed a series of specialized optimization algorithms for deep multi-task models. It is often claimed that these multi-task optimization (MTO) methods yield solutions that are superior to the ones found by simply opti- mizing weighted average of the task losses. In this paper, experiments on a variety of language and vision tasks to examine the empirical validity of these claims. We show that, despite the added design and computational complexity of these algorithms, MTO methods do not yield any performance im- provements beyond what achievable via traditional optimization approaches. We highlight alternative strategies that consistently yield improvements to the perfor- mance profile and point out common training pitfalls that might cause suboptimal results. Finally, we outline challenges in reliably evaluating the performance of MTO algorithms and discuss potential solutions.
perform large-scale
Introduction
Multitask models are ubiquitous in deep learning I 17 these models can potentially leverage transfer learning in between different tasks and modalities. Moreover, by reducing the number of the models that need to be maintained, multitask models greatly simplify serving users.
This popularity stems from the fact that
Multitask models come with their own challenges and downsides. Different tasks often compete with each other for model capacity, leading the task interference problem. Finding the training setting that strikes the right balance between different tasks is an engineering intensive endeavor that requires extensive trial and error for most realistic setups.
Over the past few years, vast number of multi-task optimization (MTO) algorithms have been proposed in the literature that claim to alleviate the task interference problem 5 20 21 These algorithms typically leverage clever intuitions about the training process to dynamically balance the different tasks throughout training. However, in exchange for this, these algorithms often drastically add to the computational and design complexity of the training process.
26
30 31
The goal of this paper is not to provide another MTO algorithm. Instead, we provide a large- scale empirical study of the algorithms presented in the literature; examine what degree the
"qqual contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

improvements presented in the literature are reproducible and whether these algorithms really reduce loss interference between different tasks. As such, our study contributes to the growing body of literature that aims to provide reality check on recent algorithmic proposals in the machine learning community 12 23 28
We provide the following observations:
Despite the added complexity, MTO algorithms fail to improve the interference profile beyond what is achievable by simple static weighting of the tasks (Section +
The performance of multi-task models is sensitive to basic optimization parameters such as learning rate and weight-decay. Insufficient tuning of these hyper-parameter in the baselines, along with the complexity of evaluating multi-task models, can create false perception of performance improvement (Section
In some instances, the gains reported in the MTO literature are due flaws in the ex- perimental design. Often times these reported gains disappear with better tuning of the baseline hyperparameters. In addition, in a handful of cases, we were unable to reproduce the reported results (Section 4.2).
Finally, we discuss the implications for the community and the potential steps that need to be taken to standardize evaluation for multi-task models (Section (5).
2 Setting
We focus our discussion on the supervised learning setup; where the model parameters, O E IRP, trained on K different tasks. We denote the loss associated with task i with (0).
For some problem instances, the parameter space contains globally optimal point that achieves the best possible performance on all tasks. Figure T (left) provides a cartoon example of one such scenario. However, for most realistic setups, globally optimal 0 doesn't 't exist. In these cases, different tasks compete with each other for model capacity. In these scenarios, the concept of Pareto optimality is used to capture the optimal trade-off in between the tasks:
Definition (Pareto Optimality). O E IRP Pareto dominates another O' if Sisk, L.( S L,()') and there exists a lask j where (0) < (0)) O is Pureto optimal if it is not dominated by any other point. The collection of the Pareto optimal points is denoted Pareto front.
Figure [ (center) provides a cartoon representation of the Pareto front for a two-task setup. The Pareto front represents the collection of parameters that achieve the best possible trade-off profile between the tasks. A practitioner can aim to land on a particular point on this trade-off curve depending on their (implicit or explicit) utility function. The location and the curvature of the Pareto curve represent the severity of the interference problem. Ideally, one would like to identify training protocols that push the trade-off curve towards the origin as much as possible (Figure rightt.
1.0
0.8
1.0
Global Optimum
Pareto
Extreme Interference
Fezsible Points
Feasible Points
0.7
Medium Interference
0.8
0.9
0.8
Low interference
Lggs
0.6
0.8
0.5
Task:
0.7
0.2
0.6
0.2
0.1
0.0
0.7
0.6
0.5
1.5
0.0
0.5
0.6
Task
Task
Figure 1: A cartoon representation of the achievable trade-offs in a two-task setup.
The traditional approach to optimize multi-task models is via scalarization
K
((w) = arg min L(0; w) where L(0 w) Σ
W,C, (.). w > 0, Σ w; = 1. (1)
0

---

Here, w is fixed vector of task weights determined by the practitioner beforehand. The algorithmic and computational simplicity of this approach has made scalarization highly popular in practice.
Scalarization comes with certain theoretical guarantees. It can be easily shown that any solution to problem (T) is guaranteed be Pareto optimal. In addition, when {Li) i=1 convex, there exists partial converse:
Theorem (Informal). Let @# be point on the Pareto front. Then there exists w 0 such that scalarization with w yields 0*
These results suggest that, at least for convex setups, sweeping the task weights should be sufficient for full exploration of the Pareto frontier. particular, in the convex setting it is provable that no algorithm can outperform properly chosen scalarization that has been trained to convergence.
The results above raise series of questions. Where are the reported improvements of MTO algorithms coming from? Is non-convexity adding additional complexity which makes scalarization insufficient for tracing the Pareto front? Is the case that neural networks trained via combination of scalarization and standard first-order optimization methods are not able to reach the Pareto Frontier? Do MTO algorithms achieve better performance trade-off curve? In following sections, we empirically examine these questions for several popular deep learning workloads.
3
Prior Work
There has been flurry of research on MTO algorithms over the past few years. 13 4 argue
that finding the appropriate scalarization weights is often costly. To alleviate this, they provide algorithms that aim to automatically find a reasonable set of task weights. Sener & Koltun (2019) (26
approach multi-task learning from a multi-objective optimization view point and suggest Multiple Gradient Descent Algorithm (MGDA) for efficiently finding Pareto optimal solutions. [8] 2 30
31 hypothesize that negative interactions between the gradients of different tasks a significant contributor to the interference problem. As such, these studies put forward various suggestions for projecting out conflicting gradients in order to improve the optimization dynamics. Finally, [0) propose algorithms that inject randomization into the training pipeline and argue that this added randomness improves the training dynamics by allowing the optimization trajectory to escape poor local minima.
important to note that MTO algorithms often come with substantial computational overhead. Chen et al. (2020) report 2-5 fold increase in the training time for 40-task benchmark [5]. Similarly, Kurin et al. (2022) observe that on some benchmarks MTO algorithms can train as much as 35 times slower compared scalarization [66..
Recently, there has been a number of studies that critically question the benefits of MTO algorithms. The closest such study to ours is Kurin et al. (2022) [16] that appeared on Arxiv during the preparation of this manuscript. The paper argues that MTO algorithms implicitly regularize the model and shows that with careful regularization, scalarization with equal weights can match the performance of MTOs on various popular benchmarks.
In contrast, we argue that MTO algorithms yield different solutions on the same trade-off curve (See Figure 2 for an example). In most cases, these solutions tend to be different from equal weighting scalarization solution. When performance on popular benchmarks is concerned, we argue that scalarization baselines are often under-tuned. With additional tuning of the pareeppaaaettrs, we find that most optimizers yield comparable results.
4 Experiments
4.1 Multilingual Machine Translation
In this section, we examine the effect of MTO algorithms on multilingual neural machine translation (NMT). In particular, we focus translation out of English as prior work has reported significant task interference in this translation direction []]
The precise statements and their proofs provided in the appendix.
3

---

We start off by examining models trained jointly on (eglish->(Frennch) Chinese) translation tasks. The two-task setup allows us to effectively visualize the performance trade-off curves. French and Chinese are specifically chosen due to the large difference in their semantic and syntactic structures. Here, we anticipate large degree of interference among the tasks setting where MTOs claim to improve upon scalarization. We repeat our experiments for English-> (French, German] and Romanian translation tasks to ensure that our observations generalize across different task setups with different levels of data imbalance. See Table for an overview of data sources, All models use (pre-LN) Transformer architecture (29] and have been trained using early stopping. See Appendix A for training details.
Table I: Overview of data sources used in our NMT experiments.
Language Pair
Dataset
# Train Examples
# Eval Examples
English-Frende French
WMT15
40, 853, 298
4, 503
English- English-Chinese
WMT19
25, 986, 436
3, 9.98
Einlish-Germar
WMT16
4, 548, 885
2, 169
English-
English-Romanial Romanian
WMT16
610, 320
1, 999
We compare the performance trade-offs achieved by various popular MTO algorithms with the Pareto frontier of scalarization. Following the NMT literature's convention, we implement scalarization via proportional sampling. Here, the average number of observations in the batch corresponding task is proportional to w. In this setup, the expected training loss is equal to
K
K
(()) = Ex [x: 0)
Σ Pa E task im 0)( E task i) = Σ 1=1
w; (O).
We compare scalarization with a series of popular MTO algorithms: Multiple Gradient Descent (MGDA) [26] 18). GradNorm [4) Gradient Surgery (PCGrad) [31], IMTL [21], and Random Loss Weighting (RLW) (00 For GradNorm's a hyper-parameter, we perform a grid search and report all non-Pareto dominated models. To give an apples-to-apples comparison, all models have been trained with the same batch-size for the same number of training steps. All models use Adam [4] as the base optimizer. For all these optimizer categories, we tune the learning rate on grid from 5 x 10 to 5 and report all non-Pareks Pareto dominated models. Details of the training and presented in Appendix
are
The overview of our experimental findings are presented in Figures and The blue dashed line corresponds to the Pareto front achieved via proportional sampling with English->French >French sampling rate ranging from 10% to 90%. Several observations are in order:
No Improvements from MTO Algorithms Despite the promise alleviate interference among the tasks, all of the MTO algorithms in our study simply yield performance trade-off points on the scalarization Pareto front. As such, their performance can be fully replicated by simply optimizing a weighted average of the losses. To understand this phenomenon better, in Figure 3 we plot the evolution of the task weights for PCGrad, MGDA, GradNorm, and IMTL during training. We observe that for the majority of the training runs, the dynamically assigned task weights do not move significantly. As such, in effect, these MTO algorithms behave similar static weighting.
Other Language Pairs En and En-ZZh are both high-resource tasks with O(10²) training examples. In these experiments, we observe minimal overfitting and excellent agreement between train and test behaviors. One might argue that MTO algorithms possess transfer learning and regularization capabilities En (Zh, Fr) experiments downplay [ To address this, we repeat our experiments in two new task setups where En->Zh task is substituted with En En->De (mid-resource) and En-+Ro (low-resource).
Figures 4 & 5 present the results of these experiments. En Fr) experiments closely resemble En (Zh, Fr] ones: MTO algorithms simply achieve different trade-off points on the scalarization Pareto front The results for Eno, Fr) are more interesting. We still observe a clear Pareto
'See [6 for overview of proposals how MTO algorithms perform implicit regularization.

---

Gereralizetion Behavior Overview En (2h, Fr))
Optimication Behavior Overview En (2h, Fr)
Sampling
Sampling
1.53
IMTL--
Text Cross-Etropy Los (tn-FF
1.0
MGDA
Gradhiorm
1.50
MSDA
GradNorm
RLW (Normal)
RUW (Diricheet)
1.45
RLW (Dirchheet)
PCCrad
POGrad
Equally Weighted
Entropy
1.40
Equally Weighted
1.4-
1.30
1.3
1.20
2.8
1.8 Train Cross Entropy Loss (En
Entropy (En
Figure 2: Performance trade-off behavior for En-+ (Fr, Zh} models. Each point corresponds to the final performance of model. We observe no improvements terms of final performance or training behavior from MTO algorithms. See Appendix B for more comparisons.
[2h. F/)
En
1.0
MGDIA
1.0
195 Gredhormm
1.0
VGDD
Chinee 2h1 Teok Wegee
Gredhtrm
(De Tox
Grachlorm
0.8
PCGrod
INT.G
0.8
FCGrad
0.0
RCGrad
0.8
0.7
0.8
0.4
Nonanizn
0.2
0.4
<OOI
Training
Figure 3: The evolution of task weights during training. All models are trained with the same (near optimal) base learning rate of 0.5. For the majority of the runs, the task weights barely move.
front for the training performance; different MTO algorithms achieve different points on this curve. For the generalization behavior however, the Pareto frontier ceases to exist. Instead, we observe models that are globally optimal. Interestingly, these globally optimal solutions are found only by scalarization (with sampling rates close to (0.3, 0.7)). In contrast, MTO algorithms find solutions with near equal task weights which yield significantly worse generalization performances. As the generalization performance in this setup is primarily driven by the amount of regularization applied to the low-res resource task during training, results cast doubt the ability of MTO algorithms to effectively regularize the model.
Generalization
(De,
En Fr)
Sampling
1.50
Sampling
Test Crav--Entropy Loss (En -00
IMTL-G
IMTL-G
GradNorm
Crocs- Enerayy IEn-
1.45
GradNorm
(Norma))
RLW (Norma))
RLW (Dirichler)
1.40
(Diricreet)
PDGred
POGrad
Equal Weights
1.35
Weights
1.4
1.30
1.25
1.3
1.5
1.6
1.9
2.1
2.3
Test Drass-Untroon (En
Train Cross- (En-
Figure Performance trade-off behavior for (De, Fr) models. The results closely resemble our observations on En-+Zh, Fr) experiments.
5

---

Generalization Behevior Overview (Ro,
Optimization Behavior Overview En (Ro, Fi))
1.8
Samping
Sampling
Train Cress-Entropy Loss Loss Fi
1.7
IMTLL-
Text Cross-Enurapy loss (En
1.7
MGDA
GradNorm
1.6
MSDA
GradNorm
(Normal)
RLWY (Normal)
RLW (Dirichiet)
RLW (Dirccwett
PCCorad
1.5
POGrad
Equal Weight
Equal Weight
1.5
1.4
1.4
1.3
1.2
1.3
1.00 -intropyyyyyyy Ho)
Cross Enttspy Loss
Figure 5: Performance trade-off behavior for En-> (Ro, Fr) models. Left: We do not observe a Pareto frontier for the test performance. Instead, scalarization with weights (0.3, 0.7) achieves globally optimal trade-off and outperforms MTO algorithms. Right: For the training loss, a clear Pareto frontier appears and MTO algorithms simply selects point on the performance profile traced out by the scalarization sweep.
Evaluation Challenges Our experiments suggest that the model performances are highly sensitive the choice of hyper-parameters. Even subtle choices regarding the hyper-parameter grid can drastically change the results. For example, it is common practice to tune the learning rate on a sparse grid, say sweeping 1) [10-5 10-2 10-1) How much do our reported metrics suffer from such a sparse sweep, and how much performance can be gained on average from further tuning of just this one hyperparameter? To answer this question, we simulate running multiple instances of sparse grid searches of the form (k x 10-3, k x 10 10-2 k x 10 for S k S 99 each choice of k produces tuning study sweeping over learning rates. We then measure the variance in the optimal performance of each 3 trial study, as & is varied from 1 to 9. The results are shown in Figure O (left). For comparison, we plot the variance performance resulting from running the best 1) multiple times with different seeds. Notably, the effective standard deviation resulting from sparse learning rate tuning is 6 to times the standard deviation observed from varying the random seed for fixed hyperparameter point. The upshot is, estimating trial variance by rerumning multiple seeds is insufficient for concluding that performance gains from a new algorithm are significant when the hyperparameters sampled on sparse grid.
The established convention in the literature for ranking MTO performances is compare some kind of average of the per-task performances. The specific average used is fixed somewhat arbitrarily for the purposes of benchmarking. However, in practice, the utility function for ranking algorithms may vary dramatically depending on the goals of the practitioner. Thus, useful MTOs need to be robust to changes in the utility function. Either they need to improve upon the performance profile curve traced out by sampling sweep, or reliably find better points on the profile curve with minimal tuning as the utility function is varied. Unfortunately, the current practice to consider just one (arbitrary) weighting scheme will bias the evaluation towards algorithms that perform well on that specific scheme but are not robust to changes in the utility function. For example, Figure 6 (right) ranks MTOs as the evaluation per-task weighting is varied. As the En Bn weight is varied, the ranking shifts from MGDA being the best MTO to PCGrad being the best. This a natural consequence of Figure 2 which shows that different MTOs find different points on the same Pareto front traced out by a sampling sweep. Notably, no algorithm outperforms sampling with a well-chosen sampling ratio.
Alternative Approaches As discussed in Section 3 MTO algorithms often drastically increase the algorithmic and computational complexity of the traiinn process. In our experiments, we observed that the requirement to compute per-task gradients (which is necessary for many MTO algorithms) led significant reduction the number of training steps per second (from s 12 to zs 5). Given these observations, it is natural to wonder if there are more effective ways to spend this extra compute budget. Figure examines how scaling the model size changes the performance trade-off behavior. We examine increasing the model depth by a factor of 2,,2,3, 3, 4]. Our largest model achieves on average 5.4 training steps per second, which comparable with the models trained using per-task gradients. Our results suggest that, unlike the observed behavior with MTO algorithms, allocating

---

2.50
1.36
Weighte Test Loss (Normalized by EW)
0.10
Test Cross-Entapy Loss -ZM
2.48
Cross-Entton Entropy Lass (En
2.46
1.33
2.42
Test Crass
1.32
-0.05
MGD4
Gradhorm
2.40
1.31
Optimal Sampling
PCGred
Equal Weights (EW)
2.38
Random
Random Seed
Grid
0.5
En-Zh Weight the Average,
Figure 6: Left: Sparse sampling of learning rates has significantly larger effect reported performance than varying the random seed of particular hyperparameter point. Right: Rankings between algorithms eval performance is ranked according to .9 En-Fr En-Zh, then MGDA is preferred to GradNorm and PCGrad. At equal weighting, GradNorm and PCGrad outperform MGDA. For any weighting, all MTOs underperform an optimally chosen sampling scheme.
depend how tasks weighted evaluation time. For example, if the
more compute scale the model yields consistent improvements the board. Larger models achieve Pareto fronts strictly to the lower-left of the base model, which corresponds to a performance improvement for all utility functions.
Table 2: Overview of models used in the scaling experiment.
Model
Optimizer
# Parameters
# Steps/Sec
3L-3L (base)
Scalarization
120M
11.52
3L-3L
MGDA
120M
4.81
61.-6L
Scalarization
142M
8.28
91--9I
Scalarization
165M
6.68
12L-12L
Scalarization
187M
5.48
4.2 Benchmarks from the Literature
The observations of Section 4.1 run contrary to many recent influential studies proposing MTOs for multi-task models 4 performance of their proposed algorithm with traditional training strategies and report significant gains. In this sec-
5.
9 26 31. These papers often compare the
Jalaer
Bolayer
124 Modil
Tee: Cross www.e-tttttteeeeeeeee Losn
tion, we attempt to reproduce these results on number of supervised-leaningg benchmarks. We present comparisons for Scapes and CelebA datasets the main text
2.4
For these experiments, we closely follow the experimental setup and the publicly available code from [26]. We modi- fied the code sparingly to address bugs, update deprecated libraries, and speed up the data loader. We perform an extensive grid search for learning rate, weight decay, and dropout. All models use early stopping, Our implementa- tion details are presented the appendix.
7.0
2.4 Teed Cross-remmyy
77
2.
2.8
3.0
Figure 7: The effect of model size on the Pareto frontier for En > /Zh, Fr) models.
"See Appendix C for comparisons and details.

---

4.3 CityScapes
CityScapes [6] a dataset for understanding urban street scenes. is constructed via stereo video sequences from different cities and contains 2975 training and 500 validation images. In the multi- task optimization literature, this dataset is popularly cast as a two-task problem with one task being 7-class semantic segmentation and the other being depth estimation. In experiments, 595 random samples from the training data to serve as our validation set. This validation set used for tuning hyper-parameters such learning rate and weight decay (See appendix for details). We use the original validation set as our test set.
choose
Figure [ provides an overview of our experimental results. Similar to Section 4.1 we observe that scalarization solutions form the generalization performance Pareto front. This frontier observable both for test loss (left) and task specific generalization metrics (right). In both cases, MTO solutions significantly mnder-perform scalarization.
Scallriization GradNorm
Text Semantt Segmestatinn LOKS
0.28
MCDA
Test Pizel Accuracy [Segmentation))
RLW (Dirchllet
0.27
PCGrad
0.915
0.26
0.912
0.909
Scalarization
MGDA
0.906
RL (Normal)
(Dirichlet)
PCGrad
0.0150
0.0138
Depth Loss
Error (Depth)
Figure 8: The generalization performance of different optimizers for CityScapes benchmark. Left: Test segmentation loss test depth loss. Points the lower left side represent better solutions. Right: Test pixel accuracy vs test absolute depth estimation error. Here, points on the upper left side better solution. In both cases, scalarization solutions form the Pareto frontier.
For CityScapes models, the segmentation task loss is an order of magnitude larger than the depth estimation task loss. This severe loss imbalance causes interesting behaviors to emerge that worth noting.
the train / test behavior of the different scalarization solutions. Contrary to recent results reported in the literature (16), observe that appropriately balancing the different losses is crucial achieving a desirable generalization behavior: Figure ? the majority of the generalization Pareto frontier is populated by models with segmentation task weight less than 0.2.
Train Train Semardic Sopmentatin Locs
0.12
0.28
Sogm Weight
Semantte Sagnn wet.eetttiionnn
0.27
0.28
0.10
0.25
0.08
0.0130
0.01.55
0.01.0 0.0145
0.0155
0.0155
0.0075
0.0080
0.0085
0095
0.0100
0.0105
103
Depth
Depth
Figure 9: Scalarization (left) and train (right) Pareto frontiers for CityScapes dataset. We plot all non-dominated experimental runs per scalarization weight mixture.

---

4.3.1 CelebA
CelebA dataset [2 is a collection of 200K face images annotated with 40 attributes. This dataset is popular benchmark for MTO research where each attribute treated as a separate binary classification task. In Section 4.1 we identified a number of evaluation challenges for MTO algorithms, namely the significance exact hyper-parameter tuning and the difficulty of comparing models via average performances. These evaluation challenges become highly visible for CelebA.
Figure preseents the overview our results. We report average performance across the tasks. Our results suggest that scalarization performance is comparable with the performance of popular MTO algorithms. This is in line with recent findings in the literature []]. More importantly. Figure I0 shows the importance of careful there is significant variation in the final performance of the models that is drastically larger than the effect of the MTO algorithm choice.
tuning: Even in the presence of early-stopping.
Distribution of the Average Error Rate (Test)
Distribution of the Average Error Rate (Valdbttion
PCGrad:
PCGred: Weights:
Equa Weights:
11.0
Average Valtssron Emror Rote
RLW (Noomma): 9:30
RLW (Normmll: 70 RLW (Drichlett: 8.54
Avcraqe Erorfat
20.5
RLW (Dirchiet:: 9.27
10.0
9.5
9.0
PCGrdd
Equii Weights
MGDA
REW (Normal) R. M (Deichlet)
PCGred
Equal Weights
MGDDA
RLW (hormal) RUM (Diichlet)
Figure 10: For CelebA, the effect of tuning hyper-paramene of the MTO algorithm choice. Each point here corresponds to the performance of an early-stopped model. We vary the learning rate from 10 4 to 5 10 and weight decay from 5 x 10-3
is much significant than the effect
Average Emror as Function of Learning Rate
10.6
PCGrad: 9:45
Equal Weights: 9.51
10.4
MGDA; 9.76
10.2
RLW (Normal): 9.49
RLW (Dirichlet): 9.27
10.0
Average Test
9.6
9.2
10-1
10
10
10"
Average test error function of learning rate for each MTO on CelebA. The performance ranking of the different MTOs is highly dependent on the learning rate. Due to this high variability. studies with sparse learning rate grids can yield misleading conclusions.
Figure 11:
This acute hyperparameter sensitivity can lead to a scenario where insufficient tuning of baseline hyper-parameters gives illusions of significant performance gains. We suspect such evaluation challenges play prominent role in the significant disagreements we observe in the literature on the effect and ranking of MTO algorithms. Table 3 presents an overview of the results presented in the literature. As the table suggests different papers report wildly different performance for the same algorithm. A large fraction of the reported statistics resemble the quantities we observe on the validation dataset. This concerning as validation performance on CelebA tends to be noisy. If only the validation performance is reported, a combination of early stopping and high evaluation frequency can artificially boost the scores. This artificial boost in scores is clearly visible between the left and the right sides of Figure 10

---

Table 3: An overview of reported results in the literature for CelebA benchmark. There is significant disagreement between reported statistics from different studies. We suspect improper tuning of baseline hyper-parametenn is likely culprit (compare reported statistics with Figure 10
Study 8.71
26
31
21
16
Ours (Test)
Ours (Validation)
Scalarization
9.62
9.99
9.1
9.23
8.73
MGDA
10.82
8.25
8.95
9.96
9.78
9.53
9.24
GradNorm
8.68
8.44
10.08
PCGrad
8.72
8.69
10.01
9.07
9.35
8.89
GradDrop
8.52
9.02
Conclusions
In this paper, presented large-scale empirical study examining the effects of multi-task opti- mization methods. often assumed that these algorithms enhance the optimization dynamics of multi-task models and yield desirable solutions that cannot be achieved via scalarization. Our results suggest the contrary. Across a variety of language and vision tasks, we showed that scalarization, with appropriate weights, can match both the optimization and the generalization behaviors of MTO algorithms. As such, in effect, scalarization solutions form a superset for MTO solutions. Our experimental results suggest effective exploration of the scalarization solution set might be a more reliable and effective strategy for boosting the model performance (See Fig
Our observations suggest the final performance of multi-task models is highly sensitive the choice of training hyper-pare ppper-parameters.. Often times, the effect size associated with subtle design decisions in the choice of the paper-parameter grid is orders of magnitude larger than the MTO effect size (See Fig 1O). As such, researchers can unknowingly create the illusion of significant performance gains by simply under-tuning the competing baselines. The fact that different studies are reporting drastically different numbers for the same dataset-algoriihm pair (Table suggests this phenomenon prevalent in this literature.
Limitations and Future Research Our results suggest that by exploring the scalarization solution space, one can attain performance on par with (or better than) many MTO algorithms. However, the grid search approach we used for computing the scalarization Pareto frontier is computationally prohibitive. Examining strategies for efficiently searching this solution space (such as [10 15) is fruitful future research direction.
In the paper, pointed out worrying concerns regarding faulty evaluation and under-tuned baselines. natural solution to alleviate these problems is to adopt the Common Task Framework (CTF) 8 19
to reliably identify and measure algorithmic improvements in multi-task optimization. With the creation of a commonly used competitive benchmark with proper validation test split, baselines will naturally become stronger subsequent papers progressively improve performance this makes substantial gains more convincing than current practice where baselines rerun by the authors themselves. We postpone the development of such pipeline future work.
Finally, to keep the discussion tractable, we focused our analysis to supervised learning benchmarks. Whether the same behavior holds for reinforeement learning and self-supervised learning setups is still an open question.
Acknowledgments and Disclosure of Funding
We thank George F. Dahl, Wolfgang Macherey, and Macduff Hughes for their constructive comments on the initial version of this manuscript. Additionally, we thank Sourabh Medapati and Zachary Nado for their help in debugging our code base. Moreover, are grateful to Soham Ghosh and Mojtaba Seyedhosseini for valuable discussions regarding the role of MTOs in large-scale models.
10

---

References
[1] Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neu- ral machine translation in the wild: Findings and challenges. arXiv preprint 2019.
[2] Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conncau. mslam: Massively multilingual joint pre-training for speech and text. arXiv preprint arXiv:2202.01374, 2022.
[3] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[4] Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pages 794-803, PMLR, 2018.
[5] Zhao Chen, Jiquan Ngiam, Yanping Huang. Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems,
2020.
[6] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke. Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[7] Jean-Antoine Désidéri. muuttple-gradiens descent algorithm (mgda) for multiobjective opti- mization. Comptes Rendus Mathematique, 350(5-6))313-133 2012.
[8] David Donoho. 50 years of data science. Journal of Computational and Graphical Statistics, 26(4):745-766, 2017.
[9] Chris Fifty, Ehsan Amid, The Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34, 2021.
[10] Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Au- tomated currieulum learning for neural networks. In international conference on machine learning, pages 1-11-1320. PMLR, 2017.
[11] Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10):222-22322, 2016.
[12] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
[13] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1482-7491, 2018
[14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[15] Julia Kreutzer, David Vilar, and Artem Sokolov. Bandits don't 't follow rules: Balancing multi- facet machine translation with multi-armed bandits. arXiv preprint arXiv:2 2021.
[16] Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, and M Pawan Ku- mar. In defense of the unitary scalarization for deep multi-task learning. arXiv preprint MrXiv:2201.04222 2022.
[17] Dmitry Lepikhin. HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint 166668,000.1666, 2020.
11

---

[18] Xian Li and Hongyu Gong. Robust optimization for multilingual translation with imbalanced data. Advances in Neural Information Processing Systems. 34, 2021.
[19] Mark Liberman. Obituary: Fred jelinek. Computational Linguistics, 36(4):595-599, 2010.
[20]
Baijiong Lin, Feiyang Ye, and Yu Zhang. A closer look at loss weighting in multi-task learning. arXiv preprint arXiv:2111.10603, 2021.
[21] Liyang Liu, Yi Li, Zhanghui Kuang. J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. ICLR, 2021.
[22] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
[23] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Unsupervised domain adaptation: A reality check. arXiv preprint arXiv:2111.15677 2021.
[24] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[25] Matt Post. A call for clarity in reporting bleu scores. arXiv preprint 1rXiv 2018.
[26] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 225-33,,5 2018
[27] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-195 2014.
[28] Xin Su, Yiyun Zhao, and Steven Bethard. A comparison of strategies for source-free domain adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8352-8367, 2022.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[30] Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. arXiv preprint arXiv:2010.05874, 2020.
[31] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, Advances in Neural Information Processing Systems, 33:5824-5836, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Sections 4.1
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [N/A]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Appendix
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
12

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URLY? [Yes] See appendix for implementation details. The code will be made public after the review period
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Figure
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Compute details are provided in the appendix
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] See Section 4.3.1
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "qskFxMfCQvH", "reviewer": "Reviewer_i2dN", "summary": "This paper performs large-scale experiments to show the recent multi-task optimization methods do not improve the performance over the simple static scalarization of different tasks. The paper also shows that the performance of multi-task learning is sensitive to the chosen hyperparameters. And suboptimal hyperparameters sometimes make some MTO method stand out from others while this trend does not hold after using better hyperparameters They also suggest the potential solution as using the common task framework to make a fair comparison for multi-task learning in the future work.", "questions": "1. What is the computation cost of search a good scalarization among tasks? Is there any method to speed up the searching for the optimal loss weights when there are more than 2 tasks.\n\n2. For the scalarization method, does the same group of hyperparameters works optimally for different scalarization of tasks? ", "limitations": "Yes, they have addressed", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper summarizes the previous SOTA MTO methods and make a fair comparison between them and simple scalarization method on NLP translation task.\n\n2. The comparison of SoTA MTO methods and simple scalarization is valuable since it shows with optimal scalarization, the static weighting of different tasks give competitive (or even better) performance than complex MTO methods.\n\n3. Also the analysis of variation of MTL performance with different hyperparameters is very interesting and insightful. \n\nWeaknesses:\n\n1. Even though the optimal scalarization gives as good performance as other MTO methods, authors seem does not count in the computation cost to find the optimal weighting for different tasks, especially when there are more than 2 tasks.\n\n2. Since the hyperparameter is very important to the MTL performance, it is unclear whether the same group of hyperparameters is optimal for all different loss weights among tasks. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "m8TY_I6F86G", "reviewer": "Reviewer_FSsD", "summary": "Here, the authors present an extensive empirical evaluation of multi-task optimization (MTO) methods across Multilingual Machine Translation, Multi-MNIST and CelebA datasets. The results showed that the MTO methods tested did not offer performance improvements over scalarization as a method of optimizing MTL models. The authors also provided a comprehensive evaluation of the impact of hyperparameters. They pointed out that this and the different evaluation strategies employed by previous papers might lead to optimistic results in favor of MTO methods.", "questions": "Could the authors make a bit clear the limitations of this work in section five?\n", "limitations": "As per the checklist, the authors did mention limitations in Section 5. I believe the limitation is stated in the second paragraph of this section. Would it be possible for the author to clarify this? There is an abrupt transition from one paragraph to another here, and I read the paragraph twice to note the limitation.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1) This work provides a comprehensive benchmarking of MTO methods using language and vision datasets. In light of the recent trend of optimization-based approaches to multi-task learning, this evaluation is of importance in the field of multi-task learning. \n\n2) The authors presented a discussion and analysis of why previous MTO methods present optimistic results, likely due to poor fine-tuning and evaluation procedures.\n\n3)  MTO methods might present a substantial computational overhead. The authors provided a careful analysis suggesting that time/investment in simpler and well-tuned baselines might be beneficial in the end.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JhuoJkWePk", "reviewer": "Reviewer_s8zi", "summary": "This paper brings into question the efficacy of multi-task optimization methods, finding that when adequately tuning the baselines, MTO methods yield solutions similar to optimizing a weighted average of each tasks loss. They show that sensitivity to training hyper parameters lead to an often unfair comparison across methods and drastically different results on the same benchmarks, and provide solutions to standardize the evaluation of these methods.", "questions": "In 4.1 the authors mention that for the English to {French, Chinese} task they \"anticipate a large degree of interference among the tasks\"; what exactly does this mean? \n\nIn 4.1, I did not understand this baseline: \"The blue dashed line corresponds to the Pareto front achieved via proportional sampling with English→French sampling rate ranging from 10% to 90%.\"Why do you only sample English to French if the task is English to French and Chinese? Am I misunderstanding the evaluation?\n\nIs RLW an MTO algorithm or the baseline of taking the weighted average of the losses?\n\nWhat dataset is used in 4.1?\n\nIs test cross-entropy the usual evaluation metric used for NMT? What about things like BLEU score?", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The issue of insufficient hyperparameter tuning leading to the illusion of performance gains is incredibly important and often overlooked, and this paper does an excellent job of highlighting this fact. The claims of the paper were clearly laid out and the evaluation of sensitivity to hyperparameters was thorough.  \n\nHowever, I have two main concerns. The first is the organization of Section 4. I don't fully understand what the experimental setup is, the methods evaluated, and the results laid out specifically for the neural machine translation were not easy to follow. I realize that there is limited space so many experimental details can't be put in the main paper, but more summarization of the task setup and evaluation metrics would clear things up a lot. Also more of a suggestion than critique, but I wonder if employing a hyperparameter stability evaluation similar to [5] would be useful in illustrating these methods sensitivity to hyperparameters. \n\nSecond, the issue of hyper parameters not being swept correctly when evaluating methods has been heavily explored in the past few years [1-4], and the issue of reproducibility of results is also a widely accepted issue in the field. Ultimately I believe this paper provides novel contributions in its evaluation and specific focus on MTO methods, but these previous works should be mentioned in the related works. \n\nWith all of this in mind I believe that the ideas in this paper are of interest to the NeurlIPS community, although the presentation of the results needs clarification and the related works mentioned should be added. Overall a very interesting read :D\n\n[1] Su et al. \"A Comparison of Strategies for Source-Free Domain Adaptation\"\n\n[2] Musgrave et al. \"Unsupervised Domain Adaptation: A Reality Check\"\n\n[3] Gulrajani et al. \"In Search of Lost Domain Generalization\"\n\n[4] Greff et al. \"LSTM: A Search Space Odyssey\"\n\n[5] Xiao et al. \"Early Convolutions Help Transformers See Better\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept