In Defense of the Unitary Scalarization for Deep Multi-Task Learning
Authors: Vitaly Kurin
Alessandro De Palma
Ilya Kostrikov
Shimon Whiteson
M. Pawan Kumar
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Multi-Task Learning
Optimization for Deep Learning
Deep Reinforcement Learning

Abstract: Recent multi-task learning research argues against unitary scalarization, where training simply minimizes the sum of the task losses. Several ad-hoc multi-task optimization algorithms have instead been proposed, inspired by various hypotheses about what makes multi-task settings difficult.  The majority of these optimizers require per-task gradients, and introduce significant memory, runtime, and implementation overhead. We show that unitary scalarization, coupled with standard regularization and stabilization techniques from single-task learning, matches or improves upon the performance of complex multi-task optimizers in popular supervised and reinforcement learning settings. We then present an analysis suggesting that many specialized multi-task optimizers can be partly interpreted as forms of regularization, potentially explaining our surprising results. We believe our results call for a critical reevaluation of recent research in the area.

In Defense of the Unitary Scalarization for Deep Multi-Task Learning
Vitaly Kurin" University of Oxford vitaly. kurinOcs. ox. ac.
Alessandro De Palma" University of Oxford adepalma@robots. ox, ac
Ilya Kostrikov
Shimon Whiteson
M. Pawan Kumar University of Oxford
University of California, Berkeley New York University
University of Oxford
Abstract
Recent multi-task learning research argues against unitary scalarization, where training simply minimizes the sum of the task losses. Several ad-hoc multi-task op- timization algorithms have instead been proposed, inspired by various hypotheses about what makes multi-task settings difficult. The majority of these optimizers require per-task gradients, and introduce significant memory, runtime, and imple- mentation overhead. We show that unitary scalarization, coupled with standard regularization and stabilization techniques from single-task learning, matches or improves upon the performance of complex multi-task optimizers in popular super- vised and reinforcement learning settings. We then present an analysis suggesting that many specialized multi-task optimizers can be partly interpreted forms of regularization, potentially explaining our surprising results. We believe our results call for critical reevaluation of recent research in the area.
Introduction
Multi-Task Learning (MTL) exploits similarities between tasks to yield models that are more accurate, generalize better and require less training data. Owing to the success MTL on traditional machine learning models 16 22 and of deep single-task learning across variety of domains, growing body of research has focused on deep MTL. The most straightforward way to train a neural network for multiple tasks at once is minimize the sum of per-task losses. Adopting terminology from multi-objective optimization, we call this approach unitary scalarization.
While some work shows that multi-task networks trained via unitary scalarization exhibit superior performance independent per-task models 29 35), others suggest the opposite 30
54 58).
As result, many explanations for the difficulty of MTL have been proposed, each motivating new Specialized Multi-Task, Optimizer (SMTO) the proposed SMTO outperforms unitary scalarization, in addition to relevant prior work. However, SMTOs usually require access per-task gradients either with respect to the shared parameters, or to the shared representation. Therefore, their reported performance gain comes significant computation and memory cost, the overhead scaling linearly with the number of tasks. By contrast, unitary scalarization requires only the average of the gradients across tasks, which can be computed via a single backpropagation.
66 These works typically claim that
Existing SMTOs were introduced to solve challenges related to the optimization of the deep MTL problem. We instead postulate that the reported weakness of unitary scalarization is linked to experimental variability or to lack of regularization, leading to the following contributions:
"Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A comprehensive experimental evaluation ( of recent SMTOs on popular multi-task bench- marks, showing that no SMTO consistently outperforms unitary scalarization in spite of the added complexity and overhead. In particular, either the differences between unitary scalarization and SMTOs are not statistically significant, or they can be bridged by standard regularization and stabilization techniques from the single-task literature. Our reinforcement learning (RL. experiments include optimizers previously applied only to supervised learning.
An empirical and technical analysis of the considered SMTOs, suggesting that they reduce overfitting on the multi-task problem and hence act as regularizers (55). We conduct an ablation
study and provide collection of novel and existing technical results that support this hypothesis.
Code to reproduce the experiments, including a unified PyTorch 50 implemen- tation of the considered SMTOs, is available at htttps///gttbbb...o//bbbbbyy umitaay--ccalariztiion
We believe that our results suggest that the considered SMTOs can be often replaced by less expensive techniques. We hope that these surprising results stimulate the search for a deeper understanding of MTL.
2 Related Work
Before diving into details of specific SMTOs in Section we provide a high-level overview of the deep MTL research. Seminal work in MTL includes hard parameter sharing [6]: sharing neural network parameters between all tasks with, possibly, a separate part of the model for each task. Hard parameter sharing is still the major MTL approach adopted in natural language processing ( [2]. computer vision [46]. and speech recognition (53) In this work, we implicitly assume that each parameter update employs information from all tasks. However, not all works satisfy this assumption, either due to a large number of tasks 4 [36], or simply as an implementation decision [25] 37 In this setting, MTL resembles other problems dealing with multiple tasks, i.e., continual 32 curriculum [47]. and meta-leamning [24]. which are not the focus of this work.
Many works strive to improve the performance of deep multi-task models. One line of research hypothesizes that conflicting per-task gradient directions lead to suboptimal models, and focuses on explicitly removing such conflicts [1] 28 41 42 62, 66). Some authors postulate that loss imbalances across tasks hinder learning, proposing loss reweighting methods [10 30 40). Sener and Koltun (54) and Navon et al. (48) propose that tasks compete for model capacity and interpret MTL as multi-objective optimization in order to cope with inter-task competition. Here, we focus on algorithms that explicitly rely per-task gradients try to outperform unitary scalarization (()) Research on multi-task architectures [19 46) or MTL algorithms exclusively motivated by determin- istie loss reweighting [18
30 43 orthogonal to our work. Both topics are investigated by recent survey on pixel-level multi-task computer vision problems (61. which found that the minimization of tuned weighted sums of losses (scalarizations) is empirically competitive with deterministic loss reweighting and MGDA in the considered settings. These results are extended to popular SMTOs by a critical review from Xin et al. (63). concurrent our work, which argues that the optimization and generalization performance of SMTOs can be matched by tuning scalarization coefficients. Our work reaches a similar conclusion, demonstrating that unitary scalarization performs on par with SMTOs when coupled with standard and inexpensive regularization or stabilization techniques. In other words, Xin et al. [63] provide complementary support for the link between SMTOs and regularization by showing that tuning scalarization weights positively affects generalization.
In addition the common supervised settings, we also consider multi-task RL, whose research can be grouped into three categories: the first adds auxiliary tasks providing additional inductive biases to speed up learning (27] on target task. The second, based on policy distillation, uses per-task teacher models to provide labels for a multi-task model or per-task policies as regularizers [49 [ (5]. The third directly learns shared policy [29], possibly via an SMTO 666. We focus on the third category, whose literature reports varying performance for unitary scalarization (better (29] or worse [66] than per-task models), indicating confounding factors evaluation pipelines and further motivating our work. PopArt [23] 60) performs scale- invariant value function updates in order to address differences in returns across environments, showing improvements in the multi-task setting while still using unitary scalarization. PopArt does not require per-task gradients but introduces additional hyperpa- rameters. our work, we address the differences in rewards by normalizing them at the replay buffer level. However, we believe both unitary scalarization and SMTOs might equally benefit from PopArt.

---

3 Multi-Task Learning Optimizers
We will now describe the deep MTL training problem and popular algorithms employed for its solution. Let (X, Y) E _dx= x ooxn be the training set, composed of n d-dimensional points and o-dimensional labels. In addition, L, R x R - R denotes the loss for the i-th task, e RS the parameter space, T := (I...., m the set of m tasks. The goal of MTL is to learn a single (generally task-aware) parametrized model ʃ IR$ x RdxR x T + Roxa that performs well on all tasks T. The parameter space is often split into a set of shared parameters across tasks (generally the majority of the architecture), denoted fl, and (possibly empty) task-specific parameters, denoted fT. so that 0 : 001, 01]² In this context, the model ff often takes on an encoder-decoder architecture, where the encoder g learns a shared representation across tasks, and the decoders hú are task-specific (g 101, X), O1). In this case, we denote by z g(0 X) E IR"xx the r-dimensional shared representation of X.
predictive heads: f(0, X,
The training problem for MTL is typically formulated as the sum of the per-task losses
11
54
66
min 0
C (0) = Eier (,) |.
(1)
Unitary Scalarization The obvious way to minimize the multi-task training objective in equation (I) is to rely standard gradient-basee algorithm. While, for simplicity, focus standard gradient descent rather than mini-batch stochastic gradient descent, the notation can be adapted by replacing the dataset size by the mini-batch size b. Equation ( corresponds with unitary weights under a multi-objective interpretation of MTL; hence, we call the direct application of gradient descent on equation (T) unitary scalarization. For vanilla gradient descent, this corresponds to taking a step in the opposite direction as the one given by the sum of per-task gradients: VoCMi = >EeT Vo Per-task gradients are not required, as it suffices to directly compute the gradient of the sum LMT, Hence, when relying on deep learning frameworks based on everse-mode differentiation, such as PyTorch [50] the backward pass is performed once per iteration (rather than m times). Furthermore, the memory cost is a factor m less than most SMTOS, which require access to each VoL, As consequence, unitary scalarization is simple, fast, and memory efficient. Our experiments demonstrate that, when possibly coupled with single-task regularization such as early stopping, & penalty or dropout layers [56), this simple optimizer is strongly competitive with SMTOs.
linear scalarization
MGDA Sener and Koltun 54 point out that equation 1 can be cast as a multi multi-objective opti- mization problem with the following objective: CMT (0) := (0), Lm (0)l². A commonly employed solution concept in multi-objective optimization is Pareto optimality. A point 0' is called Pareto-optimal if, for any another point Ot such that 3i TT: (i)))) < (0'),, then 3j T (0)) > (0*). A necessary condition for Pareto optimality at a point is Pareto stationarity, defined as the lack of a shared descent direction across all losses at that point. Sener and Koltun [54) rely on Multiple-Gradinn Gradient Descent Algorithm (MGDA) [4] to reach Pareto-stationary point for shared parameters On. Intuitively, MGDA proceeds by repeatedly stepping in a shared descent direction [4 17]. which can be found by solving the following optimization problem: min g.c
+ 1/2 ||
s.t. Ve, CT ViET,
whose dual takes the following form (corresponding the formulation from Désidéri (14):
max
- 1/2 1/ a00
s.t. Lo.Vo = g, Σ 0 =1
In other words, MGDA takes a step in a direction g given by the negative convex combination of per-task gradients, whose coefficients are given by solving equation (3) In practice, per-task gradients are rescaled before applying MGDA: the original authors' implementation (54) relies on Ve, C t Va, f://://Vo, /. (.(0). The convergence of MGDA to a Pareto-stationary point is still guaranteed after normalization [14].
IMTL Impartial Multi-Task Learning (IMTL) 42 is presented as an SMTO that is not biased
against any single task. It is composed of two complementary algorithmic blocks: IMTL-L, acting on task losses, and IMTL-G, acting on per-task gradients. IMTL-G follows the intuition that a multi-task optimizer should proceed along direction g =- - a, Vo, Cs that equally represents per-task
3

---

gradients. This is formulated analytically by requiring that the cosine similarity between and each Vo,Li C, be the same. To prevent the resulting problem from being underdetermined, Liu et 42]
add the constraint DiET 04 = resulting in a problem that admits closed-form solution for g:
Vo L1 g
g Van Ci c.
ViET g a. ΣiET a: ai = 1.
(4)
Vo 6
IMTL-L, instead, aims to reweight task losses that they are all constant over time, and equal to 1. In order to limit oscillations of the scaling factors, the authors propose to learn them jointly with the network by minimizing common objective via gradient descent. In particular, given St E Vi E T. Liu et al. [42] derive the following form for the joint minimization problem: ming.0 |E; (e (0) s:)] As proved by Liu et al. [42], IMTL-L only has rescaling effect on the update direction of IMTL-G. Unlike IMTL-G and the other SMTOs presented in this section, IMTL-L rescaling designed to affect the updates for task-specific parameters 1 as well.
PCGrad Let us write cos(x, z) for the cosine similarity between vectors x and z. Yu et al.
66
postulate that multi-task convergence is severely slowed down if the following three conditions (named the tragic triad) hold at once: (i) conflicting gradient directions: cos( (o fi, Vo, G2) < 0 for some j.j T: (ii) differing gradient magnitudes: (iii) the unitary scalarization CMT has high curvature along Vo, L The PCGrad [66 SMTO presented as solution to the tragic triad, targeted at the first condition. Consistent with the previous sections, let us denote the update direction by g. Furthermore, let [x]+ == max(x, 0). Given per-task
|Vo fill
Cy for some i, T: and
gradients Ve, C,. PCGrad iteratively projects each task gradient onto the normal plane of all the gradients with which it conflicts:
B Va (x(
gi + Von Li
gi
1 gi
Vo, f, VjET Vi WET, 8= B gi: (5)
Vap
NET
where the iterative updates of with respect to Ve, Ljz are performed in random order.
GradDrop Chen et al. 11 focus on conflicting signs across task gradient entries, arguing that such confficts lead to gradient "tug-of-wars" The GradDrop SMTO [IT], presented solution to this problem, proposes to randomly mask per-task gradients Vey C; so as to minimize such conflicts. Specifically, GradDrop computes the "positive sign purity" pj for the task gradient's j-th entry and then masks the j-th entry of each per-task gradient with probability increasing with py. if the entry is negative, or decreasing with py. if the entry is positive. Let write p = ........ dimensionality of the parameter space (see (3) for the Hadamard product and 1l, for the indicator vector on condition a. Given a vector u, uniformly sampled in [0, 1] at each iteration, GradDrop takes step in the direction given by:
where S is the
Ven Li ① 1 (12))
2 (c,,p)
② 1 [(ncp)) with p = 1 1/ Ever Vo,
g
Vo, C, @ (Vo Ci<0)
2 Σier Vop
1+
(6)
4 Experimental Evaluation
Relying on a unified experimental pipeline, we present an empirical evaluation on common MTL benchmarks of unitary scalarization ) of the popular SMTOs presented in (3) and of the recent RLW algorithms [40] due to their similarities with PCGrad and GradDrop (see (5.2). We benchmark against the two RLW instances that showed the best average performance in the original paper: RLW with weights sampled from a Dirichlet distribution ("RLW Diri."), and RLW with weights sampled from Normal distribution ("RLW Norm."). The goal of this section is to assess the efficacy of popular line of previous work, focusing on few representative or well-established optimizers. Therefore, we forego comparison with more recent SMTOs 28 48 Nevertheless, we point out that these algorithms often lack significant enough improvements over the optimizers we consider, or may have substantial commonalities with them (see $5.2 for Nash-MTL [48], which was published concurrently to the finalization of this work). Whenever appropriate, we employ "Unit. Scal." as
shorthand for unitary scalarization. We first present supervised learning experiments (44.1) I and then evaluate popular reinforcement learning benchmark (.4.)

---

108. Sca
1471
MSDH
Oraffrcc
BLD Cfr.
FLWPRom
(a) Avg. task accuracy: and 95% (10 (b) Box plots for the training time of an epnch (10 runs).
Figure 1: No algorithm outperforms unitary scalarization the Multi-MNIST dataset.
Our experiments indicate that the performance of unitary scalarization has been consistently un- derestimated in the literature. By showing the variability between runs and by relying standard regularization and stabilization techniques from the single-task literature, we demonstrate that no SMTO consistently outperforms unitary scalarization across the considered settings. This result holds in spite of the added complexity and computational overhead associated with most SMTOs. Furthermore, in supervised learning, most methods drive the training loss of all tasks in the proximity of the respective global minima. This suggests that the main difficulty of MTL is not associated with the optimization of its training objective, but rather incorporating adequate regularization (cf. 55
4.1 Supervised Learning
All the architectures employed the supervised learning experiments conform to the encoder-decoder structure detailed in ] Whenever suggested by the original authors for this context, the SMTO implementations rely on per-task gradients with respect to the last shared activation, V., rather than the usually more expensive VeC,. In particular, this is the case for MGDA, IMTL and GradDrop. See appendix for details concerning each individual algorithm. Surprisingly, several MTL works [11 40
42 66) report validation results, making easier overfit. Instead, following standard machine learning practice, we select a model on the validation set, and later report test metrics for all benchmarks. Validation results also available in appendix [D] Appendix C.] reports dataset descriptions, the computational setup, hyperparameter and tuning details.
4.1.1 Multi-MNIST
We present results the Multi-MNIST (54 dataset, simple two-task supervised learning benchmark.
We employ a popular architecture from previous work [54] 66 (see appendix (C.I). where a single dropout layer [56] (with dropout probability 0.5) is employed in both the encoder and the decoder. &2 regularization did not improve validation performance and was therefore omitted. Figure repporss the average task test accuracy, and the training time per epoch. For each run, the test model was selected as the model with the largest average task validation accuracy across the training epochs. Appendix presents the results of Figure ii tabular form, as well as the average task validation accuracy per epoch. As seen from the overlapping confidence intervals, none of the considered algorithms clearly outperforms the others. However, GradDrop displays higher experimental variability. Furthermore. Figure ((b) shows that the sums of the task cross-entropy losses is driven nearly to zero by most meth- ods. Finally, Figure shows that unitary scalarization also has among the lowest training times.
4.1.2 CelebA
We now show results for the CelebA 4 dataset, challenging 40-t multi-label classification prob-
lem. We employ the same architecture as many previous studies 40 42 54
66 (see appendix C.I).
We tuned l regularization terms À for all SMTOs in the following grid: X e {0. 10 10 The
best validation performance was attained with À 10 3 for unitary scalarization, IMTL and PCGrad.
and with À 10-1 for MGDA, GradDrop, and R.W.W Validation performance was further stabilized
1971
M9000
Gractro
Mome.
(a) Avg. task test accuracy: mean and 95% runs). (b) Box plots for training time of an epoch run)).
Figure 2: While SMTOs display larger runtimes, none of them outperforms the unitary scalarization on the CelebA dataset.
5

---

3.844
Cree
2.126
(a) Absolute depth test error: lower is better.
(b) Relative depth test error: lower is better.
3.71
3.09
Uni
Inn
SradBrop
FCGee
BLW Des
nue Norr.
Midex
(c) Test segmentation mIOU: higher is better.
(d) Test segmentation accuracy: higher is better.
Figure 3: On Cityscapes, none of the
SMTOs outperforms unitary scalarization, which proves to be the most cost-effective algorithm. Subfigures (a) (d) report means for three runs, and their 95% CIs.
Und Sca
Graatrog
FCora
Harn.
(c) Box plots for the training time of an epoch (10 runs).
by the addition of several dropout layers (see Figure 5). with dropout probabilities from 0.25 to 0.5. We present an ablation study on the effect of regularization on this experiment in $5.1 Figure TO (appendix D shows that regularization improves the peak average validation performance for all the considered methods. Analogously to our Multi-MNIST results, Figure 2 plots the distribution of the training time per epoch, and the average test task accuracy. As with Multi-MNIST, the test model for each run was the one with maximal average validation task accuracy across epochs. In other words, if the peak attained before the last epoch, we perform early stopping: as shown in Figure 8(a) in appendix D this is the case for most methods. Due to the large number of tasks, Figure 2(b) shows relatively large runtime differences across methods. PCGrad is the slowest (roughly 35 times slower than unitary scalarization). In fact, amongst the considered algorithms, it is the only that computes per-task gradients over the parameters (VgL, Vi E T) at each iteration. GradDrop, MGDA and IMTL have overhead factors (compared to unitary scalarization) ranging from roughly 1.05 to 2.4 due to the relatively small size of z for the employed architecture. The overhead of RLW is negligible: roughly 5%. Nevertheless, due largely overlapping confidence intervals in Figure ((a) none of the methods consistently outperforms unitary scalarization. In fact, owing to our adoption of explicit regularization techniques (see (5.1) its average performance is superior that reported in the literature [42] [54]. As with Multi-MNIST, Figure (()) demonstrates that the cross-entropy loss of each task be driven near its global optimum by most optimizers.
4.1.3 Cityscapes
In order to complement the multi-task classification experiments for Multi-MNIST and CelebA, present results for Cityscapes [13]. a dataset for semantic understanding of urban street scenes. We rely on a common encoder architecture from the literature [40] 42 (see appendix [.I). with a
single dropout layer in the task-specific heads [40]. As for CelebA, unitary scalarization, IMTL, and PCGrad benefit from more regularization than the other optimizers: we employ X = 10-5 for these three algorithms, it resulted in better validation performance the majority of metrics, and for the remaining methods. Cityscapes a heterogeneous MTL problem: contains tasks of different types whose validation metrics cannot be averaged perform model selection. Considering the lack of an established procedure in this context, we potentially evaluate different model for each metric, chosen the with the best (maximal minimal, depending the metric) validation performance across epochs (we perform per-run early stopping). This procedure maximizes per-task performance, at the cost of increased inference time. If inference time is a priority, an alternative model selection procedure could rely on relative task improvement [28] 41 48). assuming that per-metric improvements are to be weighted linearly. Nevertheless, any consistently applied model selection scheme serves the main goal of our work: evaluating all SMTOs on a fair ground. Figure shows test results for two metrics per task, and the distribution of the training time per epoch. with
and CelebA, no training algorithm clearly outperforms unitary scalarization (significant overlaps across confidence intervals exist), which is again the least expensive method. In contrast with popular hypothesis
[10 30 42 this holds in spite of relatively large loss imbalances. In fact, the loss for the depth task is roughly 10 times smaller than that of the segmentation task:
figures 17() (7g)) Nevertheless, both losses are rapidly driven towards their respective global

---

0.95
0.0
0.85
0.4
POdred (a) MTI0 (10 repetitions).
MoDA
Grad
bvi.
(b)
MT50 (10 repetitions).
2x
Runtime
12
Runtime
233
Unt. Scal
IMI1
MUDA
GrddDtp
Motrad
RLA Diis. HUW Nom.
UKt Scal
MoDe
GracDrop
PCDred
PUW Dini. HUW Norm.
(c) MTIO (10 repetitions).
(d) MT50 (10 repetitions).
Figure 4: On Metaworld. none of the SMTOs significantly outperforms Unit. Scal., which is the
least expensive method. Subfigures (a) (b) report mean and 95% CI for the best (over the updates)
average success rate. Subfigures (c) show box plots for the training time of 10,000 updates.
minima. Unlike CelebA (see Figure 2(b) IMTL, MGDA and GradDrop are significantly slower than unitary scalarization (factors from 1.6 to 2.3), due to the relatively (compared to the parameter space) large size of in the employed architecture. PCGrad, instead, appears be less expensive (30% more than the baseline), demonstrating the benefits of working on VoL; on this model.
4.2 Reinforcement Learning
For RL experiments, we use Meta-Wois World 65
and the Soft Actor-Critic
20 implementation from 55
Unlike 14.1 the employed network architecture (see appendix C.I is fully shared across tasks. Therefore, all SMTO implementations for these experiments rely on per-task gradients with respect to network parameters VoL; (see {5). Among the SMTOs we consider, PCGrad is the only one developed with the RL setting in mind. For fairnsss and completeness, we add all the other SMTOs from the supervised learning experiments, and are first to test these optimizers the RL setting. To stabilize learning, we increase the replay buffer size, well known technique in single-task RI., add actor l2 regularization, and modify the reward normalization employed by Sodhani et al. (55). The unitary scalarization performance reported by Yu et al. (66] is considerably lower than that of Sodhani et al. (55), which we believe due to the lack reward normalization in the former. Sodhani al. 55 keep moving average of rewards in the environment, with hyperparameter controlling the speed the moving average. As we show in Figure [6] the learning algorithm is sensitive that hyperparameter. Moreover, such normalization might make similar transitions have drastically different rewards stored in the replay buffer. To alleviate these issues, we store the raw rewards in the buffer, and normalize only when mini-batch is sampled.
Figure 4 reports the best average success rate across the updates and the runtime for 10,000 updates. In addition to these summary statistics, reported for consistency with [4.1] the learning curves shown in appendix [E] Our MT10 (10 tasks) results in Figure ((a) show that by stabilizing the baseline using standard RL techniques, unitary scalarization performs on par with other SMTOs, mirroring our findings 1 This in contrast with the previous literature, which reported that PCGrad outperforms unitary scalarization [55] 66). Figure|4(h pressenss results on MT50 (50 tasks): similarly to MT10, none the SMTOs significantly outperforms unitary scalarization, with PCGrad's 's average being slightly above unitary scalarization. We speculate that the stochastic loss rescaling performed by PCGrad (see Proposition 3 reduces the differences in task return scales, and expect that methods like PopArt (60) would have similar effect without requiring access to per-task gradients. While did not tune hyperparameters for MT50 (we employed those found for MT10), it would be much lower runtime. In fact, Figure 4(d) shows that a single unitary scalarization run takes roughly 15 hours, whereas PCGrad, MGDA and GradDrop require more than a week. Similarly to MT 10, actor regularization pushes the average performance of unitary scalarization higher (see in appendix E.2).. Overall, as in the supervised learning setting, unitary scalarization performs comparably SMTOs despite being simpler and less demanding in both memory and compute. IMTL was unstable on this RL benchmark and all of the runs crashed due to numerical overflow. We hence omit IMTL results from the main body the paper and show its results in Figure 13 in appendix E, which also describes a possible explanation. We hypothesize that the instability of TMTL. due to lack of bounds on scaling coefficients. See appendix C.2 for hyperparameter settings and ablation studies.
casier to do that for unitary scalarization due
7

---

Regularization in Specialized Multi-Tash Optimizers
The empirical results presented S motivate the need to carefully analyze existing SMTOs. We make an initial attempt in this direction by viewing their effects through the lens of regularization. Let us define a regularizer as a technique to reduce overfitting [13] We first show that the SMTOs considered in empirically act regularizers via ablation study (:..)). We then take a closer look at their behavior, presenting technical results that support their alternative interpretation as regularizers ($5.2). Finally, |5.3 provides additional empirical backing for some of the technical results. Unless otherwise stated, we assume that MTL methods apply only to 0i and that standard gradient-based updates employed for tasks-specific parameters 01. We furthermore adopt the following shorthands: (0) for LJ X, i), Y). and VoL; for VoLi( i)),))
Xi)..
5.1 Ablation Study
We repeat the experiment from 4.1.2 and remove explicit regularization: no dropout layers are added to the encoder-decodes architecture, and X = 0 for all optimizers. In addition, we examine the behavior of two different fz-regularized instances of unitary scalarization:
10-1 for "Unit. Scal. f2",, À = 2 x 10-3 for "Unit. Scal. lz+". Figure 5 shows that SMTOs behave similarly to an éz-penalized unitary scalarization. Importantly, SMTOs delay overfitting, requiring less early stopping compared to unitary scalarization to obtain comparable performance. In other words, early stopping is sufficient for unitary scalarization to perform on par with SMTOs. Finally, overfitting is further reduced by "Unit. Scal. Reg.". which plots the regularized unitary scalarization from $4.1.2 with dropout layers and weight decay of = 10-3 Further results are presented appendix D.2
5.2 Technical Results
All the methods considered in [5.1 regularize more than unitary scalarization. While RLW was shown to reduce overfitting by the original authors (40 theorem 2], we now provide a collection of novel and existing technical results that potentially explain the regularizing behavior of each of the other algorithms, complementing the presentation from } In particular, we show that MGDA, IMTL and PCGrad have larger convergence set than unitary scalarization, reducing the chances land on sharp local minima [15]. Furthermore, GradDrop and PCGrad introduce significant stochasticity, which often linked to the same effect [3] We hope these observations will steer further research.
MGDA Let us denote the convex hull of a set A by Conv(A). We now recall a well-known property of MGDA [4] and relate it to the behavior of unitary scalarization.
Proposition The MGDA SMTO (54) converges to a superset of the convergence points of unitary scalarization. More specifically, it converges to any point Oj such that: O Conv((Vo; La | TT))
See appendix B.I for simple proof. proof. As consequence of Proposition I MGDA does not necessarily reach a stationary point for LMT (that is, point for which Σier Vo, Vo,2: Z. = 0) or for any of the losses L, (Vo C: 0). For example, any point Ol for which two per-task gradients point in opposite directions is Pareto stationary. On account of the well-known [15 relationship between under-
optimizing (e.g., early stopping 7 39) and overfitting, proposition supports the interpretation of
MGDA a regularizer for equation (I) Empirical evidence that MGDA under-optimizes is provided
8515
Unio Seal
BAM 2ir
127
Putt
10"
6.85
Teainig
Teamss tacks
Figure 5: Mean and 95% (3 runs) avg. task val- idation accuracy over epochs on CelebA. SMTOs postpone the onset of overfitting, mirroring the effect of & regularization on unitary scalarization.
Figure 6: Mean and 95% CI (3 runs) for |SieT Co,Ci on CelebA. MGDA and IMTL converge away from stationary points of unitary scalarization, indicating under-optimization.
8

---

in |.. Figure (a) and Figure 5 which shows over-regularization Proposition I can be trivially extended to the recent Wash-MTZe which shares the same convergence set 48 Theorem 5.4].
IMTL We now show that aggregating per-task gradients so that their cosine similarity is the same (equation ( yields a constrained steepest-descent algorithm (Proposition 2). This view on the
update step of IMTL leads to a novel analysis of its convergence points (corollary [ Proofs can be found in appendix B.2 We will denote by Aff(A) the affine hull set A.
Proposition 2. IMTL by Liu al. [42] updates On by taking a step in the steepest descent direction whose cosine similarity with per-lask gradients is the same across lasks.
Corollary 1. IMTL by Liu et al. [42] converges to a superset of the Pareto-stationary points for O (and hence of the convergence points of the unitary scalarization). More specifically, it converges to any point ii such that: 0 Aff (( /o T
As seen for MGDA, corollary [1 implies that, even if the employed model has the capacity to reach the minimal loss on _MF TMTIL may stop before reaching a stationary point. Recalling the relationship between under-optimizing and overfitting [15), this supports the interpretation of IMTL.
as a regularizer for equation (T. This is empirically shown in [5.3 Figures 5 [a] In particular,
unitary scalarization reaches the same average performance of IMTL but requires earlier stopping.
PCGrad We provide an alternative characterization of the PCGrad update rule, highlighting its
stochasticity in the context of its interpretation loss rescaling 40 42
See appendix B.3 for a proof.
PCGrad is equivalent to dynamic, and possibly stochastic, loss rescaling for p1. Ar each teration, per-task gradients are rescaled as follows:
Proposition 3.
Von C3
Ci t (L+E)ETVO) dis Ve, dj
0,
Vo C.
Furthermore, IT 2, dj a random variable, and the above range contains its support.
The results from proposition] can be easily extended to GradVac [62] which generalizes PCGrad's projection onto the normal vector to arbitrary target cosine similarities between per-task gradients. When [T] > 2, PCGrad corresponds a stochastic loss re-weighting. As such, PCGrad bears many similarities with Random Loss Weighting (RLW) 40) RLW proposes to sample scalarization weights from standard probability distributions at each iteration, and proves that this leads the better general- ization 400 theorem 2]. Indeed, it is well- that adding noise to stochastic gradient estimations leads the optimization towards flatter minima, and that such minima may reduce overfitting [3] 34 In line with the main technical results by Yu et al. [66], we now restrict our focus to two-task problems, which allow for an easy description of PCGrad's convergence points. The result is largely based on [66] theorem I]: we relax some of the assumptions and provide proof in appendix B.3
Corollary 2. [T] = 2, PCGrad will stop at any point where cos( Ve L1, Ve, L2) -1. Further- more, if L1 and L2 are differentiable, and Ve, CMI is L-Lipschitz with L > 0, PCGrad with step size t << t, converges to a superset of the convergence points of the unitary scalarization.
Corollary 2 implies that, when |T= 2, PCGrad may under-optimize equation 4 as MGDA and IMTL. In particular, if cos( Vo, Vo, f2) -1, then 0 E Conv([Ve, L1. Vo C2) (see proposition D. We believe that PCGrad's stochasticity and enlarged convergence set potentially explain its regularizing effect.
GradDrop While the motivation behind GradDrop is to avoid entry-wise gradient conflicts across tasks, the main property of the method is to drive the optimization towards "joint minima": points that are stationary for all the individual tasks at once [1] proposition 1]. In other words: Vo, C. = 0 e T. While this property is desirable, we show that it holds beyond GradDrop, and independently of the gradient directions. Under strong assumptions on the model capacity, the above property would trivially hold for unitary scalarization (proposition
5 appendix B.4). Proposition 4 shows that it holds for a simple randomized version of unitary scalarization, which we name Random Grad Drop (RGD).
Proposition 4. Let CRGD (On) := EIET (0)) where u, Bernoulli(p) Vi E T and p (0, 1]. The gradient Vo LRGD is always zero if and only if Vo Ca = vi E T. In other words, the result from 11 proposition 1] can be obtained without any information on the sign of per-task gradients.

---

Proposition 4 (see appendix B.4 for simple proof) shows that an inexpensive signinneppendet stochastic scalarization shares GrodDrop's main reported property. _ can be directly cast an instance of RLW,, and hence as a regularization method [31 34 Furthermore, Figure [2 in
appendix [D3 shows that the empirical results of GradDrop on CelebA [4 are closely matched
by sign-agnostic gradient masking, partly undermining the conflicting gradients assumption. We believe that the above results, along with the authors' original experiments showing that GradDrop delays overfitting on CelebA [1I figure 3), suggest that GradDrop behaves as a regularizer.
5.3 Under-Optiimzation: Empirical Study
As seen in [5. MGDA and IMTL might under-optimize equation (1 compared to unitary scalar- ization due to their larger convergence sets. In order to assess whether this is empirically the case, we estimate |Eiet Ve, fill the norm of the unitary scalarization update on shared parameters O||- for all optimizers throughout the unregularized CelehA experiment from $5.1 Large magnitudes for Vo,fill Lil1 towards convergence would indicate that SMTOs steer optimization far from sta- tionary points of unitary scalarization, resulting in under-optimizzaton. We compute the update norm on the mini-batch loss every 100 updates, and report the per-epoch average in Figure Most SMTOs have a smaller update magnitude than unitary scalarization in the first 15 epochs. However, towards convergence, SMTOs display larger |SieT Vo, fill, compared to unitary scalarization. In particu- lar, IMTL and MGDA have the largest norm, denoting significant empirical under-optimization. The additional stochasticity of RLW, PCGrad, and GradDrop also appears lead to larger norm values than unitary scalarization, yet to a lesser degree. Given that MGDA and IMTL incur a larger loss than unitary scalarization in later epochs (see Figure [[a] in appendix D.2). we can conclude that they guide optimization towards regions of the parameter space that under-optimize equation (I)
[Eier Vo, fi]
providing empirical support for our analysis.
6
Conclusions
This paper made two main contributions. First, we evaluated popular SMTOs using a single experi- mental pipeline, including previously unpublished results of MGDA, IMTL, RLW, and GradDrop in the RL setting. Surprisingly, our evaluation showed that none of the SMTOs consistently outperform unitary scalarization, the simplest and least expensive method. Second, in order to explain our prising results, we postulate that SMTOS act as regularizers and present an analysis that supports our hypothesis. We believe work calls for further reevaluation of progress in developing principled and efficient MTL. algorithms.
We conclude by addressing the limitations of our work. While we covered wide range of popular benchmarks, we do not exclude the existence of settings where unitary scalarization underperforms: discovering them is an interesting direction for future work. Furthermore, our experimental results were obtained via grid searches under limited compute resources: some of the methods might benefit from further fine-tuning, Nevertheless, we remark that fine-tuning will be easier for unitary scalarization due to its shorter runtimes. Finally, we presented the regularization hypothesis only as a partial explanation of our results: we hope it will steer further analysis and consequently improve the understanding of MTL.
Acknowledgements
VK was funded by Samsung R&D Institute UK through the EPSRC Centre for Doctoral Training (CDT) in Autonomous Intelligent Machines and Systems (AIMS) at the University of Oxford ADP was funded by EPSRC for the AIMS CDT, grant EP/LO15587/11 and by an IBM PhD fellowship. SW has received funding from the European Research Council under the European Union's Horizon 2020 research and innovation programme (grant agreement number 637713). The experiments were made possible by generous equipment grant from NVIDIA. We would like to thank Lin al.
40 Sodhani et al. [55) and Sener and Koltun (54] for publicly releasing their code. The authors thank Kristian Hartikainen for helpful comments on the RI. experiments. VK thanks Ryota Tomioka
for useful discussions multitask optimization.
10

---

References
[1] Z. Allen-Zhu, Y. Li. and Z. Song. A convergence theory for deep learning via over- parameterization. In International Conference Machine Learning, 2019.
[2] V. Badrinarayanan, A. Kendall, and R. Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2017.
[3] B. Bakker and T. Heskes. Task clustering and gating for bayesian multitask learning. Journal of Machine Learning Research, 2003.
[4] Q Cappart, D. Chételat, E. B. Khalil, A. Lodi, C. Morris, and P. Velickovic. Combinatorial optimization and reasoning with graph neural networks. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, 2021.
[5] R. Caruana. Multitask learning. Machine Learning, 28(1):41-75, 1997.
[6] R. Caruana, Multitask learning. PhD thesis, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213, 1997.
[7] R. Caruana, S. Lawrence, and L. Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Neural Information Processing Systems, 2000.
[8] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decode with atrous separable convolution for semantic image segmentation. In European Conference Computer Vision, 2018
[9] S. Chen, Y. Zhang, and Q. Yang. Multi-task learning in natural language processing: An overview. CoRR, 2021.
[10] Z. Chen, V. Badrinarayanana, C.-Y. Lee, and A. Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, 2018
[11] Z. Chen, Ngiam, Y. Huang, T. Luong, Kretzschmar, Y. Chai, and D. Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In Neural Information Processing Systems, 2020.
[12] R. Collobert and J. Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICMI. 2008), Helsinki, Finland, June 5-9, 2008, 2008
[13] M. Cordts, M. Omran, Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Conference on Computer Vision and Pattern Recognition, 2016.
[14] J. Désidéri. Multiple -raadiett descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathematique, 350:313-311 2012.
[15] T. Dietterich. Overfitting and undercoomptting in machine learning. ACM Computing Surveys, page 326-327, sep 1995.
[16] T. Evgeniou and M. Pontil. Regularized multi-task learning. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2004.
[17] Fliege and B. Svaiter. Steepest descent methods for multicriteria optimization. Mathematical Methods of Operations Research, 2000.
[18] M. Guo, A. Haque, D.-A. Huang, S. Yeung, and L. Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[19] P. Guo, C.-Y. Lee, and D. Ulbricht. Learning to branch for multi-task learning. 2020.

---

[20] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018.
[21] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. Conference on Computer Vision and Pattern Recognition, 2016.
[22] T. Heskes, Empirical bayes for learning to learn. In International Conference on Machine Learning, 2000.
[23] M. Hessel, H. Soyer, I.. Espeholt, W. Czamecki, S. Schmitt, and H. van Hasselt. Multi-task deep reinforcement learning with popart. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thisty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- February 2019, pages 3796-3803, AAAI Press, 2019.
[24] T. M. Hospedales, A. Antoniou, P. Micaelli, and A. J. Storkey. Meta-lcarning in neural networks: survey, CoRR, 2020.
[25] W. Huang, Mordatch, and D. Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In Proceedings of the 37th International Conference Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, 2020.
[26] loffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference Machine Learning, 2015.
[27] M. Jaderberg, V. Mnih, W. M. Czamecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In Sth International Conference on Learning Representations, ICLR 2017. Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[28] A. Javaloy and I. Valera. Rotograd: Gradient homogenization in multitask learning. In International Conference on Learning Representations, 2022.
[29] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski. C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. CoRR, 2021.
[30] A. Kendall, Y. Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[31] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. International Conference on Learning Representations, 2017.
[32] K. Khetarpal, M. Riemer, Rish, and D. Precup. Towards continual reinforcement learning: A review and perspectives. arXiv preprint vrXiv:2012.13490 2020.
[33] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and
Y. LeCun, editors, International Conference on Learning Representations, 2015.
[34] B. Kleinberg, Y. Li. and Y. Yuan. An alternative view: When does SGD escape local minima?
In International Conference on Machine Learning, 2018.
[35] I. Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[36] V. Kurin, S. Godil, S. Whiteson, and B. Catanzaro. Can q-learning with graph networks learn a generalizable branching heuristic for SAT solver? In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
12

---

[37] V. Kurin, M. Igl. T. Rocktäschel, W. Bochmer, and S. Whiteson. My body is a cage: the role of morphology in graph-based basph incompatible control. In 9th International Conference on Learning Represeniations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[38] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- learning applied to document recognition. IEEE, 1998.
[39] M. Li. M. Soltanolkotabi, and Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International Conference on Artificial Intelligence and Statistics, 2020.
[40] B. Lin, F. Ye, and Y. Zhang. A closer look loss weighting in multi-task learning. In arXiv preprint arXiv:21 11. 106033, 2022.
[41] B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 2021.
[42] L. Liu, Y. Li, Z. Kuang, J.-H. Xue, Y. Chen, W. Yang, Q. Liao, and W. Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations, 2021.
[43] Liu, E. Johns, and A. Davison. End-to-end multi-task learning with attention. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1871-1880 2019.
[44] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
[45] S. Ma, R. Bassily, and M. Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-paramenne werrpaaameetrized learning. In International Conference on Machine Learning. 2018.
[46] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, 2016.
[47] S. Narvekar, B. Peng, M. Leonetti, J. Sinapov. M. E. Taylor, and P. Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Mach. Learn. Res., 2020.
[48] A. Navon, A. Shamsian, Achituve, H. Maron, K. Kawaguchi, G. Chechik, and E. Fetaya. Multi-task learning as bargaining game, In International Conference on Machine Learning, 2022.
[49] E. Parisotto, I. J. Ba, and R. Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[50] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Vetioo, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I.. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, -egh-performance deep learning library. In Neural Information Processing Systems. 2019.
[51] A. A. Rusu, S. G. Colmenarejo, Ç. Gülçehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[52] S. Sabour, N. Frosst, and G. E. Hinton. Dynamic Routing between Capsules. 2017.
[53] M. L. Seltzer and J. Droppo. Multi-task learning in deep neural networks for improved phoneme recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May 26-31, 2013, 2013.
[54] O. Sener and V. Koltun. Multi-task learning as multi-objective optimization. In Neural Information Processing Systems, 2018
13

---

[55] S. Sodhani, A. Zhang, and J. Pineau. Multi-task reinforcement learning with context- based representations. In M. Meila and Zhang, editors, International Conference on Machine Learning, 2021.
[56] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014.
[57] Y. W. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu. Distral: Robust multitask reinforcement learning.
Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 4496-4506 2017.
[58] Y. W. Teh, V. Bapst, W. M. Czarnecki. J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu. Distral: Robust multitask reinforcement learning. In Neural Information Processing Systems, 2017.
[59] W.-C. Tseng. Wei WeiChengTseng/ SeiChhnng/sngg/yttorrr---Gaaa git
Weichengtssnggyyttoch--ogggg 2020.
URL https:// com/
[60] H. van Hasselt, A. Guez, M. Hessel, V. Mnih, and D. Silver. Learning values across many orders of magnitude. Advances in Neural Information Processing Systems, 29:4287-4295, 2016.
[61] S. Yandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai, and L. Van Gool. Multi- task learning for dense prediction tasks: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021.
[62] Z. Wang. Y. Tsvetkov, O. Firat, and Y. Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In International Conference on Learning Representations, 2021.
[63] D. Xin, B. Ghorbani, A. Garg, O. Firat, and J. Gilmer. Do current multi-task optimization methods in deep learning even help? In Neural Information Processing Systems, 2022.
[64] Yu, V. Koltun, and Funkhouser. Dilated residual networks. In Computer Vision and Pattern Recognition, 2017.
[65] T. Yu, D. Quillen, Z. He. R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, 3rd Annual Conference Robot Learning, 2019.
[66] Yu, S. Kumar, Gupta, S. Levine, K. Hausman, and Finn. Gradient surgery for multi-task learning. In Neural Information Processing Systems, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] see 1
(c) Did you discuss any potential negative societal impacts of your work? [Yes] due
space constraints, we provide a discussion in appendix[] [
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] we provide full proofs
in the Appendix, and refer to them in the main body of the paper.
14

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] provide the code and the instructions in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] see appendix C.I
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] appendix C.3 describes licenses of all benchmarks and implementations
used for work.
(c) Did you include any new assets either in the supplemental material as a URL? [Yes] we include the code and the instructions on how to replicate the experiments into the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "ykHBZaTQ-cL", "reviewer": "Reviewer_pDqD", "summary": "The authors perform an empirical evaluation of various optimizers designed specifically for multi-task learning, particularly those that utilize per-task gradients. The experiments, which are conducted on several image classification tasks and the metaworld MT10 and MT50 multi-task RL benchmarks, generally show that unitary scalarization (simply summing loss for the tasks) is a very strong baseline for multi-task learning when its regularization (l2 + dropout) is tuned, and the benefits of more sophisticated optimizers is dubious, given their computational cost. The authors interpret their findings to mean that more sophisticated multi-task learning objectives essentially perform a form of early stopping, providing some initial theoretical arguments for why this may be the case.", "questions": "Overall, my biggest concern with the paper is Section 5. While I can appreciate the general intuition that a larger number of convergence points may lead to early stopping-like behavior, the authors don't convincingly argue that this is indeed the case in practice. For example, the statement that PCGrad has an \"enlarged convergence set\" based on the points where gradients are exactly opposed seems like a stretch- this is a vanishingly small set of points. Nonetheless, PCGrad performs reasonably favorably to the other baseline algorithms, and the alternative argument that the stochasticity in the objective leads to better performance is not developed in much depth.\n\nOne way to address these concerns would be to empirically show that, as their theoretical argument suggests, SMTOs do indeed converge to solutions that are far from the fixed points of unitary scalarization. i.e., at SMTO convergence, is the unitary scalarization update magnitude still large? Alternatively, can the authors compare the flatness of the minima for regularization unitary scalarization and SMTOs? The authors invoke the flatness intuition, but it is not validated.\n\nRegarding methodological novelty, I wonder if the authors can also provide more detailed discussion of their results; currently, while the results are interesting, they are not directly actionable for the community. What do the authors recommend the MTL community do with these results? How can these insights translate into better or more efficient models? I wonder if some of the text allocated to the theoretical section might be better used for such discussion.\n\nAs a minor point, Figure 5 is quite hard to interpret, especially for colorblind readers. I would suggest using multiple line styles to make it easier to compare between unit scale and SMTO lines.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Thorough evaluation of multi-task learning (MTL) methods, with experimental details well-catalogued\n- Surprising empirical result\n- Likely to be a useful set of reference experiments within the MTL world\n- Most of the paper is very well written\n\nWeaknesses:\n- Theoretical section seems hurried, and I'm not totally convinced by the line of argument (see below)\n- As the authors point out, more comprehensive hyperparameter tuning would be nice, given the paper's primary contribution is providing a large comparison\n- No methodological novelty (this is not a requirement for a strong paper, but without it, the key results need to be very strong)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "yKX9JZKek3r", "reviewer": "Reviewer_9gYq", "summary": "This paper considers the multitask learning setup (MTL) in which a model is trained to perform several tasks. Several existing MTL approaches argue against the simple baseline of using the sum per-task losses (unitary scalarization) as the training objective. Instead, SOTA MTL approaches usually manipulate the per-task gradients to obtain a joint update direction. Thus, the more complex per-task methods increase the training time and implementation complexity. The authors show that by adding proper regularization and carefully tuning the unitary scalarization baseline, it can perform on-par with existing MTL approaches. The paper raises some valid concerns regarding the evaluation and experimental methodology in the MTL literature. However, some questions about the experimental setup and missing comparisons/benchmarks impair the authors' claim. \n", "questions": "See above.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The paper is well organized and well written.\n- This paper offers a novel view of SOTA MTL methods as regularization methods.\n- The authors perform an important evaluation of recent MTL methods and raise valid concerns regarding the evaluation and experimental methodology in the MTL literature.\n- Novel analysis and view of well-known MTL methods through the lens of regularization.\n\n**Weaknesses:**\n- Most MTL approach uses the feature gradients as a surgent to the full task gradient (or uses other approaches to scale the computation), hence resulting in only a small increase of runtime.\n- The arguments for presenting MTL methods as regularizers appear quite weak. While interesting and novel, it appears to provide only a partial explanation.\n- The experimental setup lacks important comparisons and evaluation on more relevant benchmarks:\n  - First, the authors fail to (cite [1], and) compare with recent strong MTL approaches [1,2,3] that show large performance gains w.r.t unitary scalarization. \n  - The results presented in [1,2] show better performance on the Cityscapes benchmark for all methods (for example in terms of mIoU). This might suggest that all methods are under-optimized.\n  - A better metric for MTL evaluation is the relative task improvement (compared to STL), used in previous MTL literature [1,2,3].\n  - For Cityspcaes, it’s not clear how the early stopping was performed. How do you reduce the different metrics into a single metric as the early stopping criterion? Metrics have different scales and meanings. A good approach is using the relative task improvement as the stopping criterion.\n  - The supervised learning benchmarks are insufficient: (i) Multi-MNIST is a toy dataset with only two tasks; (ii) The vast majority of methods quickly overfit the Celeb-A benchmark. Hence, this benchmark is not the best choice for arguing something concerning MTL optimization: the effective optimization process ends early, so it’s hard to learn something meaningful about the optimization dynamics. (iii) Finally, the Cityspcaes dataset consists of only two tasks that were shown to benefit each other in previous works. Hence, we cannot learn from it about optimization with a large number of possibly conflicting tasks.\n  - I suggest the authors add more relevant and challenging MTL benchmarks like the QM9 with 11 tasks (used in [1]) and the NYUv2 dataset with 3 tasks (commonly used through MTL lit.).\n\nReferences:\n- [1] Multi-Task Learning as a Bargaining Game. ICML 2022.\n- [2] Conflict-Averse Gradient Descent for Multi-task learning. NeurIPS 2021.\n- [3] RotoGrad: Gradient Homogenization in Multitask Learning. ICLR 2022.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "P3Xv3fqcSH", "reviewer": "Reviewer_Ey2v", "summary": "This work is interested in re-examining the efficacy of multi-task optimization methods, termed STMOs in the paper. The authors believe that prior STMO work does not treat the \"unitary scalarization\" baseline, i.e. the simple sum of objectives, fairly and that, when fairly treated, unitary scalarization can achieve comparable results to SMTOs.\n\nThe core contribution of this work is a large set of experiments on 4 multi-task datasets (3 vision and 1 reinforcement learning) in which the authors compare the distribution (mean & std.) of performance using 4 common SMTOs, 2 random weighting schemes, and unitary scalarization. Their results show that no individual method significantly outperforms unitary scalarization across all tasks. Although some methods may outperform unitary scalarization on certain tasks, unitary scalarization is always well within their standard deviation, and moreover the benefits pf SMTOs are never consistent across datasets. In addition, SMTOs have a significant effect on training time, increasing it drastically for little-to-no gain in performance. \n\nThe authors postulate that this is because most  SMTOs can actually be seen as regularizers. The second contribution of this work is a theoretical analysis of SMTOs in terms of their convergence sets - MGDA, PCGrad, and IMTL increase the convergence set of optimization (i.e. the number of solutions a method may stop at), and suggest that this increase in convergence sets can behave similar to early stopping. Overall, their work suggests that current SMTOs, whose focus is on mitigating task conflict, instead benefit multi-task learning through implicit regularization effects - an effect which can be mirrored much more simply with unitary scalarization.", "questions": "The CelebA experiments in 5.1 suggest that SMTOs also simply converge slower - if training continued, would they continue to overfit similar to unitary scalarization? In other words, is the early stopping exhibited in practice simply an artifact of the number of epochs the models are trained for (a fixed number of epochs for all methods) or because the multi-task methods have truly \"stopped\"? ", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- A lot of multi-task learning research has focused on conflict-mitigating methods to improve negative transfer. However, this exhaustive  set of experiments suggests that this focus has not been as beneficial to multi-task learning as previously believed. This is an important result which suggests that future MTL work should rethink how task conflict is approached.\n- The number of experiments run is helpful, and is a meaningful contribution alone - it provides a reasonable reference for future multi-task work to build off of. Moreover, the authors have promised to release the code for all SMTOs.\n- The analysis of SMTOs acting as regularizers is interesting - To the best of my knowledge this is the first joint theoretical treatment of the effects of SMTOs on the optimization.\n\nWeaknesses:\n- There seems to be a disconnect between the analysis of SMTOs and the empirical results of section 5.1:\n\t- Much of the analysis is centered around showing that there exist solutions which an SMTO may converge to, yet a unitary scalarization approach will not. To me this suggests that SMTOs will, in some senses, early-stop themselves. However, the results in 5.1 on CelebA suggest that SMTOs converge slower than unitary scalarization, and that the early stopping regularization arises because the max number of training epochs has been reached.\n- At face value, the contributions of the paper seem somewhat limited - the core contribution of this paper is a large set of experiments using prior methods. I do not think this is a major weakness, because the results are novel and important. I think the paper would benefit from much more focus on section 5 - the biggest contribution of the paper, in my mind, is the connection between regularization and SMTOs which help to explain the results in section 4. However, section 5 is comparatively small and empirical analysis of this connection is very limited (5.1).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept