Towards Impartial Multi-task Learning
Authors: Liyang Liu
Yi Li
Zhanghui Kuang
Jing-Hao Xue
Yimin Chen
Wenming Yang
Qingmin Liao
Wayne Zhang
Venue: ICLR 2021 Poster (2021)
Keywords: Multi-task Learning
Impartial Learning
Scene Understanding

Abstract: Multi-task learning (MTL) has been widely used in representation learning. However, naively training all tasks simultaneously may lead to the partial training issue, where specific tasks are trained more adequately than others. In this paper, we propose to learn multiple tasks impartially. Specifically, for the task-shared parameters, we optimize the scaling factors via a closed-form solution, such that the aggregated gradient (sum of raw gradients weighted by the scaling factors) has equal projections onto individual tasks. For the task-specific parameters, we dynamically weigh the task losses so that all of them are kept at a comparable scale. Further, we find the above gradient balance and loss balance are complementary and thus propose a hybrid balance method to further improve the performance. Our impartial multi-task learning (IMTL) can be end-to-end trained without any heuristic hyper-parameter tuning, and is general to be applied on all kinds of losses without any distribution assumption. Moreover, our IMTL can converge to similar results even when the task losses are designed to have different scales, and thus it is scale-invariant. We extensively evaluate our IMTL on the standard MTL benchmarks including Cityscapes, NYUv2 and CelebA. It outperforms existing loss weighting methods under the same experimental settings.

Published as a conference paper at ICLR 2021
TOWARDS IMPARTIAL MULTI-TASK LEARNING
Liyang Liu', Yi Li, Zhanghui -Jingg Maag-Haa&& Xue", Yimin Chen², Wenming Yang Qingmin Liao', Wayne Zhang
Shenzhen International Graduate School/Department of Electronic Engineering, Tsinghua University
TensseTime Research
3 "Department of Statistical Science, University College London
"Qing Yuan Research Institute, Shanghai Jiao Tong University
(Liu-ly14emails, yang. weaningesz., 1iaogn8 ttsnghuss edu, [liyi, kuangzhanghui, chenyimin, wayne. zhang) ime. com jinghac. peefucl.ac.un
ABSTRACT
Multi-task learning (MTL) has been widely used in representation learning. How- ever, naïvely training all tasks simultaneously may lead to the partial training is- sue, where specific tasks are trained more adequately than others. In this paper, we propose to learn multiple tasks impartially. Specifically, for the task-shared parameters, optimize the scaling factors via closed-form solution, such that the aggregated gradient (sum of raw gradients weighted by the scaling factors) has equal projections onto individual tasks. For the task-specific parameters, we dy- namically weigh the task losses so that all of them are kept at a comparable scale. Further, we find the above gradient balance and loss balance are complementary and thus propose hybrid balance method to further improve the performance. Our impartial multi-task learning (IMTL) can be end-to-end trained without any heuristic hyper- parameter tuning, and is general to be applied on all kinds of losses without any distribution assumption. Moreover, our IMTL can converge to sim- ilar results even when the task losses are designed to have different scales, and thus it is scale-invariant. We extensively evaluate our IMTL on the standard MTL benchmarks including Cityscapes, NYUv2 and CelebA. It outperforms existing loss weighting methods under the same experimental settings.
INTRODUCTION
Recent deep networks in computer vision can match or even surpass human beings on some specific tasks separately. However, in reality multiple tasks (e.g., semantic segmentation and depth estima- tion) must be solved simultaneously. Multi-task learning (MTL) (Caruana 1997] [997] Evgeniou & Pontil,
2004 Ruder 2017 Zhang & Yang 2017) aims at sharing the learned representation among tasks Zamir et al. 2018 to make them benefit from each other and achieve better results and stronger robustness (Zamir al. 2020). However, sharing the representation can lead to partial learning is- sue: some specific tasks are learned well while others are overlooked, due to the different loss scales or gradient magnitudes of various tasks and the mutual competition among them. Several methods have been proposed to mitigate this issue either via gradient balance such as gradient magnitude normalization (Chen et al. 2018) and Pareto optimality (Sener Koltun, 2018), or loss balance like homoscedastic uncertainty (Kendall et al. 2018). Gradient balance can evenly learn task-shared parameters while ignoring task-specific ones. Loss balance can prevent MTL from being biased in favor of tasks with large loss scales but cannot ensure the impartial learning of the shared parame- ters. In this work, we find that gradient balance and loss balance are complementary. and combining the two balances further improve the results. To this end, we propose impartial MTL (IMTL) via simultaneously balancing gradients and losses across tasks.
For gradient balance, propose IMTL-G(rad) to learn the scaling factors such that the aggregated gradient of task-shared parameters has equal projections onto the raw gradients of individual tasks
"Corresponding author

---

Published as a conference paper at ICLR 2021
go
9:
9: 92
99
*9:
99
+91
GradNorm
(b) MGDA
(c) PCGrad
(d) IMTL-G
Figure I: Comparison of gradient balance methods. In (a) to (d), 91. 92 and represent the gradient computed by the raw loss of each task, respectively. The gray surface represents the plane composed by these gradients. The red arrow denotes the aggregated gradient computed by the weighted sum loss, which is ultimately used to update the model parameters. The blue arrows show the projections of g onto the raw gradients {gi J- 9 has the largest projection on g2 (nearest to the mean direction), 9s (smallest magnitude) and g2 (largest magnitude) for GradNorm, MGDA and PCGrad, respectively, while the projections are equal on {g.} in our IMTL-G.
(see Fig. (d)). We show that the scaling factor optimization problem is equivalent to finding the angle bisector of gradients from all tasks in geometry, and derive closed-form solution to it. In contrast with previous gradient balance methods such as GradNorm (Chen et al.] (018). MGDA (Sener& & Koltun| 2018) and PCGrad (Yu al. 2020), which have learning biases favor of tasks with gradients close to the average gradient direction, those with small gradient magnitudes, and those with large gradient magnitudes, respectively (see Fig. [ (a), (b) and (c)), in our IMTL-G task-shared parameters can be updated without bias to any task.
For loss balance, we propose IMTL-L(oss) automatically learn a loss weighting parameter for each task so that the weighted losses have comparable scales and the effect of different loss scales from various tasks can be canceled-out. Compared with uncertainty weighting (Kendall et al., 2018), which has biases towards regression tasks rather than classification tasks, our IMTL-- treats all tasks cquivalently without any bias. Besides, we model the loss balance problem from the optimization perspective without any distribution assumption that is required by (Kendall et al., 2018). Therefore, ours is more general and can be used in any kinds of losses. Moreover, the loss weighting parameters and the network parameters can be jointly learned in end-to-end fashion in IMTL-L.
Further, we find the above two balances are complementary and can be combined to improve the performance. Specifically, we apply IMTL-G the task-shared parameters and IMTL-L the task-specific parameters, leading to the hybrid balance method IMTL. Our IMTL is scale-invariant: the model can converge to similar results even when the same task designed to have different loss scales, which is common in practice. For example, the scale of the cross-entropy loss in semantic segmentation may have different scales when using "average" or "sum" reduction over locations in the loss computation. We empirically validate that our IMTL is more robust against heavy loss scale changes than its competitors. Meanwhile, our IMTL only adds negligible computational overheads.
We extensively evaluate our proposed IMTL on standard benchmarks: Cityscapes, NYUv2 and CelebA, where the experimental results show that IMTL achieves superior performances under all settings. Besides, considering there lacks a fair and practical benchmark for comparing MTL meth- ods, we unify the experimental settings such as image resolution, data augmentation, network struc- ture, learning rate and optimizer option. We re-implement and compare with the representative MTL methods in a unified framework, which will be publicly available. Our contributions are:
We propose a novel closed-form gradient balance method, which learns task-shared param- eters without any task bias; and develop a general learnalle loss balance method, where no distribution assumption is required and the scale parameters can be jointly trained with the network parameters.
We unveil that gradient balance and loss balance are complementary and accordingly pro- pose hybrid balance method to simultaneously balance gradients and losses.
We validate that our proposed IMTL is loss scale-invariant and is more robust against loss scale changes compared with its competitors, and we give in-depth theoretical and experi- mental analyses on its connections and differences with previous methods.
We extensively verify the effectiveness of our IMTL. For fair comparisons, a unified code- base will also be publicly available, where more practical settings are adopted and stronger performances are achieved compared with existing code-bases.
2

---

Published as a conference paper at ICLR 2021
2
RELATED WORK
Recent advances in MTL mainly come from two aspects: network structure improvements and loss weighting developments. Network-structure methods based on soft parameter-sharing usually lead high inference cost (review Appendix A Loss weighting methods find loss weights be mul- tiplied on the raw losses for model optimization. They employ a hard parameter-sharme (Ruder) 2017), where several light-weight task specifcc heads are attached upon the heavy-weight task-agnostic backbone. There are also efforts that learn to group tasks and branch the network in the middle layers (Guo et al.] 2020, Standley et al., 2020), which try to achieve better aceuracy- efficiency trade-off and can be seen semi-hard parameter-sharing. We believe task grouping and loss weighting are orthogonal and complementary directions to facilitate multi-task learning and benefit from each other. In this work we focus loss weighting methods which the most economic as almost all of the computations are shared across tasks, leading to high inference speed. Task Prioritization (Guo et al. 2018)) weights task losses by their difficulties focus on the harder tasks during training. Uncertainty weighting (Kendall et al. 2018) models the loss weights as data- agnostic task-dependent homoscedastic uncertainty. Then loss weighting is derived from maximum likelihood estimation. GradNorm (Chen et al. 2018) learns the loss weights to enforce the norm of the scaled gradient for each task be close. MGDA (Sener Koltun 2018) casts multi-task learning as multi-object optimization and finds the minimum-nowe -nrm point in the convex hull com- posed by the gradients of multiple tasks. Pareto optimality supposed be achieved under mild conditions. GLS [Chennupati et al. 2019) instead uses the geometric mean of task-specific losses as the target loss, we will show actually weights the loss by its reciprocal value. PCGrad (Yu et al. 2020) avoids interferences between tasks by projecting the gradient of task onto the normal plane of the other. DSG (Lu et al. 2020) dynamically makes a task "stop or go" by its converging state, where task is updated only once for while if it is stopped. Although many loss weighting methods have been proposed, they are seldom open-sourced and rarely compared thoroughly under practical settings where strong performances are achieved, which motivates to give in-depth analysis and a fair comparison about them.
paradigm
3 IMPARTIAL MULTI-TASK LEARNING
In MTL, we map sample E X to its labels {ye E E.hena of all T tasks through multiple task- specific mappings {ft X + V.). In most loss weighting methods, the hard parameter-sharing paradigm is employed, such that ft is parameterized by heavy-weight task-shared parameters O and weight task-specific parameters 0,, All tasks take the same shared intermediate feature ff (æ; 0) as input, and the t-th task head outputs the prediction fi (a) = fr (z; O,). We aim to find the scaling factors {a,} } for all T task losses {Le (ft (z). }. so that the weighted sum be optimized to make all tasks perform well. This poses great challenges because: 1) losses may have distinguished forms such as cross-entropy loss and cosine similarity; 2) the dynamic ranges of losses may differ by orders of magnitude. In this work, we propose hybrid solution for both the task-shared parameters 0 and the task-specific parameters { as Fig.
loss L Σ, L
3.1 GRADIENT BALANCE: IMTL-G
For task-shared parameters O we can receive T gradients {g: Vola) via back-propagation from all of the T raw losses {Le), and these gradients represent optimal update directions for individual tasks. As the parameters O can only be updated with single gradient, we should compute an aggregated gradient g by the linear combination
Impartial
Multi task Loarning
Loss
Balance
Heads
Shared Fenture
Gradient
2= f(x,
Backbonn
{g.). It also implies to find the scaling factors {a. of raw losses {L,}. since g = Σ, aggi = VoL = Ve (E, az L.). Motivated by the principle of balance among tasks, we propose to make the projections of onto (gr) } to be equal, Fig. 1 (d). In this way,
Figure 2: Overview of IMTL.
3

---

Published as a conference paper at ICLR 2021
Algorithm Training by Impartial Multi-task Learning
Input: input sample æ. specifpecific labels {y+ learning rate 17 Output: bask-shared/-speciff parameters 0/( J. scale parameters {+t}
1: compute task-shared feature 2 = f (z;; 0)
2: for ft= = 1 to T' do
3: compute task prediction by head network fi (x) = firnt (z; 0.)
4: compute raw loss by loss function I = 1.4 fune (ft (a)) Vc)
5: compute scaled loss I, ba T.
(default = e, 1)
D loss balance
6: compute gradient of shared feature 2: gi V
7: compute unit-norm gradient us llx.
8:
8: end for
9: compute gradient differences D gi 92 ................................ 91 9r
compute unit norm gradient differences U [1 W
u
wr
I: compute scaling factors for tasks 2 to T: a2: = gi U (DU*)
b gradient balance
12: compute scaling factors for all tasks: a =
102
13: update taskes parameters 0 - Vo [ oeLi)
14: for t = 1 to do
15: update task-specific parameters 0: = 0: V, Ve.Li Lr
16: update loss scale parameter 8; St 11 Bsc
BLc
17: end for
we treat all tasks equally so that they progress in the same speed and none is left behind. Formally, let {ur- 9 [[g:1]] denote the unit-norm vector of {gi} which are row vectors, then we have:
gui T gu, + g (u: u.) o, 2 L E.
The above problem is under-determined, but can obtain the closed-form results of {ar} by constraining
E.ar az 1. Assume
[a2, .r). U [u] u2 un u
D
91
92 91 gf1 and = [1,.... 1]. from Eq. I
we can obtain:
a = U (DUT)- (IMTL-G)
(2)
The detailed derivation is in Appendix B.1 After obtaining a, the scaling factor the first task can
be computed by 01 = - la since E, = 1. The optimized {a,) are used to compute L E. anL,, which is ultimately minimized by SGD to update the model. By now, back back-propagation needs to be executed T times to obtain the gradient of each task loss with respect the heavy-weigee task-shared parameters 0, which is time-consuming and non-scalable. We replace the parameter- level gradients {g. V.L. with feature-level gradients [v } to compute (az). This implies to achieve gradient balance with respect to the last shared feature z as a surrogate of task-shared parameters 0, since it is possible for the network to back-prop this balance all the way through the task-shared backbone starting from z. This relaxation allows us to do back propagation through the backbone only once after obtaining {at}, and thus the training time can be dramatically reduced.
3.2 LOss BALANCE: IMTL-L
For the task-specific parameters { we cannot employ IMTL-G described above, because Ve 0, Vt 전 t, and thus only the gradient of the corresponding task Ve. can be obtained for each 0, Instead we propose to balance the losses among tasks by forcing the scaled losses (a,L.} to be constant for all tasks, without loss of generality, we take the constant as 1. Then the most direct idea is to compute the scaling factors as (a, = 1/ I. but they are sensitive to outlier samples and manifest severe oscillations, so we further propose to learn to scale losses via gradient descent and thus stronger stability can be achieved. Suppose the positive losses {L, > 0} are to be balanced, we first introduce a mapping function : R - R to transform the arbitrarily- parameters { to positive scaling factors {h (st) >> 0}. hereafter we abandon the subseript t for brevity. Then we should construct an appropriate sealed loss g (s) so that both network parameters and scale parameter be optimized by minimizing g (s). On hand, we balance different
leamable scale
4

---

Published as a conference paper at ICLR 2021
tasks by encouraging the scaled losses h (s) L (0) to be for all tasks, so the optimality s* of is achieved when h (s) L (0) = 1, or equivalentyy:
L (0) 1 0, if =
(3)
One may expect to minimize [+)) (+) (0) to find s", however when h (5) L (0) < 1, the gradient with respect to 0, Ve / (8) = -h (s) V. (0). is in the opposite direction. On the other hand, assume our scaled loss g (s) is a differentiable convex function with respect to s, then its minimum is achieved if and only if s s", where the derivative of g (s) is zero:
o (6) - 0, 8-
(4)
From Eq. (3) and we find that the values of (s) and g' (s) are both 0 when s", we can then regard f (s) as the derivative of g (s), which is our target scaled loss and used to optimize both the network parameters O and loss scale parameter 8, then we have:
(g())= ( L(()) / (+) -8
(5)
From Eq. 33 and ( we notice that both h (s) and f (s) ds denote loss scales, so we have (s) Ch (s), where C >> a constant. According to ordinary differential equation,
(s) must be the exponential function: J (s) ds ba with a 1, > (see Appendix B.2). We then have g" (s) ka", k >> O. which is always positive and verifies our assump- tion about the convexity of g (s). Also note that the gradient g (s) with respect to 9, Veg (s)= h (s) VeL (0) ba" VeL (0), is in the appropriate direction since ba O. As instantiation,
we set ds e² (a e, b = 1), then
(s) = e² L (0)
(IMTL-L).
From Eq. (6) we find that the raw loss is scaled by ex, and -s acts as a regularization to avoid -0 while minimizing the scaled loss g (s). As for implementation, the
the trivial solution task losses { } are scaled by {e*(), ), and the scaled losses fer L - st} are used to update both the network parameters 0, {0.} and the scale parameters {
3.3 HYBRID BALANCE: IMTL
We have introduced MMTL-G/MMLL- scaling factors to be applied on the raw losses. They can be used solely, but we find them comple- mentary and able to be combined to improve the performance. In IMTIL-G even if the raw losses are multiplied by arbitrary (maybe different among tasks) positive factors, the direction of the ag- gregated gradient g stays unchanged. Because by definition g = Σ. aggr is the angular bisector of the gradients {g:). and positive scaling will not change the directions of {gr} and thus that of (proof in Theorem 2}. So we can also obtain the scale factors {az} in IMTL-G with the losses that have been scaled by {ra} from IMTL-L. IMTL-G and IMTL-L are combined as: 1) the task- specific parameters (0,) and scale parameters {st} are updated by scaled losses {e L, 3: ; 2) the task-shared parameters 0 are updated by Σ, 0g (e* L,)) which is the weighted average of (e L.). with the weights {a:) computed by (Vx [c)) using IMTL-G. Note that the regularization terms {-st} in Eq. (6) are constants with respect to 0 and z, and thus can be ignored when computing gradients and updating parameters in IMTL-G. In this way, we achieve both gradient balance for task-shared parameters and loss balance for task-specific parameters, leading to our full IMTL as illustrated in Alg.
achieve gradient/loss balance, and both of them produce
4 DISCUSSION
We draw connections between our method and previous state-of-the-ards in Fig, 3 that previous methods can all be categorized as gradient or loss balance, and thus each of them be seen specification of our method. However, all of them have some intrinsic biases or short-comings leading to inferior performances, which we try to overcome.
We will show
Our analysis of PCGrad Yu et al. 2020 be found in Appendix C33
5

---

Published as a conference paper at ICLR 2021
avallook dass fizlion bolk. senbutt ssumpptinn
GradNorm (p, ut
Teper sara neterr tun tannggnneded
uncertainty Intakes 1/2,
Brodinn
perpendcubr tull
Cnalire =1]
IMTL-L (aply = const)
complemeter
IMTL-G (p const)
MGDA Up oz |l
Soale mereans
coualisallelloo
(a,1, L/T)
PCGrad (p, x (lg.Il)
usttbri
unformscaling
number's bang
loss balanie
gradient balance
Figure 3: Relationship between our IMTL and previous methods. The blue dashed arrow indicates the characteristic of each method. In the loss balance methods, annotate the scaled loss in the bracket. Les, Leg and Lt are the raw loss of classification, regression and individual task, respectively. Olels Orreg and Or is the corresponding loss scale. L is the geometric mean loss and T is the task number. In the gradient balance methods, we annotate the projections of the aggregated gradient g = Ee agg: onto the raw gradient g, of the t-th task in the bracket. u gt/ [1g: the unit-norm vector, Pt gu, is the projection of g onto gi and u, E. un is the mean direction.
GradNorm (Chen et al. 2018) balances tasks by making the norm of the scaled gradient for each task to be approximately equal. It also introduces the inverse training rate and a hyper-parameter control the strength of approaching the mean gradient norm, such that tasks which learn slower can receive larger gradient magnitudes. However, it does not take into account the relationship of the gradient directions. We show that when the angle between the gradients of each pair of tasks is identical, our IMTL-G leads to the equivalent solution as GradNorm.
Theorem 1. If the angle between any pair of u. uh stays constant: u.u, C1, Vt o T with C1 < 1, then our IMTL-G leads to the same solution as that of GradNorm: gu, = C2 + ne = ||aggll = a4 [lge| = C3. In the above u, = g ||9: C1. C2 and C3 are constants.
Proof in Appendix C.1 In GradNorm, if without the above constant-angle condition u.u, = C the projection of the aggregated gradient g onto task-specific gradient, gu, - -E, Caur) u, C3 (E, u.) u. proportional to (E, u. u.! It tends optimize the "majority tasks" whose gradient directions are closer to the mean direction E, ue resulting in undesired task bias.
MGDA (Snner& & Koltun 2018) finds the weighted average gradient g E, argg: with minimum norm in the convex hull composed by } so that Σ, 04 1 and a₂ > 0, vt. It adopts an iterative method based on Frank- Wolfe algorithm solve the multi-objective optimization problem. We note the minimum-norm point has closed-form representation if without the constraints {ar > 0}. In this case, we try minimize gg" = (Er arg:) (E, a,g;) such that >, at 1. It implies g is perpendicular to the hyper-plane composed by {g:} as illustrated in Fig (b), and thus we have:
9 (91 9t) g (9) 9t)
0, 2 < T,
and can obtain = D (DDT)~* (see Appendix
C.2 From Eq. 7 note that the aggre-
gated gradient satisfies: 99. C. Then the projection of g onto 9t. gu. C/ [|g.]]. is inversely proportional to the norm of 9t- So it focuses on tasks with smaller gradient magnitudes, which breaks the task balance. Even with {az > 0}. the problem still exists (see Appendix C.2) in the original MGDA method. Through experiments, we note that finding the minimum -normummm-ormm -oom point with- out the constraints {a, > 0} leads similar performance as MGDA with the constraints {a, > 0}. In our IMTL-G, although we do not constrain {a, > 0}, its loss weighting scales are always positive during the training procedure as shown in Fig.
Uncertainty weighting (Kendall et al.) 2018) regards the task uncertainty as loss weight. For regres- sion, can derive L, loss from Laplace distribution: - log [x)) = ly - f (a) /b + log b, where æ is the data sample, is the ground-truth label, f denotes the prediction model and b is the di- versity of Laplace distribution. L2 loss can be found in Appendix [ppendix/C.4] For classification, takes the cross-entropy loss as scaled categorical distribution and introduces the following approximation:
f f(()) 02
- logg (y I (x)) = - log softmax,
~
T log [softmax,, (ʃ (f(x)) + log a. (8)

---

Published as a conference paper at ICLR 2021
in which softmax, (-) stands for taking the y-th entry after the softmax (-) operator. MTL corre- sponds to maximizing the joint likelihood of multiple targets, then the derivations yield the scaling factor blo for the regression/classffcation loss. (Kendall et al., 2018) learn b and as model pa- rameters which are updated by stochastic gradient descent. However, it is applicable only if we can find appropriate correspondence between the loss and the distribution. It is difficult be used for losses such as cosine similarity, and it is impossible to traverse all kinds of losses to obtain a unified form for them. Moreover, sacrifices classification tasks. From Eq. cross-entropy loss is approximated as L = c² Les if we set
(8) we can find that the scaled log o. By taking the derivative Then s is optimized to make the scaled loss e²a Les be close to 1/2. However, the scaled L1 loss is approximated as L = e Lee 8 if we set = -log b, and taking the derivative we have / = e² Ieg So a is optimized to make the scaled In loss to achieve 1, which is twice of the classification loss, and thus the classification task is overlooked.
we have 0L/0s = 2c Las
We would like to remark the differences between our IMTL-L and uncertainty weighting (Kendall eet al. 2018). Firstly, our derivation is motivated by the faireess among tasks, which intrinsically difters from uncertainty weighting which is based on task uncertainty considering each task inde- pendently. Secondly, IMTL-L learns to balance among tasks without any biases, while uncertainty weighting may sacrifice classification tasks to favor regression tasks as derived above. Thirdly, IMTL-L does not depend on any distribution assumptions and thus can be generally applied to var- ious losses including cosine similarity, which uncertainty weighting may have difficulty with. As far as we know, there is no appropriate correspondence between cosine similarity and specific dis- tributions. Lastly, uncertainty weighting needs to deal with different losses case by case. it also introduces approximations in order derive scaling factors for certain losses (such as cross-entropy loss) which may not be optimal, but our IMTL-L has a unified form for all kinds of losses.
GLS Chennuupaii et al. 2019) calculates the target loss as the geometric mean: L. = [ L, + then the gradient of L with respect to the model parameters O can be obtained as Appendix C.5 which can regarded as to weigh the loss with its reciprocal value. However, as the gradient depends on the value of L, so it is not scale-invariant to the loss scale changes, Moreover, we find it to be unstable when the number of tasks is large because of the geometric mean computation.
5 EXPERIMENTS
In previous methods, various experimental settings have been adopted but there are no extensive comparisons. As one contribution of our work, we re-implement representative methods and present fair comparisons among them under the unified code-base, where more practical settings are adopted and stronger performances are achieved compared with existing code-bases. The implementations exactly follow the original papers and open-sourced code to ensure the correctness. We rum exper- iments on the Cityscapes (Cordts et al.) 2016),, NYUv2 (Silberman et al. (012)) and CelebA (Liu et al. 2015) dataset to extensively analyze different methods. Details can be found in Appendix D
Results on Cityscapes. From Tab. D we can obtain several informative conclusions. The uniform scaling baseline, which naively adds all losses, tends to optimize tasks with larger losses and gradi- ent magnitudes, resulting in severe task bias. Uncertainty weighting (Kendall et al. 2018) sacrifices classification tasks to aid regression ones, leading to significantly worse results on semantic seg- mentation compared with our IMTL-L. GradNorm (Chen et al.. 2018) is very sensitive to the choice of the syper-parameter ? controlling the strength of equal gradient magnitudes, where the default y 1.5 works well NYUv2 but performs badly on Cityscapes. We find its best option is 7 0 which makes the scaled gradient norm to be exactly equal. MGDA (Sener & Koltun 2018) focuses on tasks with smaller gradient magnitudes. So the performance of semantic segmentation good but the other two tasks have difficulty in converging. In addition, we find our proposed closed- -fom vari- ant without the hard constraints {a, > 0} achieves similar results as the original iterative method. Through the experiments we notice the closed-form solution almost always yields {ar > 0). As for PCGrad (Yu et al. 2020). it yields slightly better performance than uniform scaling because its conflict projection will have no effect when the angles between the gradients are equal or less than T/2. In contrast, our IMTL method, in terms of both gradient balance and loss balance, yields com- petitive performance and achieves the best balance among tasks. Moreover, we verify that the two balances are complementary and can be combined to further improve the performance, with the vi- sualizations in Appendix F Surprisingly, we find our IMTL. can beat the single-task baseline where
7

---

Published as a conference paper at ICLR 2021
Table 1: Comparison between IMTL and previous methods on Cityscapes, semantic segmentation, instance segmentation and disparity/depth estimation are considered. The first group of columns shows the regular results of different methods. The second group shows the results by manually multiply the semantic segmentation loss with 10 before applying these methods. The subscript numbers show the absolute change after scaling the loss to demonstrate the robustness of various methods. The arrows indicate the values are the higher the better (1) or the lower the better (L). The best and runner up results for each task are bold and underlined, respectively.
method
ins.
disp.
sem mloUt JILL
ins. L1 hay
disp.
time s/iter.
mloUt
L +
L1
L1 4رy
buselines
single-task
76.67 58.99
21.61
4.182
uniform scaling
18.13
3,512
1.201
loss balance
uncertainty Kendall et al.
2018
74.91
16.43
2.895
74.00p.9
16.770.34
2.930p.035
1.204
GL.S Chennupati et al.
3019
75.65
17.18
2.953
66,22,
21.093.e1
3.358 405
1.202
IMTL-L
76.89
16.69
2.944
75.55
17.49u.so
2.9720.028
1.202
gradient balance
GradNorm ( = 0)
76.27
17.99
3.195
72.963.31
19.361.37
3.2160.021
1.741
GradNorm Chen et al.
2018
52.17
19.88
4.098
54.232.06 060
20.530.65
4.1080.010
1.742
MGDA (w/o [ay 2
76.95
53.19
6.296
76.360.50
29.0624.18
3.3772.910
1.777
MGDA Sener Koltun
2018
76.56
53.14
6.644
72.354.21
29.3823.76
3,3363.308
1.732
PCGrad
Yu
60.50
17.99
3.450
66.33s.83
17.99u.00
3.3860.064
2.087
IMTL-G (exact)
76.13
17.46
2.979
2.769
IMTL-G
76.52
16.61
2.997
76.060.46
17.52o.91
3.020o.023
1.776
hybrid balance
IMTL
77.00
15.96
2.905
76.56).44
15.85
2.938o.033
1.795
each task is trained with a separate model. Training multiple tasks simultancously can learn better representation from multiple levels of semantics, which can in turn improve individual tasks.
In addition, we present the real-world training time of each iteration for different methods in Tab. As shown, loss balance methods are the most efficient, and our gradient balance method IMTL- adds acceptable computational overhead, similar to that of GradNorm (Chen et al. 2018 and MGDA (Sener & Koltun, (018). It benefits from computing gradients with respect to the shared feature maps instead of the shared model parameters (the row of "IMTL-G (exact)"), which brings similar performances but adds significant complexity due to multiple (I) backward passes through the shared parameters. Our IMTL-G only needs to do backward computation on the shared pa- rameters once after obtaining the loss weights via Eq. (2). in which the computation overhead mainly comes from the matrix multiplication rather than the matrix inverse, since the inversed ma- trix DU E R(T (I- is small compared with dimension of the shared feature x.
As we outperform MGDA (Sener& & Koltun 2018) and PCGrad (Yu et al. 2020) significantly in terms of the objective metrics shown in Tab. we further compare the qualitative results of our hybrid balance IMTL with the loss balance method uncertainty weighting (Kendall et al. 2018) and the gradient balance method GradNorm (Chen et al., 2018) considering their strong performances (see Fig. (6). For depth estimation we only show predictions the pixels where ground truth (GT) labels exist to compare with GT, which is different from Fig. 7 where depth predictions are shown for all pixels. Consistent with results in Tab. [ our IMTL shows visually noticeable improvements especially for the semantic and instance segmentation tasks. It is worth noting that we conduct experiments under strong basclines and practical settings which are seldom explored before, this case changing the backbone in PSPNet (Zhao et al.) 2017) from ResNet-50 to ResNet- 101 can only improve mloU of the semantic segmentation task around 0.5% according to the public code base")
Scale invariance. We are also interested in the scale invariance, which means how the results change with the loss scale. For example, semantic segmentation, the loss scale is different if we replace the reduction method "mean" (averaged over all locations) with "sum" (summed over all locations) in the cross-entropy loss computation, or the number of the interested classes increases. The scale invariance is beneficial for model robustness. So to simulate this effeet, we manually multiply the semantic segmentation loss by 10 and apply the same methods to see how the performances are affected. In the last three columns of Tab. [ we report the absolute changes resulting from the
8

---

Published as a conference paper at ICLR 2021
Table Experimental results on the NYUv2 and CelebA datasets, semantic segmentation, surface normal estimation, depth estimation and multi-class classification are considered. Arrows indicate the values are the higher the better (f) or the lower the better (4). The best and runner up results in each column are bold and underlined, respectively.
NYUv2 norm. cos
CelebA class. acc. 1
method
sem. mloUf
depth L1 /
baselines
single-task
56.82
0.8827
0.5097
uniform scaling
57.40
0.8684
0.4248
90.01
loss balance
uncertainty (Kendall 2018
57.20
0.4400
90.34
GL.S (Chennupati al. 2019
57.84
0.8762
0.4243
IMTL-L
58.36
0.8864
0.4173
90.54
gradient balance
GradNorm (y 0)
55,96
0,8818
0.4317
90.91
GradNorm CCen et al.
2018
56.92
0.8787
0.4285
89.92
MGDA (w/ aa 2 0
49.43
0.8877
0.4839
89.68
MGDA Sener Koltun
2018)
49.44
0.8875
0.4759
90.04
PCGrad Yu et al.
2020
57.48
0.8696
0.4253
89.99 .999
IMTL-G
57.00
0.8785
0.4226
91.03
hybrid balance
IMTL
58.85
0.8888
0.4215
91.12
multiplier. Our IMTL achieves the smallest performance fluctuations and thus the best invariance, while other methods are more or less affected by the loss scale change.
find similar patterns as Cityscapes, but NYUv2 is rather small dataset, so uniform scaling can also obtain reasonable results. Note that uncertainty weight- ing (Kendall et al. 2018) cannot be directly used estimate the normal surface when the cosine similarity is used as the loss, since no appropriate distribution can be found to correspond to cosine similarity. In this case, surface normal estimation owns the smallest gradient magnitude, so MGDA (Sener & Koltun 2018) learns it best but it performs not so well for the rest two tasks. Again, our IMTL performs best taking advantage of the complementary gradient and loss balances.
Results NYUv2. In Tab.
Results on CelebA. To compare different methods in the many-task setting, in Tab. 2 we also conduct the multi-label classification experiments on the CelebA (Liu al.] (015)) dataset. The mean accuracy of 40 tasks is used as the final metric. Our IMTL outperforms its competitors in the scenario where the task number large, showing its superiority. Note that in this setting, GLS (Chennupati et al. 2019) has difficulty in converging and reasonable results can be obtained.
CONCLUSION
We propose an impartial multi-task learning method integrating gradient balance and loss balance, which are applied task-shared and task-specific parameters, respectively. Through in-depth analysis, we have theoretically compared our method with previous state-of-the-arts. We have also showed that those state-of-the-ane can all be categorized as gradient specific bias among tasks. Through extensive experiments we verify our analysis and demonstrate the effectiveness of our method. Besides, for fair comparisons, we contribute a unified code-base base, which adopts more practical settings and delivers stronger performances compared with existing code-bases, and will publicly available for future research.
loss balance, but lead to
ACKNOWLEDGEMENTS
This work was supported by the Natural Science Foundation of Guangdong Province (No. 2020A15150107111 the Special Foundation for the Development of Strategic Emerging Industries of Shenzhen (No. SCY12000010914000000 the Hong Kong Special Administrative Region, China (Enterprise Support Scheme under the Inno- vation and Technology Fund B/E030/18).
and the Innovation and Technology Commission of

---

Published as a conference paper at ICLR 2021
REFERENCES
Rich Caruana. Multitask learning, Machine learning, 28(1):41-75, 1997.
Liang-Chiee Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4): 834 848, 2017.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning. pp. 794-803, 2018.
Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213-3223, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, IEEE, 2009.
Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning. In Proceedings of the tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1099-17,40 2004.
Yuan Gao, Jiayi Ma. Mingbo Zhao, Wei Liu, and Alan Yuille. Nddr-enn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3205-3214 2019.
Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task Vask-agnostic neural architecture search towards general-purpont multi-task learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11543-15522 2020.
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task priori- tization for multitask learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 270-287, 2018.
Pengsheng Guo, Chen- Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference Machine Learning, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and Conference Computer Vision and Pattern Recognition, pp- 713 7132-7141, 2018.
networks. In Proceedings of the IEEE
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning Volume 37, pp. 448 448,456,777 2015.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1871 1880 2019.
10

---

Published as a conference paper at ICLR 2021
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730-3738, 2015.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10437-10446, 2020.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multi- ple tasks by learning to mask weights. Proceedings of the European Conference on Computer Vision (ECCV), pp. 67-82, 82, 2018.
Kevis-Kokins Maninis, Iija Radosavovic, and Iasonas Kokkinos. Attentive single-tasking of multi- ple tasks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851-1860, 2019.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hehert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994-4003, 2016.
Chao Peng, Tete Xiao. Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: A large mini-batch object detector. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6181-6189, 2018.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, pp. 506-516, 2017.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv:1706.05099, 2017.
arXiv preprint
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi-task ar- chitecture learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4822 4829, 2019.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems, pp. 527-538, 2018
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In European Conference Computer Vision, pp. 746-760. Springer, 2012.
Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning. 2020.
Gjorgji Strezoski, Nanne van Noord, and Marcel Worring. Many task learning with task routing. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1375-1384, 2019.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.066 06782, 2020.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3712-3722, 2018
Amir Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjic Cao, Jitendra Malik, and Leonidas Guibas. Robust learning through cross-task consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11197-11206, 2020.
Yu Zhang and Qiang Yang. A survey on multi-task learning. arXiv preprint arXiv: arXiv: 1707.08144, 2017.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881-2890, 2017.
11

---

Published as a conference paper at ICLR 2021
Barret Zoph and Quoe Le. Neural architecture search with reinforeement learning. In Proceedings of the International Conference on Learning Representations, 2017.
A RELATED WORK OF NETWORK STRUCTURE
Cross-stitch Networks (Misra et al. 2016) learn coefficients to linearly combine activations from multiple tasks to construct better task-specific representations. To break the limitation of channel- wise cross-task feature fusion only. NDDR-CNY (Gao et al. 2019) proposes the layer-wis siye-wiiee cross- channel feature aggregation as
convolutions on the concatenated feature maps from multiple tasks. More generally, MTL-NAS (Gao al., 2020) introduces cross-layer connections among tasks to fully exploit the feature sharing from both low and high layers, extending the idea in Sluice Networks (Ruder al., 2019) by leveraging neural architecture search (Zoph & Le 2017). The parameters of these methods increase linearly with the number of tasks. To improve the model compactness, Residual Adapters (Rebufñi et al., 2017) introduce small amount of task musk-specifie parameters for each layer and convolve them with the task aggossiccii representations to form the task- related ones. MTAN (Liu et al. 2019) generates data-dependent attention tensors by task- specifc parameters attend to the task-shared features. Single-tasking (Maninis et al. 2019) instead applies squeeze-and-exxe -xdiicttionn (Hu et al. 2018) module to generate attentive vectors each task. In Task Routing (Strezoski et al. 2019), the attentive vectors randomly sampled before training and are fixed for each image. Piggyback (Mallya et al.) 2018) opts to mask parameter weights place of activation maps, dealing with task task-sharing from another point-of-view: The above methods share parameters among tasks to a large extent, however, they are not memory-effiiient because each task still needs compute all of its own intermediate feature maps, which also leads to inferior inference speed compared with loss weighting methods.
B DETAILED DERIVATION
B. GRADIENT BALANCE: IMTL-G
Here give the detailed derivation of the closed-form solution of our IMTL--G, also demonstrate the scale-invariance property of our IMTL-G, which is invariant to the scale changes of losses. Solution. As we want to achieve:
gui gu, + (ui - uz) = 0, 2 T,
(9)
where u, = 9t || recall that we have g = =. 0.gi and Σ Oz 1, if we set a .. .rl
and G [[ ... .r. .r. then an la and Eq. ( can be expanded as:
Σ augi [u] T U2 T
/ uT = 0 +
1 - 1- a 0 U - [
(10)
where T
[u. 22
uT] 1 and 0 indicate the all-one and all-zero row vector,
respectively. Eq. (10) can be solved by:
[(1 la 1) gi + aG] =
=0 a - =
(11)
Assume D = - GT = =99T ...... .gT ..
94
then we reach:
DDUT U + a UIU (DUT)-*
(12)
Property. We can also prove the aggregated gradient = E, Ongr with (ax) given in Eq. 12 is invariant to the scale changes of losses [T.}] (or gradients (g. Vol.)), as the following theorem.
Theorem 2. Given 9 E, a.gi. E, ar (kr > 0} (equivalently, }g.} are scaled by {ke)), ifg' Ee of (kig:), E, af satisfies g' "
satisfying gu,"
C, when {Tt} are scaled by
C", then g' = >g. In the above we have u.
9r
kegt
X. C and C' are constants.
l
Tkg..
12

---

Published as a conference paper at ICLR 2021
0.900
0.0600
0.970
0.060
8.0000-3
0.0400
0.950
0.940
4.000e-3
0.0200
0.00
0.90
0,000 4 000k
8.000k
12.00k
16.00k
0.000
6.000k
12.00k
18, 18.00k
0.000
4.000k 8.000k 12 ook 16.00k Disparity Estimation
Semantic Segmentation
Instance Segmentation
Figure 4: Loss scales of IMTL-G for different tasks when training on the Cityscapes dataset.
Proof. As we have:
Σ 0ng
and gu, C, kig:
(13)
ki
by constructing:
d
a1 Rit
Qr
and g' Σ af (kigt) =9/ Σ ar Ag.
(14)
Kr
we have:
Ed =1 and g'u, c/ =C a4 Kr
(15)
From Eq. 12) we know that {ox) has unique solution, and thus g' satisfying IMTL-G is unique, so it must be the one given by Eq. (14). then we can prove that g' and g are linearly correlated. L
B.2 LOSS BALANCE: IMTL-L
With the ordinary differential equation, we can derive that the form of the scale function f (8) ds our IMTL-- must be exponential function. As we have:
f h ds Ch C> O.
(16)
we set y = ffh. h (s) ds, then:
= C dy
ds 7 dy ds.
(17)
c
By taking the antiderivative:
[ dy 1 f ds = Iy
1 C
(18)
y
Then we have:
/ y o (et)² = bo. a 1, b> 0.
(19)
C DETAILED DISCUSSION
C.I CONDITIONAL EQUIVALENCE OF IMTL-G AND GRADNORM
First we introduce the following lemma.
Lemma 3. If u C1, Vt # т, then the solution {ai) of IMTL-G satisfies {ar 0}.
13

---

Published as a conference paper at ICLR 2021
Proof. As u, gt/ 11 by constructing 9 E. aggi where:
02 = 119el Σ g- 10-1-
(20)
then we have E, 04 = and:
gul Σ ur Ue
Σ 19-1 - 1IC + +1] / || C2.
(21)
From Eq.
we know the solution {ar} of IMTL-G is unique, so it must the one given by Eq. where {a 0), the lemma is proved.
20
Then we prove Theorem | which states that IMTL-G leads to the same solution as GradNorm when the angle between any pair of gradients {g.} is identical: u.u, C1, Vt # T.
Proof. (=> Necessity) Given constant projections in IMTL-GG, we have:
gu! Σ
9r u, C2-
(22)
Recall that u, 9:/ ||gel| and u.u, C1, AI + r. From Lemma 3 we know that {ax) given by IMTL-G must satisfy {ar > 0}. If we assume ni = |a:g:]| then we know a:g: = R2 and:
Σ u.uuu
"_En n,C1 C2-
(23)
T/
Now we obtain:
Σ + Σ n - Cz. T2
(24)
As Ci < I, can then prove n4 Cz, Vt. implies the norm of the scaled gradient is constant, which is requested by GradNorm (Chen et al. 2018). Moreover, we can obtain the relationship among constants from Eq. 24
C2 - 1
= C + =2
(25)
(= Sufficiency) GradNorm, {au} are always chosen to satisfy faz > 0}, so if we assume 7y = Ja.gel,, then given the constant norm of the scaled gradient in GradNorm, have:
agg: n44 Czul,
(26)
where u, gt/ 19:1|- As we have g E, aggt and u C1, Vt & then obtain:
gu. --aagg
Σ a.gr u: E Cour u,.
C2. means the projections g onto {gt} are constant, which requested by our IMTL-G.
(27)
Corollary 4. In GradNorm, if the solution {ar) satisfies E, a 1, then its constants given by C3 = 1/ E, |Igel| and C2 = [[T - C1 + 11 E, [lg.1 and its scaling factors are given by aa = || E, ||g,
Proof. By using at = Ca/ ||g/|| from Eq. 26
we have Σ, C3/ || = 1. then C3 /E, ||g -| As the relationship of C2 and C3
1/ E, [lg: I_, and also we have |g
from Eq. (2) is given by Ca [T - 1) C2, so C2 [[T C1 + 1)/2, ||9:1] O
14

---

Published as a conference paper at ICLR 2021
C.2 CLOSED-FORM SOLUTION OF MGDA
In our relaxed MGDA (Sener & Koltun 2018) without {as d 0}. finding = Eg arg with
E, on 1 such that ! has minimum norm equivalent find the normal vector of the hyper- plane composed by {gt). So we let to be perpendicular to all of {g1 - -g on the hyper-plane:
g (91 gt) + 991 -g:)
A
T.
(28)
we set a = [ar,,... ... aT] and GT can be expanded as:
.99 then we have æi = 1 1a T, and Eq.
Σ a.gtt
|6 -
9 - - = 00 + [ 11-1a a
91 G
D - 0, (29)
where D
| 9 .... 9 -9T and 0 indicates the all-one and all-zero row vector. Eq. can be represented as:
[(1 - 1a gi + aG] DJ = 0 + a ="=
1 As also have 1 91 G, then the closed-form solution
of solutioon
is given by:
aDD DD a D (DDT)-
(30)
Bias of MGDA. In the main text we state that MGDA focuses on tasks with small gradient mag- nitudes, where we relaxed MGDA by not constraining {a, e 0). However, even with these con- straints, the problem still exists. For example in the context of two tasks, assume llg < ||g2|l. if the minimum-norm point of g satisfying g = agi + (1 a) g2 is outside the convex hull composed by {g or equivalently a 1, MGDA clamps to the projections of g" onto 91 and 92 will be ||g/l| and 9a uz (u2 = g2/ g2/llgal),, respectively. As Iigi u so MGDA still focuses on tasks with smaller gradient magnitudes.
and the optimal g 91. Then
C.3 ANALYSIS OF PCGRAD
PCGrad Yu al. 2020) mitigates the gradient conflicts by projecting the gradient of one task the orthogonal direction of the others, and the aggregated gradient can be written as:
g -
Σ g: + [cru))
(31)
with u, 9.///l9 and the coefficients:
|- [-(0 9t + Σ
|
Cu = 0, Cir =
u
Vl, T;
(32)
where [+,, means the ReLU operator. Note that the tasks have been shuffled before calculating the aggregated gradient to achieve expected symmetry with respect to the task order. Eq. (31) can be represented more compactly in the matrix form:
g = 1 +CI++CN)G==G
(33)
where IT is the identity matrix, C (Cer] is the coefficient matrix whose entries are given in Eq. is the diagonal normalization matrix. In Eq. (33 we G and a denote the gradients and scaling factors of all tasks. We find that PCGrad can also be regarded as loss weighting. with the loss weights given by a = 1 1 + CN). However, still may break the balance among tasks. For example with two tasks, assume the angle between
32 and N = diag (1/ [91 (9.ll......1//lgg /|rgr]
15

---

Published as a conference paper at ICLR 2021
0
9ig |1221l
the gradients is Φ: 1) when 7/2 & ø < n, then C =
and the
91 91
91
projections onto the two raw gradients are 9 sin Φ and g2 sin 6; 2) when 0 < Φ < π 7/2, then C and the projections are || + |ga cos and g ll || cos Φ. In both cases, the projections are equal if and only if ||g will be trained more sufficiently, which may encounter the same problem uniform scaling that naively adds all the losses despite that the loss scales are highly different.
gall ll. Otherwise, the task with larger gradient magnitude
C.4 L2 Loss IN UNCERTAINTY WEIGHTING
For regression, uncertainty weighting (Knddall et al. 2018) regards the L2 loss as likelihood esti- mation on the sample target which follows the Gaussian distribution:
log (y fx)) = ( ll + log
(34)
where the data sample, y the ground-truth label, f denotes the prediction model and a the standard deviation of Gaussian distribution. By setting 8 log a² the scaled T.2 loss is = 5 Lee - s), which has a similar form as the scaled L1 loss except the front factor 1/2. So uncertainty weighting has difficulty in reaching unified form for all kinds of losses, which is less general than our IMTL-L.
C.5 GRADIENT OF GEOMETRIC MEAN
GLS (Chennupati et al. 2019) computes the loss as the geometric mean, its gradient with respect to model parameters are:
+-1
Vol
1
Tut
Σ
Il
Volu
(35)
T
T+t
1 T
VoLn
+ I T
L
Tu Σ
(VoT.).
(36)
where I. is the geometric mean loss and T is the task number. It is equivalent to weigh the task- specific loss with its reciprocal value, except that there exists another term L/T in the front where
I. (II T, so GL. is sensitive the loss scale changes of {T.,) and not scale-invariant.
D IMPLEMENTATION DETAILS
To solely compare the loss weighting methods, we fix the network structure and choose ResNet- 50 (He et al., 2016) with dilation (Chen et al.) 2017) and synchronized (Peng et al. normalization (loffe& & Szegedy (015) as the shared backbone and PSPNet (Zhao al. 2017 the
2018 batch
task-specific head, and the backbone model weights are pretrained on ImageNet (Deng et al. 2009 Following the common practice of semantic segmentation, training we adopt augmentations as random resize (between 0.5 to 2), random rotate (between -10 to 10 degrees), Gaussian blur (with a radius of 5) and random horizontal flip. Besides, we apply strided cropping and horizontal flipping as testing augmentations. The predicted results in the overlapped region of different crops are averaged to obtain the aggregated prediction of the whole image. Only pixels with ground truth labels are included in loss and metric computation, while others are ignored. Semantic segmentation, instance segmentation, surface normal estimation and disparity/depth estimation are considered. As for the losses/metrics, semantic segmentation uses cross-entropy//llo,, surface normal estimation adopts (1 - cos) /cosine similarity and both instance segmentation and disparity/depth estimation use L1 loss. We use polynomial learning rate with power of 0.9, SGD with momentum of 0.9 and weight 1
decay of 10-4 as the optimizer, with the model trained for 200 epochs. After passing through the shared backbone where strided convolutions exist, the feature maps have 1/8 size as that of the
16

---

Published as a conference paper at ICLR 2021
semantic segmentation
foreground masking
foreground extraction
regression
computattion
genmenn
disparity astimation
Figure 5:
Pipeline used in the Cityscapes visual understanding experiment. The centroids are com- puted from the offset regression results. Each pixel is assigned to its nearest candidate centroid.
input image. Then the results predicted by PSPNet (Zhao et al. 2017) heads are up-sampled to the original image size for loss and metric computation.
For the Cityscapes dataset, the batch size is 32 (2 x 16 GPUs) with the initial learning rate 0.02. We train on the 2975 training images and validate on the 500 validation images (1024 2048 full resolution) where ground truth labels are provided. Three tasks are considered, namely semantic segmentation, instance segmentation and disparity/depth estimation. Training and testing are done on 713 x 713 crops. Semantic segmentation is to differentiate among the commonly used 19 classes. Instance segmentation is taken as offset regression, where each pixel p: = (I,, y:) approximates the relative offset o, (dz, dy.) with respect to the centroid Cd(í;)) of its belonging instance id (p,). To conduct inference, we abandon the time-consuming and complicated clustering methods adopted by the previous method (Kendall et al., 2018). Instead, directly predicted by the model to find the centroids of instances. By definition, the norm of a centroid's offist vector should be 0, so we can transform the offset vector norm ||o,]| to the probability qi of 7 edge filter is applied
the offset vectors {o:)
being centroid with the exponential function 9i e looll, Next on the centroid probability map to filter out the spurious centroids on object edges resulting from the regression target ambiguity. The locations with centroid probability 9i 0.1 are also manually suppressed. Then 7 x 7 max-pooling on the filtered probability map is used to produce candidate centroids and filter out duplicate ones. With the predicted centroids {c:]. we can then assign each pixel P; to its belonging instance id (p.) by the distance between its approximated centroids p, o, and the candidate centroids }c id (p.) = arg min, Ip. + o - c,ll. Depth is measured in pixels by the disparity between the left and right images. Fig. 5 shows the whole process. Note that we need to carefully deal with label transformation during data augmentation. For example, disparity ground truth needs to be up-scaled by times if the image is up-sampled by times. Also, the predicted offset vectors of the flipped input should be mirrored to comply with the normal one.
On the NYUv2 dataset, the hatch size is 48 (6 8 GPUs) with the initial learning rate 0.03. We use the 795 training images for training and the 654 validation images for testing with 480 x 640 full resolution. 401 401 crops are used for training and testing. 13 coarse-grain classes are considered in semantic segmentation. The surface normal is represented by the unit normal vector of the corresponding surface. When doing data augmentation, surface normal ground truth n (I, y. 2) should be processed accordingly. If we resize the image by times, the coordinate of the normal vector should be scaled by and renormalized: n' = (xx y, s2) [1( y. sz)]l- the image is rotated by the rotation matrix R, the normal vector should also be in-plane rotated ue',y)) = (I. R. with z unchanged. Moreover, the left-right flip should applied on the normal vector n' = (-z, 2) when mirroring the image horizontally. During testing, the normal vectors in the overlapped region of crops are averaged and renormalized to produce the aggregated results. Depth is the absolute distance to the camera and measured by meters, which is inverse- svverseprropottonaa to the disparity measurement adopted by Cityscapes. So the depth in meters needs to be scaled by 1/s when the image is scaled by times, which is the reciprocal of disparity transformation.
CelebA contains 202,599 face images from 10,177 identities, where each image has 40 binary at- tribute annotations. We train the 162,770 training images and test the 19,867 validation
17

---

Published as a conference paper at ICLR 2021
images. Most of the implementation details the same as those the Cityscapes dataset, except that: 1) we employ the ResNet-18 18 as the backbone and linear classifiers as the task lask-specific heads, so totally 40 heads are attached the backbone 2) the binary-cross entropy is used as the clas- sification loss for each attribute; 3) the batch size 256 (32 x 8 GPUs) and the model is trained from scratch for 100 epochs; the input image has been aligned with the annotated 5 landmarks and cropped to 218 X 178.
E QUALITATIVE RESULTS
18

---

Published as a conference paper at ICLR 2021
Uncertainty
GradNorm
IMTL (Ours)
Ground Truth
Uncertainty
GradNorm
Ground Truth IMTL (Ours)
Uncertainty
GradNom
IMTL (Ours)
Ground Truth
Figure 6: Qualitative comparisons between our IMTL and previous methods on Cityscapes.
19

---

Published as a conference paper at ICLR 2021
Figure 7: Qualitative results of our IMTL on Cityscapes. Semantic segmentation, instance seg- mentation and disparity estimation predictions are produced by single network. The task-shared backbone ResNet-50 and the specifc heads are PSPNet. The image resolution is x024x2045
x
20

---

Published as a conference paper at ICLR 2021
Figure 8: Qualitative results of our IMTL on NYUv2. Semantic segmentation, surface normal estimation and depth estimation predictions are produced by a single network. The task-shared backbone is Re and the task-specific heads are PSPNet. The image resolution is 480 x 640.
21


Reviews:
[{"review_id": "n3QLXplMU9l", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper presents a satisfying solution to the open problem of how to train all tasks at approximately the same rate in multi-task learning. There has been a bunch of work on this problem in the last few years. This paper characterizes existing work w.r.t. the fairness of training across tasks in order to motivate two new methods, one applied to shared parameters and the other to task-specific parameters, which overcome the shortcomings of previous methods. The two new methods can be naturally combined to yield a complete method for fair training. Experiments on common MTL benchmarks show the new method compares quite favorably to previous approaches.\n\nThe methods are developed in a theoretical manner from a reasonable set of assumptions. Despite the theoretical derivation, the resulting methods are straightforward to implement, especially compared to more complex iterative or probabilistic methods. Due to its high performance and simplicity of implementation, my expectation is that this new method could quickly become a standard tool for multi-task learning applications.\n\n\nStrong points:\n\nThe paper is overall well-written and well-motivated in the literature.\n\nThe paper addresses an important open problem in multi-task learning.\n\nThe theoretical derivations are well-motivated based on fairness desiderata.\n\nThe resulting methods are effective and simple-to-implement.\n\nThe theoretical and conceptual comparisons to existing methods make it clear how the paper unifies and extends the existing literature.\n\nThe experiments are convincing, and the value of each aspect of the method is independently confirmed.\n\n\nWeak points:\n\nThe paper could benefit from a discussion of the limitations of using fairness as a proxy for MTL performance. Fairness is an intuitive requirement, but one can imagine cases where some tasks should be trained more than others, e.g., if some tasks begin to overfit at higher losses than others.\n\nThe motivation is framed as addressing the problem of some tasks being sacrificed/undertrained. However, there is a common and closely-related issue of some tasks overfitting before others are satisfactorily trained. So, the more general question is how to synchronize the state of training so that all tasks reach peak performance at the same time. Does the approach in the paper address this more general question? Or is it only effective in the setting where overfitting is not an issue because the datasets are so large?\n\nFor example, in the paper the comment “NYUv2 is a rather small dataset, so uniform scaling can also obtain reasonable results” emphasizes that underfitting is the focus, but if the dataset is so small, could uniform scaling lead to overfitting that IMTL could address?\n\nIn the experiments, are all the key details of the existing methods reimplemented, or are some of them taken from official implementations? This should be made clear in the paper and in the released code.\n\n\nMinor comments:\n\nIn the first paragraph of the introduction, are Zamir et al. 2018 and 2020 really the key references for MTL overall? There is no evidence from the introduction that MTL was around before 2018.\n\nIn the first full sentence after Eq. 8: “corresponds to maximize the” -> “corresponds to maximizing the”.\n\nIn “Results on Cityscapes”: “Surprisingly, we find our IMTL can beat the single-task baseline where each task is trained on a separate model.” I don’t think this is surprising. By your definition of MTL in the introduction, this is what you expect if you do MTL right. However, it is somewhat surprising that none of the other MTL methods exceed the single-task baseline on all three tasks; this highlights the fact that they are missing something.\n\nIn Appendix D: “The output size is initially 8x down-sampled and later up-sampled…” Why?\n\n---------------------\n\nUpdate: The authors have adequately addressed my questions, and I am happy to maintain the rating of 7.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UK-bz9wIH-k", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an impartial multi-task learning approach to balance gradient and loss of multiple tasks. Since biased learning can degrade learning efficiency and this work tries to make a balance in a principled way. The approach is simple but looks effective.\nHowever, the gradient balance adopts the last shared feature instead of weights for faster training. From this, I wonder whether this kind of approximation decreases the capability of the weight-level balancing strategy. Theoretical or empirical analyses on this make the proposed method stronger.\n\nThe proposed method gives comparable performance compared to other strong competitors but the gap is not remarkable. I wonder if this performance gap can deserve the attention of practitioners. It would be great for more rigorous analyses to show qualitative results and how this approach differentiates it from others. The proposed method needs to show its potential further.\n\nIn addition, if this approach applies to multiple datasets (as multiple tasks, for example, visual decathlon challenge), the proposed approach may not work well due to the different characteristics when we train them using a hard parameter sharing approach. For a general approach, it would be great to analyze a variety of scenarios.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "98OXPKcMmFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to balance multi-task training using IMTL-G on the shared backbone and IMTL-L on the task-specific branches. IMTL-G enforces equal gradient projections between tasks with a close-form formulation to calculate the desired gradient weightings $\\alpha$. IMTL-L learns the loss weightings $e^s$ with a regularization term $-s$. Additional constraint by making all loss weightings sum to one is used. The paper compares the effectiveness of the proposed IMTLs with their counterparts on Cityscapes, NYUv2, and CelebA and claims state-of-the-art performance.\n\nFirst of all, the authors start out by claiming multi-task learning architecture improvement will lead to high inference cost (Sec 2 first paragraph), so they decided to investigate loss weighting balancing. However, methods have been developed to explore better task hierarchies to design better task grouping in multi-task network architectures in recent publications [R1, R2]. They show SOTA results without high inference cost. In fact, these works also show that better task grouping in network design can already alleviate the need to use sophisticated loss balancing tricks. Although multi-task network design and loss balancing can be orthogonal efforts, it is important to have reasonable motivation and fair discussion.\n\nThe authors claim state-of-the-art performance on all tasks. However, the proposed IMTL method obtained 91.12% accuracy on CelebA where method in [R2] obtained 91.62% accuracy. Therefore, it is incorrect to claim SOTA in the submission. \n\nEquation (2) requires expensive matrix inverse operation for every iteration during training. How much additional computational overhead is needed compared to other straightforward gradient balancing approaches? It would be great to have a clear time complexity comparison with real world runtimes measured.\n\nThe authors claim the proposed IMTL-L does not require any distribution assumption, resulting in the obtained loss weighting $e^s$ with a regularization term $-s$. The proposed formulation is actually quite similar to the original uncertainty weighting approach (Kendall et al. v1 eq.11) with the Gaussian distribution assumption. The only difference is the 0.5 scale for the first term. This similarity also reflected in the experimental results that in Table 1 we can see IMTL-L performs quite similarly to uncertainty weighting (Kendall et al). Uncertainty weighting has even better performance on instance segmentation and disparity compared to the IMTL-L. Can the authors discuss in more details the differences between the two approaches? \n\nBesides, I found Algorithm 1 on page 4 is quite straightforward. It would be great to use the space to further analysis the difference between the proposed IMTL-L and the uncertainty weighting approach (Kendall et al).\n\n[R1] Standley et al. Which Tasks Should Be Learned Together in Multi-task Learning? In ICML 2020. \\\n[R2] Guo et al. Learning to Branch for Multi-Task Learning. In ICML 2020.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is proposing a multi-task learning approach extending existing weighting approaches. An important and novel contribution of the paper is separating the magnitude and direction information in gradient based information. The joint gradient direction is searched by using angle bisectors of task gradients and magnitude is searched by simply finding a scaling which results in uniform loss scales. This approach solves issues like small gradient norm bias of MGDA, etc. The proposed method works well and authors show that it is conceptually relevant to most of the existing algorithms. These conceptual unification is a strong contribution of the paper. The paper is reviewed by three reviewers and received both accept and reject scores. Specifically,

- R#2: Championed the paper and argued for its acceptance
- R#3: Argues that the novelty is limited and SOTA claim is problematic.
- R#4: Argues that the gap between the empirical performance of the proposed method and existing algorithms is small. 

Arguments on the empirical performance and the SOTA are irrelevant to the decision since ICLR does not require algorithms to be SOTA or performed significantly better. Hence, the remaining issues are: claim of the SOTA being true or not, and lack of novelty. I read the paper in detail and decided to accept it with the following comments about the reviews:
- The paper is clearly novel. Direction and magnitude are first time treated separately. Moreover, resulting unification of the existing approaches and theoretical derivations of the important connections of existing methods are also significant.
- The SOTA claim of the paper is technically correct but little misleading. I would recommend authors to simply rephrase it "proposed method outperforms existing methods loss weighting methods under the same experimental settings". The reason for this is the fact that; in principle, "art" includes every possible solution for that problem. Hence, claiming SOTA in a fair and limited evaluation is rather misleading.

In addition to the reviewer comments, here are additional issues which should be addressed by the camera-ready deadline:
- I think the discussion about MGDA is a bit problematic since removing $\alpha \geq 0$ assumption simply removes the Pareto stationarity guarantee of the method. The resulting direction can increase some loss function and this disagrees with the main point of the Pareto optimality. Hence, I would recommend authors to clarify this while making the connection. Frank-Wolfe algorithm is also not really inefficient and unstable since the problem is quadratic with linear constraints and the stability as well as extremely quick convergence can trivially be proved.
- In addition to the previous point, the proposed method can actually increase some loss functions as there is no consistency constraint enforced. This is an interesting observation and empirical results suggest that increasing loss of some objectives might actually be valuable. I think this observation deserves some discussion even in the introduction.


Decision: Accept (Poster)