RotoGrad: Gradient Homogenization in Multitask Learning
Authors: Adrián Javaloy
Isabel Valera
Venue: ICLR 2022 Spotlight (2022)
Keywords: multitask learning
conflicting gradients
negative transfer

Abstract: Multitask learning is being increasingly adopted in applications domains like computer vision and reinforcement learning. However, optimally exploiting its advantages remains a major challenge due to the effect of negative transfer. Previous works have tracked down this issue to the disparities in gradient magnitudes and directions across tasks, when optimizing the shared network parameters. While recent work has acknowledged that negative transfer is a two-fold problem, existing approaches fall short as they only focus on either homogenizing the gradient magnitude across tasks; or greedily change the gradient directions, overlooking future conflicts. In this work, we introduce RotoGrad, an algorithm that tackles negative transfer as a whole: it jointly homogenizes gradient magnitudes and directions, while ensuring training convergence. We show that RotoGrad outperforms competing methods in complex problems, including multi-label classification in CelebA and computer vision tasks in the NYUv2 dataset. A Pytorch implementation can be found in https://github.com/adrianjav/rotograd.

Published as a conference paper at ICLR 2022
ROTOGRAD: GRADIENT HOMOGENIZATION IN MULTITASK LEARNING
Adrián Javaloy
Isabel Valera
Department of Computer Science
Department of Computer Science
Saarland University
Saarland University
Saarbrücken, Germany
Saarbrücken, Germany
wnw.uuiiiaaaa. rawaaarlanddddee ajavaloyecs, wnt.oyyyyc.uuuisnaaaaeeeee
ABSTRACT
Multitask learning is being increasingly adopted in applications domains like computer vision and reinforcement learning. However, optimally exploiting its ad- vantages remains a major challenge due to the effect of negative transfer. Previous works have tracked down this issue to the disparities in gradient magnitudes and directions across tasks when optimizing the shared network parameters. While recent work has acknowledged that negative transfer is two-fold problem, existing approaches fall short. These methods only focus on either homogenizing the gradi- ent magnitude across tasks; or greedily change the gradient directions, overlooking future confficts. In this work, we introduce RotoGrad, an algorithm that tackles neg- ative transfer as a whole: it jointly homogenizes gradient magnitudes and directions, while ensuring training convergence. We show that RotoGrad outperforms compet- ing methods in complex problems, including multi- classification in CelebA and computer vision tasks in the NYUv2 dataset. A Pytorch implementation can found https: //ttbbb. com/ adriaadaaaaaaa /
INTRODUCTION
As neural network architectures get larger in order to solve increasingly more complex tasks, the idea of jointly learning multiple tasks (for example, depth estimation and semantic segmentation in computer vision) with a single network is becoming more and more appealing. This is precisely the idea of multitask learning (MTL. (Caruana, 1993), which promises higher performance in the individual tasks and better generalization to unseen data, while drastically reducing the number of parameters (Ruder, 2017).
Unfortunately, sharing parameters between tasks may also lead to difficulties during training as tasks compete for shared resources, often resulting in poorer results than solving individual tasks, a phenomenon known as negative transfer (Ruder, 2017). Previous works have tracked down this issue to the types of differences between task gradients. First, differences in magnitude across tasks can make some tasks dominate the others during the learning process. Several methods have been proposed to homogenize gradient magnitudes such MGDA-UB (Sener & Koltun, 2018), GradNorm (Chen et al., 2018), or IMTL-G Liu et al. (2021b). However, little attention has been put towards the second source of the problem: conflicting directions of the gradients for different tasks. Due to the way gradients are added up, gradients of different tasks may cancel each other out if they point to opposite directions of the parameter space, thus leading a poor update direction for a subset or even all tasks. Only very recently handful of works have started to propose methods to mitigate the conflicting gradients problem, for example, by removing conflicting parts of the gradients (Yu et al., 2020), or randomly 'dropping' some elements of the gradient vector (Chen et al., 2020).
In this work we propose RotoGrad, an algorithm that tackles negative transfer as a whole by ho- mogenizing both gradient magnitudes and directions across tasks. RotoGrad addresses the gradient magnitude discrepancies by re-weighting task gradients at each step of the learning, while encourag- ing learning those tasks that have converged the least thus far. In that way, it makes sure that no task is overlooked during training. Additionally, instead of directly modifying gradient directions, RotoGrad smoothly rotates the shared feature space differently for each task, seamlessly aligning gradients in

---

Published as a conference paper at ICLR 2022
2
5
NAsrRrad (Tank 1)
Vanilla
T22
RonGGrad
21
21
2
f2 (b) Non-convex experiment.
(a) Convex avocado-shaped experiment.
Figure I: Level plots showing the evolution of two regression MTL problems with/without RotoGrad, see $4. RotoGrad is able to reach the optimum (X²) for both tasks. (a) the space of z, RotoGrad rotates the suuction-spaees to align task gradients (blue/orange arrows), finding shared features (green arrow) closer to the (matched) optima. (b) In the space of rk. RotoGrad rotates the shared feature: z, providing per-task features F% that better fit each task.
the long run. As shown by theoretical insights, the cooperation between gradient magnitude- and prrection-homogenzzeee extensive experiments empirically demonstrate that RotoGrad leads to stable (convergent) learning, scales up to complex network architectures, and outperforms competing methods in multi-label classification settings in CIFARIO 10 and CelebA, well in computer vision tasks using the NYUv2 dataset. Morcover, we provide a simple-to-us use library to include RotoGrad in any Pytorch pipeline.
ensures the stability of the overall learning process. Finally, we run
2 MULTITASK LEARNING AND NEGATIVE TRANSFER
The goal of MTL is simultaneously learn K different tasks, that is, finding K mappings from common input dataset X E R RND to task-specific set of labels Ys E Y Most settings consider hard-parameter sharing architecture, which characterized by two components: the backbone and heads networks. The backbone uses set of shared parameters, O, to transform each input æ E X into a shared intermediate representation Additionally, each task &- 2,2,... ... K has head network hg. with exclusive parameters Φb. that takes this intermediate feature z and outputs the prediction hxr = hy (2; Φbk) for the corresponding task. This architecture is illustrated in Figure 2, where we have added task-specifh rotation matrices Rg that will be necessary for the proposed approach, RotoGrad. Note that the general architecture described above is equivalent to the one in Figure 2 when all rotations Rs correspond to identity matrices, such that тą = z for all
fæx; 0) Rd. where is the dimensionality of
MTL aims to learn the architecture parameters 0, Φ1, Ø2..... ΦK by simultancously minimiz- ing all task losses, that is, (he yı.) for 1....,K K. Although this a priori a multi- objective optimization problem (Sener& Koltun, 2018), in practice a single surrogate loss consist- ing of linear combination of the task losses, L Σe Lk. is optimized. While this ap- proach leads simpler optimization problem,
L, (hi((r))
Ri
S0
hez L2(22(F2))
Ra
82
Rs
han Lk(hg (Fx) y
TK
Figure 2: Hard- -arammeter sharing architecture, in- cluding the rotation matrices R3 of RotoGrad. may also trigger negative trunsfer between tasks, hurting the overall MTL performance due to an imbalanced competition among tasks for the shared parameters (Ruder, 2017).
The negative transfer problem can be studied through the updates of the shared parameters 0. At each training step, updated according to linear combination of task gradients, V = Σ. wa VoLg. which may suffer from two problems. First, magnitude differences of the gradients across tasks may lead to a subset of tasks dominating the total gradient, and therefore to the model prioritizing them the others. Second, conflicting directions of the gradients across tasks may lead to update directions that do not improve any of the tasks. Figure shows an example of poor direction updates (left) as well magnitude dominance (right).
2

---

Published as a conference paper at ICLR 2022
In this work, tackle negative transfer whole by homogenizing tasks gradients both in magnitude and direction. To reduce overhead, we adopt the usual practice and homogenize gradients with respect to the shared feature (rather than 0), all tasks share gradient up to that point, Vola = Voz VzLn. Thus, from now on we focus on feature-level task gradients V Lgg-
3 ROTOGRAD
In this section we introduce RotoGrad, a novel algorithm that addresses the negative transfer problem whole. RotoGrad consists of two building blocks which, respectively, homogenize task aask-gradient magnitudes and directions. Moreover, these blocks complement each other and provide convergence guarantees of the network training. Next, detail each of these building blocks and show how they are combined towards an effective MTL learning process.
3.1 TRADIENT-MAGNITUUN HOMOGENIZATION
As discussed in $2, we aim to homogenize gradient magnitudes across tasks, as large magnitude disparities can lead to subset of tasks dominating the learning process. Thus, the first goal of RotoGrad is to homogenize the magnitude of the gradients across tasks at each step of the training.
Let us denote the feature-level task gradient of the k-th task for the n-th datapoint, at iteration t, by In.k V ( kn,k), and its batch versions by G 1g -2.k. .......... IR,k). ........... where B is the batch size. Then, equalizing gradient magnitudes amounts to finding weights w/ that normalize and scale each gradient G2, that is,
C
|| Gxll = Gall Vi + wi G = ||G
G₁ = CUR
Ak.
where U == G2 denotes the normalized task gradient and C' is the target magnitude for all tasks. Note that, in the above expression, C' is free parameter that need to select.
|..
In RotoGrad, we select C' such that all tasks converge at a similar rate. We motivate this choice by the fact that, by scaling all gradients, we change their individual step size, interfering with the convergence guarantees provided by their Lipschitz-smoothness (for an introduction to optimization see, for example, (Nesterov, 2004)). Therefore, we seek for the value of C providing the best step-size for those tasks that have converged the least up to iteration t. Specifically, we set C' to be convex combination of the task-wise gradient magnitudes, C = Σn a | |l. where the weights 01 ......., O'K measure the relative convergence of each task and sum up to one, that is,
0% = ||G*I//||G||
(2)
with Go being the initial gradient of the k-th task, i.e., the gradient at iteration = 0 of the training.
As result, obtain (hyper)parammtee--fee free approach that equalizes the gradient magnitude across tasks to encourage learning slow-converging tasks. Note that the resulting approach resembles Normalized Gradient Descent (NGD) (Cortés, 2006) for single-task learning. which has been proved to quickly escape saddle points during optimization (Murray et al., 2019). Thus, we expect a similar behavior for RotoGrad, where slow-converging tasks will foree quick-converging tasks escape from saddle points.
Whilst the algorithm works well in general, its simplicity also facilitates unfavorable settings. For example, in the presence of noisy tasks that do not progress; or in scenarios where, when one task improves, there is always another task that deteriorates. In Appendix A we show that, when gradients do not conflict in direction with each other (which we pursue next), following the gradient C C U improves all task losses for the given batch. This result, while simple, provides insights in favor of having desideratum of an efficient MTL pipeline the absence of conflicting gradients.
3.2 GRADIENT-DIRECTION HOMOGENIZATION
In the previous subsection, we have shown that avoiding conflicting gradients may not only be necessary avoid negative transfer, but also ensure the stability of the training. In this section
3

---

Published as a conference paper at ICLR 2022
introduce the second building block of RotoGrad, an algorithm that homogenizes task- gradient directions. The main idea of this approach is to smoothly rotate the feature-space & in order to reduce the gradient conflict between tasks-ii following iterations of the training by bringing (local) optima for different tasks closer to each other (in the parameter space). As a result, it complements the previous magnitude-scaling approach and reduces the likelihood of the training to diverge.
In order to homogenize gradients, for each task A: = 1,..., K, RotoGrad introduces a matrix Rg so that, instead of optimizing La (z) with being the last shared representation, we optimize an equivalent (in optimization terms, as it is a bijective mapping) loss function La [Rs 2). As we are only interested in changing directions (not the gradient magnitudes), we choose R& E SO(d) to be rotation matrix' leading to per-task representations т사 = R& z. RotoGrad thus extends the standard MTL. architecture by adding sesc-specifie rotations before each head, as depicted Figure 2.
Unlike all other network parameters, matrices R& do not seek to reduce their task's loss. Instead, these additional parameters are optimized to reduce the direction conflict of the gradients across tasks. To this end, for each task we optimize R3 to maximize the batch-wise cosine similarity or, equivalently, to minimize
C = - In.k, Un),
where In.k == Tr L& (x.) Un,k)) (which holds that 9r.k = R gn.k) and vn is the target vector that we want all task gradients pointing towards. We set the target vector v, to be the gradient we would have followed if all task gradients weighted the same, that is, Vn == K Σn Un,k, where Un is a row vector of the normalized batch gradient matrix Un, as defined before.
As a result, in each training step of RotoGrad we simultaneously optimize the following two problems:
Network: minimize 0.(+)
Σ Wi Lk.s
Rotation: minimize
Σ Prot
(4)
{Re}*
The above problem can be interpreted as Stackelberg game: a two player-game in which leader and follower alternately make moves in order minimize their respective losses, I, and L,,, and the leader knows what will be the follower's s response to their moves. Such an interpretation allows us to derive simple guidelines to guarantee training convergence-that is, that the network loss does not oscillate as a result of optimizing the two different objectives in Equation 4. Specifically, following Fiez et al. (2020), we can ensure that problem 4 converges as long as the rotations' optimizer (leader) is a slow-learner compared with the network optimizer (follower). That is, as long as we make the rotations' learning rate decrease faster than that of the network, we know that RotoGrad will converge to local optimum for both objectives. A more extensive discussion can be found in Appendix B.
3.3 ROTOGRAD: THE FULL PICTURE
After the two main building blocks of RotoGrad, we can now summarize the overall proposed approach in Algorithm 1. At each step, RotoGrad first homogenizes the gradient magnitudes such that there is no dominant task and the step size is set by the slow-converging tasks. Additionally, RotoGrad smoothly updates the rotation matrices-using the local information given by the task gradients to seamlessly align task gradients in the following steps, thus reducing direction conflicts.
3.4 PRACTICAL CONSIDERATIONS
In this section, we discuss the main practical considerations to account for when implementing RotoGrad and propose efficient solutions.
Unconstrained optimization. As previously discussed, parameters R& are defined as rotation matrices, and thus the Rotation optimization in problem 4 is constrained problem. While this would typically imply using expensive algorithms like Riemannian gradient descent (Absil et al., 2008), we can leverage recent work on manifold parametrization (Casado & Martínez-Rubic 2019) and, instead, apply unconstrained optimization methods by automatically parametrizing R2 via exponential maps on the Lie algebra of (O)d).
'The special orthogonal group, (O(d), denotes the set of all (proper) rotation matrices of dimension d. Forr example, Geotorch (Casado, 2019) makes this transparent the user.
4

---

Published as a conference paper at ICLR 2022
Algorithm 1 Training step with RotoGrad.
Input input samples X, task labels {Yx}, network's (RotoGrad's) learning rate 1) (Thoto)
Output backbone (heads) parameters 0 (fox)). RotoGrad's parameters {
1: compute shared feature Z ()
2: for k == 1,,, K do
3: compute task-specific loss La = =En Lx (hah Φu), yo.kk)
4: compute gradient of shared feature G₁= = V2Lk
5: compute gradient of task-specific feature GR R&Gs
Treated as constant w.r.t. Ra.
6: compute unitary gradients U₁ Gx/|Gx||
7:
compute relative task convergence o* ||
8: end for
9: make }ax} sum up to one [a), 02.......
,.....,,,,. [KK
[an, (21,02,.. 02...., /x ork
10: compute shared magnitude C = Σ 0 |
11: update backbone parameters
17 V C. U
12: compute target vector V = * Σ* Uk
13: for k ... K do
14: compute RotoGrad's loss L = En RR In.k. Vn))
update RotoGrad's s parameters R; Rs- -roto R2 Li roto
16: update head's parameters Φb = 0b - 11V de Lu
end for
Memory efficiency and time complexity. As need one rotation matrix per task, have to store O(Kd) additional parameters. practice, we only need Kd(d - 1)/2 parameters due to the aforementioned parametrization and, in most cases, this amounts to small part of the total number of parameters. Moreover, as described by Casado & Martinez-Rubio (2019), parametrizing Rg enables efficient computations compared with traditional methods, with time complexity of (d independently of the batch size. In our case, the time complexity is of O ([K'" dd)), which scales better with respect the number of tasks than existing methods (for example, ((*ad) for PCGrad (Yu et al., 2020)). Forward-pass caching and GPU parallelization can further reduce training time.
Scaling-up up RotoGrad. Despite being able to efficiently compute and optimize the rotation matrix Rg. in domains like computer vision, where the size d of the shared representation z is large, the time complexity for updating the rotation matrix may become comparable to the one of the network updates. In those cases, we propose to only rotate subspace of the feature space, that is, rotate only m << d dimensions of z. Then, we can simply apply a transformation of the form rs [ 1:m Im)1:d], where aath denotes the elements of with indexes a, + 1...., b. While there exist other possible solutions, such as using block-diagonal rotation matrices RĄ, we defer them to future work.
ILLUSTRATIVE EXAMPLES
In this section, we illustrate the behavior of RotoGrad in two synthetic scenarios, providing clean qualitative results about its effect on the optimization process. Appendix C.I provides detailed description of the experimental setups.
To this end, we propose two different multitask regression problems of the form
(5) where & is test function with single global optimum whose position is parametrized by the second argument, that is, both tasks are identical (and thus related) up to translation. We use a single input IR.2 and drop task-specific network parameters. As backbone, we take simple network of the form W2 max Wix b1, 0) + b2 with b 10 by E R², and W,, W.J E 10x2
L(x) (xx Lz(x) = o0( [z; (z; + (Reff( 0),
For the first experiment we choose a simple (avocado-shaped) convex objective function and, for the second one, we opt for non-convex function with several local optima and single global optimum. Figure shows the training trajectories experiments, depicted as level plots in the space of and rk, respectively. We can observe that in the first experiment (Figure la). RotoGrad finds both optima by rotating the feature space and matching the (unique) local optima of the tasks. Similarly, the second experiment (Figure 1b) shows that,
the presence (and absence) of RotoGrad in both
5

---

Published as a conference paper at ICLR 2022
have two symmetric tasks and non-equidistant starting point, in the vanilla case the optimization is dominated by the task with an optimum closest to the starting point. RotoGrad avoids this behavior by equalizing gradients and, by aligning gradients, is able to find the optima of both functions.
5 RELATED WORK
Understanding and improving the interaction between tasks is one of the most fundamental problems of MTL, since any improvement in this regard would translate to all MTL systems. Consequently, several approaches to address this problem have been adopted in the literature. Among the different lines of work, the one most related to the present work gradient homogenization.
Gradient homogenization. Since the problem is two-fold, there are two main lines of work. On the one hand, we have task weishheigt approaches that focus on alleviating magnitude differences. Similar to us, GradNorm (Chen et al., 2018) attempts to learn all tasks at a similar rate, yet they propose to learn these weights as parameters. Instead, we provide a closed-form solution in Equation I, and so does IMTL-G Liu et al. (2021b). However, IMTL-G scales all task gradients such that all projections of G onto Gy are equal. MGDA-UB (Sener & Koltun, 2018), instcad, adopts an itcrative method based on the Frank-Wolff algorithm in order to find the set of weights }w&} (with Ex 4 = 1) such that Ex ti G3 has minimum norm. On the other hand, recent works have started to put attention on the conflicting direction problem. Maninis al. (2019) and Sinha et al. (2018) proposed to make task gradients statistically indistinguishable via adversarial training. More recently, PCGrad (Yu al., 2020) proposed drop the projection of one task gradient onto another if they are in conflict, whereas GradDrop (Chen et al., 2020) randomly drops elements of the task gradients based on sign-purity score. Contemporaneously to this work, improved versions of MGDA (Désidéri, 2012) and PCGrad have been proposed by Liu et al. (2021a) and Wang et al. (2021), respectively.
The literature also includes approaches which, while orthogonal to the gradient homogenization, are complementary to our work and thus could be used along with RotoGrad. Next, we provide a brief overview of them. A prominent approach for MTL is task clustering, that is, selecting which tasks should be learned together. This approach dates back to the original task-clustering algorithm (Thrun O' O'ullli,,n,, 1996). but new work in this direction keeps coming out (Standley et al., 2020; Zamir et al., 2018; Shen et al., 2021; Fifty et al., 2021). Alternative approaches, for example, scale the loss of each task differently based on different criteria such as task uncertainty (Kendall et al., 2018), task prioritization (Guo et al., 2018), or similar loss magnitudes (Liu et al., 2021b). Moreover, while most models fall into the hard-parameter sharing umbrella, there exists other architectures in the literature. Soft-parameter sharing architectures (Ruder, 2017), for example, do not have shared parameters but instead impose some kind of shared restrictions to the entire set of parameters. An interesting approach consists in letting the model itself learn which parts of the architecture should be used for each of the tasks (Guo et al.. 2020; Misra et al., 2016; Sun et al., 2020; Vandenhende et al., 2020). Other architectures, such as MTAN (Liu et al., 2019b), make use of task-specific attention to select relevant features for each task. Finally, similar issues have also been studied in other domains like meta-learning (Flennerhag et al., 2019) and continual learning (Lopez-Paz. & Ranzato, 2017).
EXPERIMENTS
In this section we assess the performance of RotoGrad on wide range of datasets and MTL architectures. First, we check the effect of the learning rates of the rotation and network updates on the stability of RotoGrad. Then, with the goal of applying RotoGrad to scenarios with large sizes of z, we explore the effect of rotating only a subspace of z. Finally, we compare our approach with competing MTL solutions in the literature, showing that RotoGrad consistently outperforms all existing methods. Refer to Appendix C for a more details on the experiments and additional results.
Relative task improvement. Since MTL uses different metrics for different tasks, throughout this section we group results by means of the relative task improvement, first introduced by Maninis al. (2019). Given a task k. and the metrics obtained during test time by a model. M2. and by baseline model, Sg. which consists of K networks trained on each task individually, the relative task improvement for the k-th task is defined as
4 100 (-1) Me St
(6)
Sp

---

Published as a conference paper at ICLR 2022
where 14 if Mj Sk means that our model performs better than the baseline in the k-th task, and 4y = 0 otherwise. We depict our results using different statistics of A such as its mean (avg, Ax), maximum (max, As), and median (med, A&)) across tasks.
Statistical significance. We highlight significant improvements according to a one-sided paired t-test (a 0.05), with respect to MTL with vanilla optimization (marked with in each table).
6.1 TRAINING STABILITY
Sum digits
At the end of $3.2 we discussed that, by casting problem 4 as Stackelberg game, we have convergence guarantees when the rotation optimizer is the slow-learner. Next, we empirically show this necessary condition.
MSE
4.0
3.5
Experimental setup. Similar to (Sener & Koltun, 2018), use multitask version of MNIST (LeCun et al., 2010) where each image is composed of left and right digit, and use backbone reduced version of LeNet (LeCun et al., 1998) with weight heads. Besides the left- and right-digit classification proposed in (Sener & Koltun, 8), we consider three other quantities to predict: i) sum of digits; ii) parity of the digit product: and iii) number of active pixels. The idea here is to enforce all digit-related tasks to cooperate (positive transfer), while the (orthogonal) rmage-related task should not disrupt these learning dynamics.
3.0
2.5
1o.4 5.04 1o:3
5e-3 1e-2 5e-2 1o-1
RoloGrad's learning rate
Figure 3: Test error the sum of digits task for different values of RotoGrad's learning rate multi-MNIST.
over ten independent runs of changing the rotations' learning rate terms of test error in the sum task, while the rest of tasks are shown in Appendix C.2. We observe that, the bigger the learning rate is, in comparison to that of the network's parameters (le-3), the higher and noisier the test error becomes. MSE keeps decreasing as we lower the learning rate, reaching sweet-spot at half the network's learning rate (5e-4). 4). For smaller values, the rotations' learning is too slow and results start to resemble those of the vanilla case, in which no rotations are applied (leftmost box in Figure 3).
Results. Figure 3 shows the effect-averagee
6.2 ROTOGRAD BUILDING BLOCKS
In this section, we empirically evaluate to which extent each of the RotoGrad building blocks, that we denote Scale Only ($3.1) 1) and Rotate Only ($3.2), contribute to the performance gain in MTL.
Experimental setup. We test all methods on three different tasks of NYUv2 (Couprie et al., 2013): 13-class semantic segmentation; depth estimation; and normal surfaces estimation. To speed up training, all images were resized to 288 x 384 resolution; and data augmentation was applied to alleviate overfitting, As MTL architecture, decoder split into three convolutional heads. This is the same setup as that of Liu al. (2019b).
SegNet (Badrinarayanan et al., 2017) where the
Results. The three top rows of Table show the performance of RotoGrad and its both components in isolation. All the methods with the same number of parameters. Compared to Vanilla optimization (4th row), Rotate Only improves all metrics by homogenizing gradient directions. Scale Only avoids overlooking the normal estimation task and improves on segmentation by homogenizing gradient magnitudes, the expense of higher depth estimation error. Remarkably, RotoGrad exploits its scaling and rotation components to obtain the best results in semantic segmentation and depth estimation, while still achieving comparable performance in the normal estimation task.
6.3 SUBSPACE ROTATIONS
We now evaluate the effect of subspace rotations as described at the end of $3.4, assessing the trade-off between avoiding negative transfer and size of the subspace considered by RotoGrad.
Experimental setup. We test RotoGrad on 10-task classification problem on CIFARIO (Krizhevsky et al., 2009), using binary cross-entropy and fl fl-score as loss and metric, respectively, for all tasks. We use R8 (He et al., 2016) without pre-training as backbone (d = 512), and linear layers with sigmoid activation functions task-specific heads.
7

---

Published as a conference paper at ICLR 2022
Table 1: Median (over five runs) on the NYUv2 dataset. RotoGrad obtains great performance in segmentation and depth tasks, and significantly improves the results on normal surfaces. As, Ap. and A denote the relative task improvement for each task.
Relative improvements
Segmentation
f
Depth L
Normal Surfaces
Angle Dist. +
Within t"
Method
As
Ap
Aw
mloU
Pix
Rel.
Mcan
Median
22.3
Single
0.0
0.0
39.21
64.59
0.70
0.27
30.01
57.33
69.30
With R (m = 1024)
Rotate Only
3.3
20.5
-6.6
39.63
06.16
0.53
0.21
26.12
20.93
26.85
53.76
66.50
Scale Only
-0.3
20.0
-7.9
38.89
65.94
0.54
0.22
26.47
21.24
26.24
53.04
65.81
RotoGrad
1.8
24.0
-6.1
39.32
66.07
0.53
0.21
26.01
20.80
27.18
54.02
66.53
Vanilla
2.7
20.6
25.7
38.05
64.39
0.54
0.22
30.02
26,16
20.02
43.47
56.87
GradDrop
-0.9
14.0
-25.2
38.79
64.36
0.59
0.224
29.80
25.81
19.88
44.08
57.54
PCGrad
-2.7
20.5
-26.3
37.15
63.44
0.55
0.22
30.06
26.18
19.58
43.51
56.87
MGDA-UB
31.2
0.6
21.60
10.60
0.77
0.229
24.74
30.32
69.88
GradNorm
0.6
19.5
10.5
37.22
63.61
0.54
0.22
26.68
21.67
25.95
52.16
64.95
IMTL-G
-0.3
17.6
-7.5
38.38
64.66
0.54
0..22
26.38
21.35
26.56
52.84
65.69
Vanilla
0.9
16.8
-2.00
37.11
63.98
0.56
0.22
29.93
25.89
20.34
43.92
57.39
Without Ra
GrudDrop
0.1
15.7
27.0
37.51
63.62
0.59
0.23
30.15
26.33
19.32
43.15
56.59
POGrad
0.5
20.0
24.6
38.51
63.95
0.55
0.22
29.79
25.77
20.61
44.22
37.64
MGDA-UB
32.2
-8.2
1.5
20.75
51.44
0.73
0.28
24.70
18.92
30.57
57.95
69.99
GradNorm
2.2
20.6
10.2
39.29
64.80
0.53
0.22
26.77
21.88
25.39
51.78
64.76
IMTL-G
1.9
21.4
6.7
39.94
65.96
0.55
0.21
26.23
21.14
26.77
53.25
66.22
Results. Table 2 (top) shows that rotating the entire space provides the best results, and that these worsen as we decrease the size Rg. Remarkably, rotating only 128 features already outperforms vanilla with extra per-task parameters (Ist row); and rotating 256 features already yields com- parable results to vanilla optimization with extra capacity (6th row) despite its larger number of (task-specific)) parameters. These results can be further explained by Figure in Appendix C.2, which shows a positive correlation between the size of R& and cosine similarity.
6.4 METHODS COMPARISON
Experimental setup. In order to provide fair comparisons among methods, all experiments use identical configurations and random initializations. For all methods we performed a hyperparameter search and chose the best ones based on validation error. Unless otherwise specified, all baselines use the same architecture (and thus, number of parameters) as RotoGrad, taking each rotation matrix RA as extra task-specific parameters. Further experimental details can be found in Appendix C. as well as extra experiments and complete results in Appendix C.2.
NYUv2. Table shows the performance of all baselines with and without the extra capacity, RotoGrad significantly improves performance on all tasks compared with vanilla optimization, and outperforms all other baselines. Remarkably, a total of millions) and, as a result, RotoGrad stays on par in training time with the base- lines (around 4 h, Appendix C.2). We can also assert the importance of learning the matrices R& properly by comparing in Table the different baselines with and without extra capacity. This comparison reveals that the extra pa- rameters do not solve the negative trans- fer but instead amplifies biases (methods that overlook subset of tasks, keep over- looking them) and, in the best case, pro- vides trade-off solutions (also shown in Appendix C.2). Note, moreover, that Ro- toGrad (due to its Rotate Only component) is the only method to tackle conflicting gra- dient directions that manages to not over- look the normal surfaces task.
rotate only 1024 dimensions of (out of
Vanilla
MGDA-UB IMTL-G
RotaGrad 256 RotoGrad 512
GradDrop
PCGrad
GradNorm
Cosine similarity
0.3
0.2
10k
40k
60k
70k
Iterations
Figure 4: Similarity between task and update gradients to compare different MTL baselines for different methods on CIFAR10, averaged over tasks and five runs.
CIFAR1O. We reuse the setting from $6.3
terms of relative improvements (Table 2)
8

---

Published as a conference paper at ICLR 2022
Table 2: (Top) Relative task improvement on CIFARIO for RotoGrad with matrices R& of different sizes; and (bottom) comparison with baseline methods including rotation matrices as extra task-specific parameters. Ta- ble shows median and standard deviation over five runs.
Table 3: Fi-score statistics in CelebA for two neural network architectures. Median over five different runs.
task fl-scores (%)
Method
ming
medy, avga std,
Method
avga Ax f
med, Au t
max: Au t
Vanilla
4.59
50.28 56.03
25.65
Conv, Net. with
RR (m = 236)
Vanillat
0
2.58 0.54
1.90 + 0.53 11,14 + 3.35
GradDrrp
3.18 50.07
54.43
27.21
RotoGrad
64
2.90 + 0.49
1.79 + 0.57
13.16 + 2.40
PCGrad
1.44
53.05
54.72
27.61
RotoGrad
128
2.97 + 1.08
2.25 + 1.07
12.64
3.56
GradNorm
2.08 52.53
56.71
24.57
RotoGrad
256
3.68 + 0.68
2.76
14.01
+ 3.22
IMTL-G
0.00
37.00
42.24
33.46
RotoGrad
512
4.48 + 0.99
3,67 1.40
15.57 1 3.99
RotoGrad
4.59
55.02
57.20
24.75
Vanilla
3.12 0.79
3.10 1.29
14.23 + 2.86
Vanilla
19.71 63.56
63.23
21.16
ResiNet18 with
Rs (27 1536)
GradDrop
3.54 1.10
3.27
: 1.61
13.88 +: 2.95
GradDrop
12.33 62,40 62.74
21.74
With R
PCGrad
3.29
2.67
= 0.88
13.44
1.86
PCGrad
14.71 63.65 62.61
22.22
MGDA-UB
0.67
0.57
0.62
4.78 : 2.15
GradNNrm
9.05 60.20 60.78
22.31
GradNorm
3.21 1.04
3.10 1.01
10.88 + 4.73
IMTL-G
17.11 61.68 60.72
22.80
IMTL-G
3.02 +
1.81 + 0.87
12.76 + 1.77
RotoGrad
9.96 63.84 62.81 21.80
and cosine similarity (Fig. 4), averaged over five different runs, Table (bottom) shows that, similar to the NYUv2 results, both direction-aware solutions (PCGrad and GradDrop) behave similar to vanilla optimization, marginally increasing the average improvement. Unlike previous experiments, all magnitude-awan methods substantially worsen (at least) one of the statistics. In contrast, RotoGrad improves the relative task improvement across all statistics using the same number of parameters. Figure shows the cosine similarity between task and update gradients, averaged over all tasks and runs (shaded areas correspond to 90 % confidence intervals). It is clear that RotoGrad obtains the highest cosine similarity, that other direction-ave aware methods also effectively align task gradients and, combined with the low cosine similarity achieved by MGDA-UB, suggests that there exists correlation between cosine similarity and performance.
CelebA. To finalize, we test all methods in 40-class multi- colti-classification problem on CelebA (Liu al., 2015) and two different settings: one using a convolutional network as backbone (d 512); and another using ResNet18 8 (He et al., 2016) as backbone (d = 2048). As above, we use binary cross-entropy and fl f1-score as loss and metric for all tasks, thus accounting for highly imbalanced attributes. Results in Table 3 show that RotoGrrad performs great in all fl fc-se statistics and both architectures, specially in the convolutional neural network, outperforming competing methods.
Moreover, RotoGrad achieves these results rotating 50 5 of the shared feature for the convolutional network, and 75 % for the residual network, which further demonstrates that RotoGrad can scale-up to real-world settings. We believe it is important to remark that, due to the high number of tasks, this setup is specially demanding. Results in Appendix C.2 show the performance of all baselines without the rotation matrices, demonstrating the negative effect that the extra capacity can have if not learned properly, as well as that RotoGrad stays on par with non-extended baselines in training time.
CONCLUSIONS
In this work, we have introduced RotoGrad, an algorithm that tackles negative transfer in MTL by homogenizing task gradients in terms of both magnitudes and directions. RotoGrad enforces similar convergence rate for all tasks, while the same time smoothly rotates the shared representation differently for each task in order to avoid conflicting gradients. As a result, RotoGrad leads to stable and accurate MTL. Our empirical results have shown the effectiveness of RotoGrad in many scenarios, staying on top of all competing methods in performance, while being on par in terms of computational complexity with those that better scale to complex networks.
We believe our work opens up interesting venues for future work. For example, it would be interesting to study alternative approaches to further scale up RotoGrad using, for example, diagonal- block or sparse rotation matrices; to rotate the feature space in application domains with structured features (e.g., channel-wis wise rotations in images); and to combine different methods, for example, by scaling gradients using the
of IMTLL- and the "favor slow-leamnns" policy of RotoGrad.

---

Published as a conference paper at ICLR 2022
REFERENCES
Pierre-Antoine Absil, Robert E. Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008. ISBN 978-0-691-13298-3. URL http: //press. pr princeton. wwweeettooendneeeeeeeeeeeeeeeeee adu/titles/ 8586. html.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 39(12): 2481-2495, 2017. doi: 10.1109/TPAMM222112244444 URL https: /tt.///oiiooo///o org/ 1109/ TPAMI 2016 6. 4445
Rich Caruana. Multitask learning: A knowledge-basee based source of inductive bias. In Paul E. Utgoff (ed.), Machine Learning, Proceedings of the Tenth International Conference, Univer- sity of Massachusetts, Amherst, MA, USA, June 27-29, 1993, pp. 41-48, Morgan Kaufmann, URL https:
1993. doi: 10.
b978-1-55860 -006600-----000 307 50
Mario Lezcano Casado. Trivializations for gradient-based optimization on manifolds. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Aiché-Buc. Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 9154-9164, 2019. URL https: http:///coceeeeeeeeeeooooe 2019 htsha/hhaaaa 1b331113311111
9b (11.71199911111 wew.eeeeeednnnnn..nn............ newrips.......... cC/ /ppppppr////// Abstract html.
Mario Lezcano Casado and David Martínez- Rubio.
Cheap orthogonal constraints in neural networks:
A simple parametrization of the orthogonal and unitary group. 97:3794-3803, 2019. URL http:/ http://proceeeeeee ngs ml www.m.rr.... www...prss/se. /v7/ wwwanooooooooo casado htmm..
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Jennifer G. Dy and Andreas Krause (cds.), Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmüsssan, Stockholm, Sweden, July 10-15, 2018, volume 80 Proceedings of Machine Learning Research, pp. 793-802. PMLR, 2018. URL http: http:://prrceeedngg. wwl..press/oe vv0/ chen html.
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzsehmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL /ttpt:psppppocoeeeiiiiee 1.6 16002. 7 4 a 94 91 ecc3eeecb99240
neurips. ract html.
Jorge Cortés. Finite-time convergent gradient flows with applications to network consensus. Autom., (2(11):193-2000 2006. doi: 10.1016/j.autoon irlommttiaa2006.06.0000 URL htRLtt:///do..ooo org a.t.:o/a........attaaaaa automat ca 2006. 06. 015.
10.1016/5
Camille Couprie, Clément Farabet, Laurent Najman, and Yann LeCun. Indoor semantic segmentation using depth information. In Yoshua Bengio and Yann LeCun (eds.), Ist International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Conference Track Proceedings, 2013. URL http: arxiv. org/
Désidéri. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Muthematique, 350(5-6):1133116 2012.
Tanner Fiez, Benjamin Chasnov, and Lillian Ratlfff. Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study. 119:3133-3144, 13-18 Jul 2020. URL peetps:/pproceddee
mlr srress/vll9 fiez20a html.
Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu. Rohan Anil, and Chelsea Finn. Efficiently iden- tifying task groupings for multi-task learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https:tpp... http////eeeeveeeeeeeeeeeeee ew. net mete/formme forum? 65vg
10

---

Published as a conference paper at ICLR 2022
Sebastian Flennerhag, Andrei A Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-leaming with warped gradient descent. arXiv preprint 1XXi::999.00022. 2019.
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task pri- oritization for multitask learning. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14. 2018, Proceedings, Part XVI, volume 11220 of Lecture Notes in URL Computer Science, pp. 282-299 Springer, 2018. doi: 10.1007/978- 00077/99788--0000227-00000 https:/ /ttl:///oooo or worg/r/ggoooo.... / 10. 1007 978- 3 -00-0000------0 010000007000000 117
Pengsheng Guo, Chen- Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3854-3863. PMLR, 2020. URL http::ttp//.. htt.::://pooceeeliiippt proceedi ngs m1 press / v11 gwtpg/ooeeeeeeeeoooooo html.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016. URL https: /ooi org/ /10 1109/ CVPR, 201 2.166......0000000 90.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in Sanconvex-noncooecn minimax optimization? In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4880-4889 PMLR. 13-18 Jul 2020. URL https: btttps://proeee httpss///cccceeieegeeeee mlr.
press/ v11 / in20e. html.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7482-7491, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69800 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. tradient-basee learning applied to document recognition. Proceedings of the IEEE, 86( (6(11):2278 11::2278-2324, 1998.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online], 2, 2010. URL http: /yann. 1 lecun. com/ exdb / mn its.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and qiang liu. Conflict-averss gradient descent for multi-task learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 202 la. URI https:ttp//// /owporoeeeeeeeeeee net wwt.:/ntttooottieee forum? i d== _61_________________ Qh8tULi
Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations, 2021b. URL https: : /owporpeeeiieeeeeeeee http:/iwwwwweeeeeeeeeeeeeeeeeeee forttt/foummmee IMI.MMMMXMPIPIP
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint 1908v 03265, 2019a.
Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20, 2019, pp. 1871-1880. Computer Vision Foundation / IEEE, 2019b. doi: 10.1109/CVPR2201900005 //openaccess.
URL http: http://ttttnniinnniieieeeeeeeeee End-To- End_ htw.r//nw......ttttttoooooooor.. 2019__ape1_________ __paper html.
wwwpaaacccesss..ei..eoo.. thecvf.com
11

---

Published as a conference paper at ICLR 2022
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015. pp. 3730-373333 IEEE Computer Society, 2015. doi: 10.1109/ICCV22015..223 URL https: / /doi. .org/ 10. 1109/ TCCV. 201 425.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N.
Vishwanathan, and Roman Garnett Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6467 2017. URL ht /ttp:///cccceeee.oooooo...c...ee wtt.::eewwiiinnn...eneeeeeeeeeee www..ps......... cc/ paper 2017 /hash/
html.
Kevis- Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos. Attentive single- tasking of multiple tasks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20, 2019, pp. 1851- 1860. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/VVR22211.00195 URL
com/ wwtt/otttttttteceeeccoeteooce CVPR 2019 Mantttm/MMaaiiii... www.eoeeeeilpeeeeeeeeeeeeeeeeeee Saaetiippe_Tassssseeee CL_____V_________ p_peeer .hml1.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for task learning. In 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 3994-4003 IEEE Computer Society, 2016. doi: N0.1000CVP...201..455 URL https:tpp..... : / doii. wrggorgg///o. 1t//111.1101 110./ /CV/R////RRRRR 2016 6.1.6... 433.
Ryan W. Murray, Brian Swenson, and Soummya Kar. Revisiting normalized gradient descent: Fast evasion of saddle points. IEEE Trans. Autom, Control., 64(11):41184424 2019, doi: 10.1 1.11199A..2009.2999 199999 URL https: /ooi. org/10 1ww.:111199111.... 2019, 2914998.
Yurii E. Nesterov. Introductory Lectures on Convex Optimization A Basic Course, volume 87 of Ap- plied Optimization. Springer, 2004, ISBN 378-1-4613-4661-3. doi: 11007/97-11-441-8888
URL /ettps://doi.ooo org /10 111111110000 1007 978 4419-8853-99-
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. CoRR, abs/ htss/170.00099 2017. URL /ttt:://arzee org/ abs/ C5098.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Resa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 525-536, 2018 URL https: http///ooocccccccococooooooooccc i ngs neurips...... cc wwtpppppp//01111. 8 8/hash/ 432aca3ale34 3ataaaeae..s33333330033333eeeee ADs html.
Jiayi Shen, Xiantong Zhen, Marcel Worring, and Ling Shao. Variational multi-task learning with gumbel-softmax priors. In Thirt--Fifth Conference Neural Information Processing Systems, 2021. URL A
http///pee/eeeeeeee/eeee/ee net foret//orummiee 2rkcde
Ayan Sinha, Zhao Chen, Vijay Badrinarayanan, and Andrew Rabinovich. Gradient adversarial training of neural networks. arXiv preprint arXiv:18066 100.00080. 2018
Trevor Standley, Amir Roshan Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 9120-9132. PMLR, 2020. URL http:/ /tt::/prrooo prwceedings.....oooo mlr. pwwssressssseeee /vlsss/vlii 9/ sttssaandllllll yyndley22aaa html..
Ximeng Sun, Rameswar Panda, Rogério Feris, and Kate Saenko. Adashare: Learning what share for efficient deep multi-task learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Mlria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing
12

---

Published as a conference paper at ICLR 2022
Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httttp:://procceednggs.ssssss/ss neurips. cc/ paper/ 634.hhh33444440 3000://s11//14111666666622222113 att.://ccbb..t....---aaatttttt html.
Sebastian Thrun and Joseph O' O'Sullivan. Discovering structure in multiple learning tasks: The TC algorithm. In Lorenza Saitta (ed.), Machine Learning, Proceedings of the Thirteenth International Conference (ICML '96). Bari, Italy, July 3-6, 1996, pp. 489-497. Morgan Kaufmann, 1996.
Simon Vandenhende, Stamatios Georgoulis, Luc Van Gool, and Bert De Brabandere. Branched multi-task networks: Deciding what layers share. In 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event UK, September 7-10, 2020. BMVA Press, 2020. URL https: // //www. www.b//ww222.. www.eeeeeeooocceeeeeeeeeeeeeeeee
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In International Confer- ence on Learning Representations, 2021. URL hwwph:ttp///ott http::///ppeeeveew.. net forum?id-
FIVE 1
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5824- 5836. Curran Associates, Inc., 2020. URL https: http::/:s////ccoeeegegee. pdf. http://e//E............25.5555.2
Amir Roshan Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In 2018 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 3712-3722. IEEE Computer Society, 2018. doi: 10.1109/CVPR. 2018.00391. URL http: /twp//ccpppaaeaapeeeeeac/cc.sasa theevf. com/ content_cvpr__2000 2001/html// Zami r_ www..aaaa Di ong.ooonomnyyissttttagggngnnnnao Sacpalliinn______o__ee hatpaaaannnnnnnnaaaaa____ee____ 8 paper html.
13

---

Published as a conference paper at ICLR 2022
Appendices
A PROOFS
Proposition A.1. Suppose Jn. == Lx o hy is infinitely differentiable real-valued function and let us call G5 (Z) its derivative with respect to Z, for every k: = 1,2,..., K. If 1) pairwise; then there exists small-enough step size >
cos sim G2 G5) > -1/(K such that, for all k, we have that Lu(hx(Z - - Cx Øz) Yx) L 0x); Yk), where U₁ = Gx/l| and C' 2 O.
Proof. Since fa is infinitely differentiable, we can take the first-order Taylor expansion of fx around Z, for any k. evaluated at Z =V for given vector V:
(x - = = fx(2) - (Gu, V) + os))
(7)
In our case, V =C Us with C 2 0, then:
- -V) (2) = - Cl|G Σ (Us, U.) + 0(s)
(8)
= ---- Clllul 1 1+ i4
+ Ui) +
[( U.)
(9)
Since ||Un|| = for all = =,2,, 22...,,,
1,2, K, it holds that -1 v cos_sim U2 U.) (Ux. U,) 1
If cos_ sim(G G,) -1/(K 1) for all + k then have that 1 + Eijs (Uk U.) 0 and fz =V) (v) for small enough 0.
Q.E.D.
B STACKELBERG GAMES AND ROTOGRAD
In game theory, Stackelberg game (Fiez et al., 2020) is asymmetric game where two players play alternately. One of the players-whae whose objective is to blindly minimize their loss function-is is known as the follower, F. The other player is known as the leader, L. In contrast to the follower, the leader attempts to minimize their own loss function, but it does so with the advantage of knowing which will be the best response to their move by the follower. The problem be written
Leader: min (C(r) my argmin F(r,, Z1EXi
yEX,
(10)
Follower: min F(z1, zyEX,
where xi E X1 and xy E X are the actions taken by the leader and follower, respectively.
While traditionally one assumes that players make perfect alternate moves in each step of problem 10, gradient-play Stackelberg games assume instead that players perform simultancous gradient updates,
1+1 = rf af Vz, ((x1, 1+1 xty al V L(11,
(11)
where ag and ay the learning rates of the leader and follower, respectively.
An important concept in game theory that of an equilibrium point, that is, a point in which both players satisfied with their situation, meaning that there is available move immediately improving any of the players' scores, so that none of the players is willing to perform additional actions/updates. Specifically, focus the following definition introduced by Fiez al. (2020):
14

---

Published as a conference paper at ICLR 2022
Definition B.1 (differential Stackelberg equilibrium). A pair of points 2t E X1, 27 E X,, where s; r(af) is implicitly defined by V F=; = O, is differential Stackelberg equilibrium point if V., [(x r(xt)) ) = O, and V2 [ r(af)) )) positive definite.
Note that, when the players manage to reach such equilibrium point, both of them are in local optimum. Here, we make use of the following result, introduced by Fiez et al. (2020), to provide theoretical convergence guarantees an equilibrium point:
In the given setting, if the leader's learning rate goes to zero at a faster rate than the follower's. s. that is, (t) (t)), where at(t) denotes the learning rate of player i at step t, then they will asymptotically converge to differential Stackelberg equilibrium point almost surely.
Proposition B.1.
In other words, long as the follower learns faster than the leader, they will end up in situation where both are satisfied. Even more, Fiez et al. (2020) extended this result to the finite-time case, showing that the game will end close to an equilibrium point with high probability.
As we can observe, the Stackelberg formulation in Equation (10) is really similar to RotoGrad's formulation in Equation (4). Even more, the update rule in Equation (11) is quite similar to that one shown in Algorithm 1. Therefore, it is sensible to cast RotoGrad as a Stackelberg game. One important but subtle bit about this link regards the extra information used by the leader. In our case, this extra knowledge explicitly appears in Equation 3 in the form of the follower's gradient, Ji.k- which is the direction the follower will follow and, as it is performing first-order optimization by assumption, this gradient directly encodes the follower's response.
Thanks to the Stackelberg formulation in Equation we can make use of Prop. B. and, thus, draw theoretical guarantees on the training stability and convergence. In other words, we can say that performing training steps as those described in Algorithm 1 will stably converge as long as the leader asymptotically the slow learner, that is af = o(at)) where denotes the little-o notation.
In practice, however, the optimization procedure proposed by Fiez et al. (2020) requires computing the gradient of gradient, thus incurring a significant overhead. Instead, we use Gradient Ascent- Deesett (GDA), which only computes partial derivatives and enjoys similar guarantees (Jin et al., 2020), as we empirically showed in the manuscript.
C EXPERIMENTS
C.I EXPERIMENTAL SETUPS
Here, we discuss common settings across all experiments. Refer specific sections further below for details concerning single experiments.
Computational resources. All experiments were performed on a shared cluster system with two NVIDIA DGX-A 100. Therefore, all experiments were run with (up to) 4 cores of AMD EPYC 7742 CPUs and, for those trained on GPU (CIFARIO, CelebA, and NYUv2), a single NVIDIA A100 GPU. All experiments were restricted to 12 GB of RAM.
Loss normalization. Similar as in the gradient case studied in this work, magnitude differences between losses can make the model overlook some tasks. To overcome this issue, here we perform loss normalization, that is, we normalize all losses by their value the first training iteration (so that they are all 1 in the first iteration). To account for some losses that may quickly decrease at the start, after the 20th iteration, we instead normalize losses dividing by their value at that iteration.
Checkpoints. For the single training of a model, we select the parameters of the model by taking those that obtained the best validation error after each training epoch. That is, after each epoch we validation loss, Σx Lk. and use the parameters that obtained the best
evaluate the linearly-scalarize error during training. This can be seen as an extension of carly-stopping where, instead of stopping, we keep training until reaching the maximum number of epochs hoping to keep improving.
Baselines. We have implemented all baselines according to the original paper descriptions, except for PCGrad, which we apply to the gradients with respect to the feature z (instead of the shared parameters 0, as in the original paper). Note that this is in accordance with recent works, for example Chen et al. (2020) and Liu et al. (2021b), which also use this implementation of PCGrad in the feature
15

---

Published as a conference paper at ICLR 2022
space. This way, all competing methods modify gradients with respect to the same variables and, as backpropagation performs the sum of gradients at the last shared representation 2, PCGrad can be applied to reduce conflict at that level.
Hyperparameter tuning. Model-specific hyperparameters were mostly selected by a combination of: i) using values described in prior works; and ii) empirical validation the vanilla case (without any gradient-modin modifiers) to verify that the combinations of hyperparameters work. To select method- specific hyperparameters we performed a grid search, choosing those combinations of values that performed the best with respect to validation error.
Specifically, took {0, 1, and R& mRmx with m {0.25d, 0.5d, 0.75d, d} (restrict- ing ourselves to < 1500) for RotoGrad. Regarding the learning rate of RotoGrad (GradNorm), we performed grid search considering Thoto {0.05m, O,1 0.5m, 17, 2n), where Thoto and 17 are the learning rates of RotoGrad (GradNorm) and the network, respectively.
Statistical test. For the tabular data, we highlight those results that are significantly better than those from the multitask baseline (that is, better than the vanilla MTL optimization without the R3, matrices). To find these values, we run a paired one-sided Student's t-test across each method and the baseline. For those metrics for which higher is better, our null hypothesis is that the method's ' performance is equal or lower than the baseline, and for those for which lower is better, the null hypothesis is that the method's performance is equal or greater than the bascline. We use a significance level of a 0.05.
Notation. Along this section, we use the following to describe different architectures: [Conv-F-C) denotes convolutional layer with filter size F and C number of channels; [Max denotes max-pool layer of filter size and stride 2, and [Den.eeH]] a dense layer with output size H.
C.1.1 ILLUSTRATIVE EXAMPLES
Losses and metrics. Both illustrative experiments use MSE as both loss and metric. Regarding the specific form of in Equation (5), the avocado-shaper shaped experiments uses
p((x,, y). s) (z s)² + 25y²
(12)
while the non-convex second experiment uses
((( 8). sin(3x + 4.5s) sin (3y + 4.5s)
+ jxr + 1.5s| ly + 1.5s]
(13)
+ 1.58
+ 1.58
Model. As described in the main manuscript, we use single input E R²² picked at random from standard normal distribution, and drop all task-specific network parameters (that is, (rx) = rg). As backbone, we take simple network of the form by E 10 b2 E R2, and W1, W E 1002
W2 max(Wi + b,, 0) + b2 with
Hyperparameters, convex-case. We train the model for one hundred epochs. As network optimizer we use stochastic gradient descent (SGD) with a learning rate of 0.01. For the rotations we use RAdam (Liu et al., 2019a) with a learning rate of 0.5 (for visualization purposes need high learning rate, in such simple scenario it still converges) and exponential decay with decaying factor 0.999 99.
Hyperparameter, non-convex case. For the second experiment, we train the model for 400 epochs and, once again, SGD as the network optimizer with a learning rate of 0.015. For the rotations, we use RAdam (Liu et al., 2019a) with a learning rate of 0.1 and an exponential decay of 0.999 99.
C.1.2 MNIST/SVHN
Datasets. We use two modified versions of MNIST (LeCun al., 2010) and SVHN (Netzer et al., 2011) in which each image has two digits, one each side of the image. In the case of MNIST, both of them are merged such that they form an overlapped image of 28 x 28, as shown in Figure 5a. Since SVHN contains backgrounds, images of size 32 x 64, as shown in Figure 5b. Moreover, we transform all SVHN samples to grayscale.
simply paste two images together without overlapping, obtaining
Tasks, losses, and metrics. In order to further clarify the setup used, here we describe in detail each task. Specifically, we have:
16

---

Published as a conference paper at ICLR 2022
MNIST
SVHN
5
3
6
68
9
6
-
&
2
5
9
2
3
85
7852
1
6
3
26
1
2
23
144
41
3
2
8
38
322
3
$
06
7
9
3
12
6
2
143
4
3
8
58
4 172
0
8
60
3
7
4
8
3
2
3
18
1
(a) Multi-MNIST.
(b) Multi-SVHN.
Figure 5: Samples extracted from the modified MNIST and SVHN datasets.
Left digit classification. Loss: negative log-likelihood (NLL). Metric: accuracy (ACC).
Right digit classification. Loss: NLL. Metric: ACC.
Parity of the product of digits, that is, whether the product of both digits gives odd Loss: binary cross entropy (BCE). Metric: fl-score (F1).
number
Sum both digits (regression). Loss: MSE. Metric: MSE.
Active pixels the image, that is, predict the number of pixels with values higher than 0.5, where we use pixels lying in the unit interval (regression). Loss: MSE. Metric: MSE.
Model. Our backbone is an adaption from the original LeNet (LeCun et al., 1998) model. Specifically,
MNIST.
[Conv-5-10] [Max]
[ReLU] [Conv-5-20] [Max] [Dense-50] [ReLU] [BN],
SVHN.. [Cenv-5-10] [Max] [ReLU] [Conv-5-20) [Max] [Conv-5-20] [Dense-50] [ReLU] [BN],
where refers Batch Normalization (Ioffe & Szegedy, 2015). Depending on the type of task, we use a different head. Specifically, we use:
Regression. [Dense [RelU] [Dense-1],
Classification. [Dense-50) [ReLU] [Dense-10) -00 [Log-Softmax]
Binary-clasiffiationn [Dense-1] [Sigmoid].
Model hyperparameters. For both datasets, we train the model for 300 epochs using a batch size of 1024. For the network parameters,
RAdam (Liu al., 2019a) with learning of 1e-3.
Methods hyperparameters. In Tables and we show the results of GradNorm with: i) MNIST with R2, O; ii) MNIST without R2, a 0.5; iii) SVHN with Rą. 1; and iv) SVHN without Rs. a = 2. We train RotoGrad with full-size rotation matrices (m = d). Both methods use RAdam with learning rate 5 and exponential decay of 0.9999.
C.1.3 CIFAR10
Dataset. We CIFAR10 (Krizhevsky et al., 2009) dataset, with 40 000 instances training data and the rest as testing data. Additionally, every time we get a sample from the dataset we: i) crop the image by randomly selected square of size 3 32 x 32; ii) randomly flip the image horizontally; and
17

---

Published as a conference paper at ICLR 2022
iii) standardize the image channel- using the mean and standard deviation estimators obtained on the training data.
Model. We take backbone ResNet18 (He et al., 2016) without pre- pre-training, where remove the last linear and pool layers. In addition, we add a Batch Normalization layer. For each task saee-ipc linear layer followed by sigmoid function, that is, [Dense-1] [Si [sigmoid].
head, we simply
Losses and metrics. We treat each class (out of ten) as a -inary-classificttion task where we use BCE and Fl loss and metric, respectively.
Model hyperparameters. We use a batch size of 128 and train the model for 500 epochs. For the network parameters, we optimizer SGD with learning rate of 0.01, Nesterov momentum of 0.9, and a weight decay of 5e-4. Additionally, we use for the network parameters a cosine learning-rate scheduler with a period of 200 iterations.
Methods hyperparameters. Results shown in Tables 2 and use a = 0 and a = 0.5 for GradNorm with and without Ra. respectively, and we RAdam (Liu al., 2019a) as optimizer with learning rate 0.001 and an exponential decay factor of 0.999 95 for both GradNorm and RotoGrad.
C.1.4 NYUv2
Setup. In contrast with the rest of experiments, for the NYUv2 experiments shown in $6, instead of writing our own implementation, we slightly modified the open-source code provided by Liu et al. (2019b) at https: /gi thub. com/ www./mt/remmmmmmmmmeoo///// /otpmttttaatnnnnmmmmmm //mmaaa an (commit 268c5c1). We therefore use the exact same setting as Liu et al. (2019b)-and refer to their paper and code for further details, with the addition of using data augmentation for the experiments which, although not described in the paper, included in the repository as command-lin line argument. We will provide along this work diff file to include all mandient-modifae methods into the aforementioned code.
Methods byperparameters. For the results shown in Table 1 RotoGrad with rotations R& of size 1024. We use a similar optimization strategy as the rest of parameters, using Adam (Kingma & Ba, 2014) with learning rate 5 (half the one of the network parameters) and where we halve this learning rate every 100 iterations.
GradNorm with 0 and
C.1.5 CELEBA
Dataset. We use CelebA (Liu al., 2015) as dataset with usual splits. We resize each sample image so that they have size 3 x 64 x 64.
Losses and metrics. We treat each class (out of forty) as binary-classification task where BCE and Fl as loss and metric, respectively.
ResNet model. As with CIFARIO, we use as backbone ResNet18 (He et al., 2016) without pre- training, where we remove the last linear and pool layers. In addition, we add a Batch Normalization layer. For each task-specific head, we use linear layer followed by a sigmoid function, that is, [Dense-1] (SSiimmiiio
ResNet hyperparameters. We use batch size of 256 and train the model for 100 epochs. For the network parameters, we use RAdam (Liu et al., 2019a) optimizer with learning rate 0.001 and exponential decay of 0.999 95 applied every 2400 iterations.
Convolutional model. For the second architecture, we use convolutional network as backbone, [Conv-3-64 [BN] [Max]
[BN] 1oov-33-28) [BN] [Max] [Conv-3-255 3-666 [BN] [BN] [Dense-512] [BN]. For the task-specific heads, we [BN] Dense 1] [Sigmoid).
[Conv-3-28) 256] [BN] [Max]
take a simple network of the form Deesse- 128
Convolutional by typerparrmeters. We a batch size of 8 and train the model for 20 epochs. For the network parameters, we use RAdam (Liu et al., 2019a) as optimizer with learning rate 0.001 and exponential decay of 0.96 applied every 2400 iterations.
Methods hyperparameters. Results shown in Tables and 7 use GradNorm with: i) convolutional network with Rg, 0; ii) convolutional network without R& a 1; iii) residual network with Rg. a = 0.5; and iv) residual network without R4, a = For RotoGrad, we rotate 256 and 1536 elements of for the convolutional and residual networks. As optimizer, we RAdam (Liu et al.,
18

---

Published as a conference paper at ICLR 2022
2019a) with learning rate 5 and exponential decay factor of 0.999 95 for both GradNorm and RotoGrad.
Note that for these experiments omit MGDA-UB (Sener & Koltun, 2018) it is computationally prohibitive in comparisons with other methods. In single-seed experiments, we however observed that it does not perform too well (specially in the convolutional network).
C.2 ADDITIONAL. RESULTS
C.2.1 MULTI-MNIST AND MULTI-SVHN
Table 4: Test performance (median and standard deviation) on two set of unrelated tasks on MNIST and SVHN, across ten different runs.
MNIST
SVHN
Digits
Act Pix
Digits
Act Pix
Method
avga Ax f
MSE +
8V8 As 1
MSE +
Single
0.00 + 0.00
0.01 0.01
0.00 + 0.00
0.17 + 0.06
Vanilla
-1.43 3.24
0.14 = 0.05
4.78 0.88
3.04 + 2.53
GradDrop
1130 1.82
0.16 + 0.04
0.92 5.34
2.99 2.59
PCGrad
-1.22 + 2.81
0.13 0.01
5.01
2.70 = 2.25
MGDA-UB
-99.14 + 9.23
0.06 = 0.00
-4.36
1.00 = 0.57
GradNorm
0.86 + 1.93
0.09 0.04
0.89
4.12 + 9.46
IMTL-G
2.12 1.46
0.07 = 0.02
+.94
1.70 - 1.05
RotoGrad
1.55 2.22
0.08 + 0.03
6.08 + 0.48
1.61 = 2.72
We reuse the experimental setting from $6.1 16.1-now using the original LeNet (LeCun et al., 1998) and a multitask-version of SVHN (Netzer al., 2011)-in order to evaluate how disruptive the orthogonal image-related task is for different methods. We can observe (Table 4) that the effect the image-related task is more disruptive in MNIST, in which MGDA-UB utterly fails. Direction-aware methods (GradDrop and PCGrad) do not improve the vanilla results, whereas IMTL-G, GradNorm, and RotoGrad obtain the best results.
We also provide the complete results for all metrics in Table 5. the case of MNIST, we can observe that both regression tasks tend to be quite disruptive. GradNorm, IMTL--G, and RotoGrad manage improve over all tasks while maintaining good performance on the rest of tasks. MGDA-UB, however, focuses on the image-related task too much and overlooks other tasks. In SVHN we observe similar behavior. This time, all methods are able to leverage positive transfer and improve their results on the parity and sum tasks, obtaining similar task improvement results. Yet, the image-related task is more disruptive than before, showing bigger differences between methods. As before, MGDA -UB completely focuses on this task, but now able to not overlook any task while doing so. Regarding the rest of the methods, all of them improved their results with respect to the vanilla case, with RotoGrad and GradNorm obtaining the second-ber best results.
19

---

Published as a conference paper at ICLR 2022
Table 5: Complete results (median and standard deviation) of different competing methods on MNIST/SVHN on tasks, see Appendix C.1.2 and Appendix C.2.
Left digit Acc.
Right digit Ace.
Product parity
Sum digits MSE
Act. Pix. MSE.
Method
av8a 4x
Single
95.70 0.20
94.05 0.16
92.09
0.76
1.90 : 0.10
0.00 1 0.00
0.01 E 0.01
Without R.
Vanillaf
94.94 0.20
93.26
0.27
93.07
0.48
2.10
0.11 0.01
GradDrop
95.33 + 0.39
93.55 = 0.29
93.32
0.54
2.14
1.63
0.13 = 0.02
PCGrad
95.07 1 0.39
93.28 : 0.18
93.34 0.51
2.19
3.886
0.12 + 0.02
MGDA-UB
94.46 1 1.04
92.23 1 1.54
83,89
1.84
2.50 2.600
10.45 -10.80
0.06 1 0.002
GradNorm
95.19 0.37
93.70
0.31
0.39 93.24
98.31
2.06 28.71
-181 +3.999
0.09 + 7.46
MNIST
IMTL-G
95.28
0.21 93.84
82.48
0.07 2.05
Vanilla
95.13 0.20
93.41
93.54
0.50
+.223 0.17
0.14 + 0.05
GradDrop
95.14
93.47 0.12
93.59
2.00
1.82
0.16 + 0.04
PCGrad
95.04
0,33
93.49 0.30
1.98
2.82
0.13 0.01
MGDA-UB
89.99 2.21
86.76 1.18
79.24 2.83
3.65 = 0.42
-22.14
0.06 + 0.00
GradNorm
95.28 0.18
93.56 0.57
1.86 0.07
0.86 1: 1.93
0,09 1 0.04
IMTL-G
95.47 +: 0.27
93.79
93.56 0.57
1.73 = 0.09
2.12 1.46
0.07 - 0.02
RotoGrad
95.45 1 0.19
0.19 93.83
93.22 0.35
1.85 1 0.13
1,55 .. 2.22
0.08 1 0.03
Single
85.05 0.45
84.58
0.24
77.47
5.84
0.00
0.17
Without Ra.
Vanillat
84.18 + 0.30
84.18 + 0.38
80.11 111100001 0.85
4.81 + 0.06
$.14 0.83
2.75 + 3.17
GradDrop
84.38 0.29
84.48
0.41
80.11
0.69
4.69 1 0.12
5.68 1.05
1.91 1 0.86
PCGrad
0.31
79.92 0.79 84.23
+.09
0.75
2.26 - 0.85
MGDA-UR
0.75
84.38
1.38+0..5 0.45
77.44
4.47 0.18
+.99
0.66 0.75
GradNorm
8..1 0.33
79.40
0.07
1.00
4.30 2.18
SVHN
IMTL-G
8..6
84.39
79.63 1.10
4.57 0.13
5.81
0.88
2.47 + 1.65
Vanilla
0.41
84.11 = 0.40
0.40
79.83
0.79
1.14
0.88
3.04 2.53
GradDrop
0.35
84.33 0.40
80.10 0.83
4.73 0.09
5.94
2.99 + 2.59
With Rl
PCGrad
84.21
79.64
0.48
1
1 0.06
1.66
2.70 1 2.25
MGDA-UB
77.05
78.00
71.76
-176-4.33 4.32
1.56
6.72
1.00 1.57
GradNorm
84.37 0.334
84.30 0.46
79.97 0.75
4.72+0011
+.24
4.12 9.46
IMTL-G
84.23
79.77
4.51 1.12
5.94
RotoGrrad
84.50
0.44
79.14
4.45
6.08
1.61 2.72
20

---

Published as a conference paper at ICLR 2022
C.2.2 ILLUSTRATIVE EXAMPLES
We complement the illustrative figures shown in Figure by providing, for each example, an illustration of the effect of RotoGrad shown transformation. In an active transformation (Figure 6 left), points in the space are the ones modified. In our case, this means that we rotate feature 2, obtaining r1 and r2, while the loss functions remain the same. other words, for each z we obtain a task- specific feature Fx that optimizes its loss function. In contrast, a passive transforma- tion (Figure right) keeps the points unal- tered while applying the transformation to
active and passive
the space itself. In our case, this translates to rotating the optimization landscape of each loss function (now we have L4 Rg instead of Lx), so that our single feature x has an easier job at optimizing both tasks. In the case of RotoGrad, we can observe in both right figures that both optima lie in the same point, as we are aligning task gradients.
Figure 6: Level plots showing the illustrative examples of Figure for RotoGrad. Top: Convex case. Bot- tom: Non-convex case. Left: Active transformation (trajectories of ry and the level plot of L1 + L.2 Right: Passive transformation (trajectory of z and level plot of (L1 o R1) + (L2 R2)).
Besides the two regression experiments shown $4, we include here an additional experiment where test RotoGrad in the worst-case scenario gradient conflict, that is, one in which task gradients op- posite to each other. To this end, we solve a 2-task binary classification problem where, as dataset, we take 1000 samples from a 2D Gaussian mixture model with two clus- ters; Vn.k = 1 if 2, was sampled from cluster l;; and Yn. otherwise. We use as model logistic regression model of the form yk W2 max Wi + bi, 00) b2 with b R b2 R. W1 E 22x² and W2 E I2x2, Because rotations in ID are ill-posed (there is unique proper rotation), here we add task parameters to increase the dimensionality of and make all pa- rameters shared, so that there is still no task-specific parameters. To avoid complete conflict where V=Lı + Vel2 O, randomly flip the labels for the second tasks with 5 % probability. Figure shows that, in this extreme scenario, RotoGrad is able to learn both tasks by aligning gradients, that is, by learning that one rotation the inverse of the other R R.
Accuracy Vanilia RatoGrad
Task
Task 2
84.95 54.99% 87.57%
28.5 28.50%
73.14%
1.0
1.0
Cluster 2
Cluster
0.6
0.6
0.2
0.2
Figure 7: Logistic regression for opposite classifica- tion tasks. Test data is plotted scattered gray dots. RotoGrrad learns both opposite rotations R. = R.
C.2.3 TRAINING STABILITY
While showed in 1 only the results for the sum-of-digits task they were nice and clear, here we show in Figure the results of those same experiments in $6. for all the different tasks. The same discussion from the main manuscript observe that the vanilla case (learning rate zero) completely overlooks the image-related task (Active pixels) while performing the best in the parity task.
be carried out for all metrics. Additionally, we
Additionally, let us clarify what we mean here with stability, as in the main manuscript we mainly talked about convergence guarantees. In these experiments measure the convergence guarantees
21

---

Published as a conference paper at ICLR 2022
of the experiments in terms of "training stability'. meaning the variance of the obtained results across different runs. The intuition here that, since the model does not converge, we should expect some wriggling learning curves during training and, take the model with the best validation error, the individual task metrics should have bigger variance (that is, less stability) across runs.
Left digit
Right digit
0
g 1e-4
1e-4
@ 5-ef
5-e4
1ee3-
-ee-3
5e-3-
1e-2-
1e-2
5e-2-
5e-2
1e:1-
-eel
91.5
93.5
Acc(%) 1
Acc(%)
Sum digits
Active pixels
Learning g rate
-le
5.04
-5.04
1e-3-
(0I-
1e-3
Se:3-
$
-e
1e-2-
(++
-le2
5e-2-
1e-1-
2.5
3.0
3.5
4.0
4.5
5.0
0.1
0.2
0.3
0.4
MSE
MSE
Product of digits odd
Digit-related tasks
0-
0
1e-4
1e-4
5-e4
5-e4
1ee3-
-ee-3
5e-3-
5e-3
1e-2-
1e-2
5e-2-
5e-2
1e:1-
-le-1
75
80
85
90
-30
-25
-20
-15
-10
-5
F1-score(%)
avg. A.
Figure 8: RotoGrad's performance on all tasks for the experiments in $6.1 for all metrics. We can observe training instabilities/stiffess on all tasks we highly increase/decrease RotoGrad's learning rate, as discussed in the main manuscript.
C.2.4 CIFARIO AND CELEBA
For the sake of completeness, we present Table and Table the same tables as $6, but with more statistics of the results. For CIFAR10, we now included in Table 6 the minimum task improvement across tasks and, while noisier, we can still observe that RotoGrad also improve this statistic. The standard deviation of the task improvement across tasks is, however, not too informative. We also include in Figure the cosine similarity plots for the different subspace rotations from $6.3, showing clear positive correlation between the cosine similarity and the size of the considered subspace. While the cosine similarities look low, we want to remark that we are computing the cosine similarity of huge space, and we only align subspace of it. If we, instead, showed the cosine similarity with respect to each specific subspace, the cosine similarity should look similar to that of RotoGrad 512. In the case of CelebA, we added in Table the maximum f'lscore across tasks and, similar the last case, it is not too informative, as all methods achieve almost perfect f'l f'l-score in one of the classes. We also include the training times for some baselines, showing that RotoGrad stays on par with them.
C.2.5 NYUV2
Complementing the results shown in $6, show in Table 8 the results obtained combining RotoGrad with all other existing methods (rows within RotoGrad +), for gradient scaling methods we only apply the rotation part of RotoGrad. Results show that RotoGrad helps improve/balance all other
22

---

Published as a conference paper at ICLR 2022
Table Complete task musk-improvement statistics in CIFARIO for all competing methods and RotoGrad with different dimensionality for Rg, see $6.
Method
d
ming A
med, As
avgg A
stdı A +
max; A 1
Vanilla
0
-0.81 +: 0.37
1.90 + 0.53
2.58 + 0.54
3.38 + 0.94
11.14 1: 3.35
RotoGrad
64
-1.70
1.79 0.57
2.90 0,49
3.98 0.62
13.16 2.40
RotoGrad
128
1.32
2.25 + 1.07
2.97 1.08
1.887
12.64 : 3.56
RotoGrad
256
0.17 1.01
2.16
3.68 + 0.68
0.74
14.01 + 3.22
RotoGrad
512
0.76
3.67 1.40
4.48 + 0.99
4.23
15.57 3.99
Vanilla!
-0.81 0.37
1.90
2.58 0.54
3.38 0.94
11.14 3.35
Without Ri.
GradDrop
0.33
1.20
3,41 1 0.45
1.34
13.58 + 1.50
PCGrad
0.99
1.95
2.86
0.69
12.01 3.19
MGDA-UB
-1.21 1.36
-1.75
3.55
3.67 + 0.98
GradNorm
-0.35 0.59
2.45
3.23:0.35
4.02
14.25 + 1.35
IMTL-G
-0.39 -0.99
1.97
2.73 0.27
3.25
10.20 2.98
Vanilla
-0.85 -0.55 0.58
3.10 1.29
3.12
4.05 + 0.56
14.23 2.86
GradDrop
0.778
3.27 1.61
3,54 1.10
4.11 0.56
13.88 + 2.95
With R
(d = 512)
PCGrad
0.58
2.67 0.88
3.29
3.90
13.44 13.44:10 1.86
MGDA-UB
0.57
0.21 0.67
0.52
4.78 2.15
GradNorm
-0.79
3.10
3.21
0.86
10.88 1 4.73
IMTL-G
0.52
1.81 0.87
3.02 0.69
3.81
12.76 - 1.77
RotoGrad
0.76
3.67 1.40
4.48 + 0.99
4.23 = 0.82
15.57 3.99
Table Complete fl-score statistics and training hours in CelebA for all competing methods and two different architectures/setinngs (median over five runs), see $6. For the convolutional network we use m 256, and m 1536 for the residual network.
Convolutional (d 512)
ResNet18 (d 2048)
task fl-scores (%) f
task fl-scores (%) f
Method
ming
med; avga std; + maxi: Hours ming
meds avga stdj. 4 max; Hours
Without R
Vanillat
1.62 53.39 58.49 24.26 96.97
7.62
15.45
63.04 62.85 22.09 96.58
1.49
GradDrop
2.63 52.32 57.33
25.27
96.72
8.53
13.31
64.37
63.95
20.93
96.59
1.60
PCGrad
2.69
54.60
56.87
25.75
97.04
34.05
13.61
62.45
62.74
21.60
96.64
5.75
GradNorm
2.17
52.98
56.91
24.72
96.84
20.93
17.42
62.49
62.62
21.93
96.55
3.61
IMTL-G
0.00 14.81
31.90
33.58
93.31
9.46
9.87
62.22
62.03
22.47
96.51
1.73
Vanilla
4.24
49.85
55.33
26.03
96.88
16.29
19.71
63.56
63.23
21.16
96.55
9.33
GradDrop
3.18 50.07
54.43
27.21
96.80
17.20
12.33
62.40
62.74
21.74
96.65
9.41
With R&
PCGrad
1.44
53.05
54.72
27.61
96.90
41.79
14.71
63.65
62.61
22.22
96.59
13.72
GradNorm
2.08 52.53
56.71
24.57
96.96
30.02
9.05
60.20
60.78
22.31
96.38
11.36
IMTL-G
0.00
37.00
42.24
33.46
94.34
18.05
17.11
61.68
60.72
22.80
96.44
9.52
RotoGrad
4.59
55.02
57.20
24.75
96.79
27.20
9.96
63.84
62.81
21.80
96.45
6.68
23

---

Published as a conference paper at ICLR 2022
Vanilla
RotoGrad 128
RotoGrad 512
RotoGrad
RotoGrad 256
0.375
0.350
Cosine similarity
0.325
0.300
0.275
0.250
0.225
0.200
0
10k
20k
30k
40k
50k
60k
70k
80k
Iterations
Figure 9: Cosine similarity between the task gradients and the update gradient on CIFAR10. Results averaged over tasks and five runs, showing 90 % confidence intervals (shaded areas).
Table 8: Results for different methods on the NYUv2 dataset with a SegNet model. RotoGrad obtains the best performance in segmentation and depth tasks all metrics, while significantly improving results on normal surfaces with respect to vanilla case.
Relative improvements
Segmentation
Depth
Normal Surfaces
Time 1
Angle
Within
Method
As
Ap
Ax
mloU
Ace
Abs.
Rel.
Mean
Median
11:25
22.5
30
h
Single
0.0
0.0
0.0
39.21
64.59
0.70
0.27
25.09
19.18
30.01
57.33
69.30
8.90
GradDrop
26.47
53.38
66.05
3,94
RotoGrad
PCGrad
-0.0
19.7
-8.3
39.08
64.68
0.54
0.21
26.41
21.29
26. 13
52.99
65.
3.89
MGDA-UB
2.5
23.2
8.1
39.32
65.48
0.51
0.21
26. 43
21.22
26. 16
53.16
66.07
3.85
GradNorm
21.4
52.59
65.52
IMTL-G
1.7
21.2
6.9
40.13
65.17
0.55
0.21
26.20 20
21.06
26.69
53.39
66.04
3.96
With 1021)
Rotate Only
3.3
20.5
-6.6
39.63
66.16
0.53
0.21
26.12
20.93
26.85
53.76
66.50
3.82
Scale Only
-03
20.0
-7.9
38.89
65.94
0.54
0.22
26.47
21. 21.24
26.24
58.04
65.81
3.87
RotoGrad
1.8
24.0
-6.1
339.32
66.07
0.53
0.21
26.01
20.80
27.18
54.02
66.53
3.83
Vanilla
2.7
20.6
-5.77
38.05
6-1.99
0.54
0.22
30.02
26.16
20.02
43.47
56.87
3.81
GradDrop
-0. 14.0
-5.22
38.79
64.36
0.59
0.24
29. 80
25.81
19.88
44.08
57.
4.01
PCGrad
-2.7
20.5
-26.3
37.15
63.41
0.55
0.22
30.06
26.18
19.58
43.51
56.87
3.89
MGDA-UB
-31.2
-0.7
0.6
21.60
51.60
0.77
0.29
24.74
18.90
30.32
57.95
69.88
3.85
GradNoem
-0.6
19.5
10.55
37 22
63.61
0.54
0.22
26.68
21.67
25.95
52.16
64.95
3.85
IMTL-G
-0.3
17.6
77.5
38.38
641.66
0.51
0.22
26.38
2..35
26.56
52.84
65.69
3.99
Vanillal
16.8
-25.0
37.11
63. 98
0.56
0.22
29.93
25.89
20.34
43.92
57.39
3.46
Without R
GradDrop
-0.1
15.7
-27.0
37.51
63.62
0.59
0.23
30.15
26.33
19.32
43.15
56.59
3.55
PCGrad
-0.5
20.0
-24.6
38.51
63.95
0.55
0.22
29.79
25.77
20.61
44.22
57.64
3.51
MGDA-UB
32.2
20.75
51.44
0.73
0.28
21.70
18.92
30.57
57.95
69.99
3.52
GradNorm
2.2
20.6
10.2
39.29
64.50
0.53
0.22
26.77
21.88
25.39
51.78
64.76
3.50
IMTL-G
1.9
21.4
6.7
39.94
65. 96
0.55
0.21
26.23
21.14
26.77
53.25
66. 66.22
3.61
methods, which is specially true for those methods that heavily overlook some tasks. Specifically, MGDA-UB stops overlooking the semantic segmentation and depth estimation tasks, while PCGrad and GradDrop stop completely overlooking the surface normal loss. Note that we also show in Table 8 the training times of each method, and RotoGrad stays on par with non-extended methods in training time. As mentioned in Appendix C.I, due to cluster overload, some times were deceivingly high (specifically those baselines with R₂)) as we had to run them on different machines, and were omitted to avoid confusion.
24


Reviews:
[{"review_id": "eEhSzcPxgm0", "reviewer": "Reviewer_gsqy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- **Questions:**\n    1. Why do you need to specifically enforce $R_k$ to be a rotation matrix? Why it cannot be free to be a (non)-linear transformation? Have you tried experiments in this direction? Would that make sense?\n    2. Regarding the comparison with other methods: From the supplementary, it looks that the PCGrad projection has been computed differently (wrt features rather than weights), can you elaborate more on this difference? \n    3. By looking at the PCgrad paper (which seems to be your main \"competitor\") I found that they apply pcgrad in combination with other multitask methods, e.g. MTAN. Can ROTOGRAD work together with other methods? Did you try these experiments?\n    4. It would be great to go beyond supervised learning experiments and include multitask RL experiments in your analysis.\n- **Minors**\n    - I suggest guiding the reader a bit more on table1, by highlighting the best and worst results as you do in the supplementary.\n", "summary_of_the_paper": "- This paper proposes a new way of dealing with negative transfer or catastrophic interference in multitask learning. The main idea is to avoid conflicting gradients by making them homogeneous both in terms of magnitude and direction. Gradient Magnitude are normalized by taking into account the convergence rate of all tasks. Directions are homogenised by smoothly rotating the feature-space introducing rotation matrices $R_k \\in SO(d)$ parametrized via exponential maps on the Lie algebra of $SO(d)$. \n- Rotation matrices are optimized to reduce the direction conflicts by maximizing the cosine similarity within the the batch, while the resto of the network is trained to minimize the multitask loss. The training algorithm result in an alternated optimization procedure that can be interpreted as a Stackelberg game with a leader and a follower alternatively minimizing their own losses.\n", "summary_of_the_review": "The paper is solid and it looks has received several iterations. It is a nice contribution to the multi-task learning field, although I think there are still some questions to be addressed first. I'm leaning towards acceptance and I'll be more than happy to increase my rate after the rebuttal and discussion with other reviewers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NiRBcZ_NcaB", "reviewer": "Reviewer_cExt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n- The paper is well written and well structured.\n- The paper tackles an important problem and proposes an intuitive approach for alleviating the problem. The motivation for the method is clear.\n- The strong experimental section presents a clear improvement over previous MTL approaches.\n\n**Weaknesses:**\n- It would be useful to fully observe and understand the tradeoff between $m$ and runtime, as the scalability of the approach for large values of $d$ is limited.\n- It appears the proposed update step may increase the loss (objective) for some tasks. While intuitive, it’s not clear to me if the average gradient is the right choice for the target vector ($v_n$). Have you experimented with other choices for $v_n$, for example, the common descent direction in MGDA?\n- Missing citation to relevant works: [1]  propose a method to handle conflicting gradients in MTL. [2] theoretically discuss challenges of MTL optimization. Specifically, while different significantly than the proposed method, their covariance alignment approach appears to relate to the rotation matrices in the proposed method (although being applied to the input instead of the shared feature space).\n\nExperiments:\n- Please report standard errors or some other measure of variability for the results in Table 1. How many random seeds were considered?\n- According to C1, you modified the implementation of the SOTA approach PCGrad to compute gradient w.r.t $z$, instead of all shared parameters as in the original paper. It is unclear how this modification affects PCGrad’s performance and if this choice maintains a fair comparison.\n- Missing details of hyperparameters tuned for each method.\n\n\n[1] Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models, ICLR 2021.\n\n[2] Understanding and Improving Information Transfer in Multi-Task Learning, ICLR 2020.\n\n\n", "summary_of_the_paper": "This paper proposes a novel algorithm to tackle negative transfer in MTL optimization, by homogenizing the task gradients’ magnitude and direction. The method first homogenize gradients’ norm towards the norm of the task that converged the least. Next, it employs rotation matrices to align task gradients with the direction of the average gradient.\n", "summary_of_the_review": "Overall it is a good paper with clear motivation and an intuitive approach. It tackles a major challenge in MTL optimization. Strong empirical results highlight the benefit of the approach compared to previous related methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ASq2SCwaBVl", "reviewer": "Reviewer_2soJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Early recent work (2015-2018) in multi-task learning focused on dynamic task reweighing methods (i.e. how to change the loss scale throughout training), but it seems that the community has shifted to focusing on direct gradient modifications (2020-2021) with a plethora of methods being developed that modify both the scale and direction of the gradient update on the shared parameters (PCGrad, GradDrop, GradVac, IMTL, CAGrad, etc.). With at least 5 somewhat conceptually similar methods having been released in the past year, it's more important than before for new approaches in this space to offer a new perspective or approach this challenge from a different direction.\n\nIt is my opinion this work meets the requisite threshold. The concept of applying (and learning) a geometric rotation to align the optimization trajectories is novel; although, the concept of gradient-magnitude homogenization has been well-studied, with the presentation offered by this work feeling very reminiscent of GradNorm. Returning to the former point, I have not seen a rotation transformation in SO(d) applied to MTL -- the closest analogue are modulation modules which filter/modify information in either the forward and/or backward pass -- and I think the dissemination of this perspective may lead to other interesting and powerful work. \n\nNevertheless, the current conceptualization of rotation to maximize cosine similarity may have some problems. Two identical tasks with identical initialization will have perfect alignment, and results in an effective doubling of the learning rate for the shared parameters, but does not lead to the benefits that may be conveyed from multi-task learning models (i.e. inductive bias on different hypothesis classes, regularization, etc.). Moreover, the (hyper)parameter free approach which enforces a strict constraint on equal norm among all gradients seems highly prohibitive (and perhaps undesirable since certain tasks should naturally train at different rates?). That said, many of the methods I mention in this first paragraph adopt a similar perspective with regards to a correlation between positive cosine similarity and modeling performance, and one can invert the question to posit that two tasks with perfectly opposite gradients will not even begin to train.\n\nStrengths:\n\n1. The approach is well motivated with illustrated examples and intuition.\n2. Extremely well written and polished paper. \n3. Empirical methodology is sound. It appears as if all methods are fairly compared and assessed.\n4. Experimental results are compelling.\n\nWeaknesses:\n\n1. I don't see any significant flaws in this work.\n\nNitpicks:\n\n1. *to the shard feature z* => to the shared feature representation z.\n2. The results in the table are a bit difficult to read as nothing stands out. Perhaps rerun the NYUv2 experiments 3-5 times and bold significant results? \n3. *While recent work has acknowledged that negative transfer is a two-fold problem, existing approaches fall short as they only focus on either homogenizing the gradient magnitude across tasks; or greedily change the gradient directions, overlooking future conflicts.* => This sentence is grammatically off. Consider breaking it into two smaller sentences. \n4.  *Previous works have tracked down this issue to the disparities in gradient magnitudes and directions across tasks, when optimizing the shared network parameters.* => remove the comma or bring the dependent clause (when optimizing the shared network parameters) to the beginning of the sentence (i.e. When optimizing the shared network parameters, prior wok has indicated discrepancies in gradient magnitude among tasks may inhibit model performance or something like this).\n5. Several *very* recent papers have been overlooked that should probably be included at different points in the related work section.\n\n[1] Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models, ICLR 2021 (spotlight)\n\n[2] Conflict-Averse Gradient Descent for Multi-task learning, NeurIPS 2021\n\n[3] Efficiently Identifying Task Groupings for Multi-Task Learning, NeurIPS 2021 (spotlight)\n\n[4] Variational Multi-Task Learning with Gumbel-Softmax Priors, NeurIPS 2021", "summary_of_the_paper": "This work offers an approach to improve the training dynamics of multi-task neural networks by making two assumptions:\n\n1. Magnitude differences among task gradients inhibit convergence.\n2. Directional differences (often termed \"conflicting gradients\") in the gradients among tasks inhibit convergence. \n\nIt adapts the optimization process by:\n\n1. Ensuring all gradients have equal norm. \n2. Introducing a geometric rotation in SO(d) transformation after the final shared feature map and before the task-specific layers.\n\nThe intuition underpinning this method is supported with several illustrative (toy) examples as well as evaluation on MNIST, CelebA, CIFAR-10, and NYUv2.", "summary_of_the_review": "I think this is a good paper. It is clearly written, the empirical evaluation is fair, and presents a novel perspective (rotation) to reduce inter-task conflict in multi-task learning paradigms. \n\nI recommend it for acceptance and have no additional or follow-up questions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "38ap85kdh9z", "reviewer": "Reviewer_uDos", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed optimization scheme appears to be a unique technical approach among a larger set of recent work that attempt to deconflict gradients in multi-task learning.  Prior work has, e.g., focused on gradient projection operations [Yu et al., 2020], which, while similar in motivation, may lack the flexibility of the proposed approach's learnable rotations.\n\nExperiments demonstrate proof-of-concept on synthetic optimization problems, as well as good results on real datasets (CIFAR10, CelebA, NYUv2).  Results on NYUv2 (semantic segmentation, depth estimation, normal estimation) look especially promising in comparison to both a baseline system and recently-published multi-task optimization procedures.\n\nOverall, the proposed RotoGrad approach seems quite promising and could become a standard tool for multi-task neural network training.  It would be interesting to see results on larger-scale problems (both larger networks and larger datasets), as well as examine how RotoGrad might be applicable in transfer-learning and meta-learning scenarios.\n", "summary_of_the_paper": "This paper introduces a training algorithm for neural networks that reduces conflict between multiple output tasks that utilize a shared portion of the network.  Considering a common shared representation from which multiple task-specific subnetworks branch, the approach aims to homogenize task-specific gradient magnitudes and directions at this branch point.  The proposed strategy is to introduce additional parameterized rotation matrices, each of which modifies the shared representation before it is passed to a corresponding task-specific branch.  The parameters of these rotation matrices are optimized to maximize gradient similarity between different tasks at the branch point; this optimization step is interlaced with standard updates of other network parameters to minimize total task loss.", "summary_of_the_review": "This paper presents a parameterization scheme and optimization procedure for improved training of shared multi-task networks.  The technical approach is concise, well-motivated, and novel with respect to existing work.  Experiments across synthetic and real datasets show convincing results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of inconsistent gradients in multi-task learning, proposing ways to handle both their magnitude nd direction. Gradient directions are aligned by introducing a rotation layer between the shared backbone and task-specific branches. 
Reviewers appreciated the technical approach, higlighting the novelty of the rotation layers in this context. The empirical evaluations are systematic fair and insightful, and the presentation is polished. Reviewers unanimously supported accepting the paper.
Decision: Accept (Spotlight)