RotoGrad: Gradient Homogenization in Multi-Task Learning
Authors: Adrián Javaloy
Isabel Valera
Venue: NeurIPS 2021 Submitted (2021)
Keywords: multi-task learning
gradient conflict
negative transfer

Abstract: Multi-task learning is being increasingly adopted in applications domains like computer vision and reinforcement learning. However, optimally exploiting its advantages remains a major challenge due to the effect of negative transfer. Previous works have tracked down this issue to the disparities in gradient magnitudes and directions across tasks, when optimizing the shared network parameters. While recent work has acknowledged that negative transfer is a two-fold problem, existing approaches fall short as they focus only on either homogenizing the gradient magnitude across tasks; or greedily change the gradient directions, overlooking future conflicts. In this work, we introduce RotoGrad, an algorithm that tackles negative transfer as a whole: it jointly homogenizes gradient magnitudes and directions, while ensuring training convergence. We show that RotoGrad outperforms competing methods in complex problems, including multi-label classification in CelebA and computer vision tasks in the NYUv2 dataset.

RotoGrad: Gradient Homogenization in Multi-Task Learning
Anonymous Author(s)
Affiliation Address email
Abstract
Multi-task learning is being increasingly adopted in applications domains like computer vision and reinforcement learning. However, optimally exploiting its ad- vantages remains major challenge due to the effect of negative transfer. Previous works have tracked down this issue to the disparities in gradient magnitudes and directions across tasks, when optimizing the shared network parameters. While recent work has acknowledged that negative transfer is two-fold problem, exist- ing approaches fall short as they focus only on either homogenizing the gradient magnitude across tasks; or greedily change the gradient directions, overlooking future conflicts. In this work, we introduce RotoGrad, an algorithm that tackles negative transfer as a whole: it jointly homogenizes gradient magnitudes and direc- tions, while ensuring training convergence. We show that RotoGrad outperforms competing methods in complex problems, including multi-label classification CelebA and computer vision tasks in the NYUv2 dataset.
10
11
12
Introduction
14
As neural network architectures get larger in order to solve increasingly more complex tasks, the 16 idea of jointly learning multiple tasks (for example, depth estimation and semantic segmentation in computer vision) with single network is becoming more and more appealing. This is precisely the 18 idea of multi-task learning (MTL) [3], which promises higher performance in the individual tasks and better generalization unseen data, while drastically reducing the number of parameters [27].
20
Unfortunately, sharing parameters between tasks may also lead difficulties during training as tasks compete for shared resources, often resulting in poorer results than solving individual tasks, phenomenon known as negative transfer [27]. Previous works have tracked down this issue to the two types of differences between task gradients. First, differences in magnitude across tasks can make some tasks dominate the others during the learning process. Several methods have been proposed to homogenize gradient magnitudes such as MGDA [28]. GradNorm [6], or IMTL-G [18]. However, little attention has been put towards the second source of the problem: conflicting directions of the gradients for different tasks. Due to the way gradients are added up, gradients of different tasks may cancel each other out if they point to opposite directions of the parameter space, thus leading to a poor update direction for subset or even all tasks. Only very recently handful of works have started to propose methods to mitigate the conflicting gradients problem, for example, by removing conflicting parts of the gradients [33], randomly 'dropping" some elements of the gradient vector [7].
21
22
23
24
26
28
30
31
32 In this work we propose RotoGrad, an algorithm that tackles negative transfer as whole by ho- mogenizing both gradient magnitudes and directions across tasks. RotoGrad addresses the gradient magnitude discrepancies by re-weighting task gradients each step of the learning, while encourag- ing learning those tasks that have converged the least thus far. In that way, it makes sure that no task is overlooked during training. Additionally, instead of directly modifying gradient directions, RotoGrad
34
36
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

2
Vanilla
Naniila
FE.2
RotoGrad
Z1
Z1
2
21
2
fk. (b) Non-convex experiment.
(a) Convex avocado-shaped experiment.
Figure I: Level plots showing the evolution of two regression MTL problems with/without RotoGrad, see Section RotoGrad able reach the optimum (1x) for both tasks. (a) In the space of
RotoGrad rotates the function-space, to align task gradients (blue/orange arrows), finding shared features (green arrow) closer to the (matched) optima.
(b) In the space of rk. RotoGrad rotates the shared feature providing per-task features TĄK that better fit each task.
37 smoothly rotates the shared feature space differently for each task, seamlessly aligning gradients in the long run. As shown by our theoretical insights, the cooperation between gradient magnitude- 39 and direction-homogenizzttoo ensures the stability of the overall learning process. Finally, we run extensive experiments empirically demonstrate that RotoGrad leads to stable (convergent) learning,
40
41 scales up to complex network architectures, and outperforms competing methods in multi-label
42 classification settings in CIFARIO and CelebA, well as computer vision tasks using the NYUv2
43 dataset. Alongside this paper, we will provide simple-to-uss library to include RotoGrad in any Pytorch pipeline with few lines of code.
44
Multi-task learning and negative transfer
45
48
The goal of MTL is to simultancously learn K different tasks, that is, finding K mappings from common input dataset X E N to task-speeific set of labels Y, E Y Most settings consider hard-parameter sharing architecture, which is characterized by two components: the backbone and heads networks. The backbone uses set of shared parameters, 0, to transform each input æ E X
47
48
49
50 into shared intermediate representation z = f(x; 0) Rd, where d is the dimensionality of
51 Additionally, each task & = 1,2..... has head network hi. with exclusive parameters Φi. that takes this intermediate feature and outputs the prediction (x) = hi (z; Φb) for the corresponding 53 task. This architecture is illustrated Figure 2, where we have added task-specific rotation matrices R& that will be necessary for the proposed approach, RotoGrad. Note that the general architecture described above is equivalent to the one in Figure 2 when all rotations R& correspond to identity matrices, such that тą for all k.
55
57
MTL aims learn the architecture parameters 0, Φ1. $ ΦK by simultaneously minimiz-
heg
58
Ri > P1
ing all task losses, that is, L& (x), yya) for ......K. K. Although this is priori multi- 61 objective optimization problem [28], in practice single surrogate loss consisting of a linear com- 63 bination of the task losses, L = Σx Lg. is op- timized. While this approach leads to simpler optimization problem, it may also trigger nega- tive transfer between tasks, hurting the overall MTL performance due to an imbalanced competition among tasks for the shared parameters [27].
59
k 1,....
f0
h0z (2(22(12), (T2),
a
2
R2 + 12
Rx PK
hok LK(h (rx),
Figure 2:
Hard- Hard-parameter sharing architecture in- cluding the rotation matrices R2 of RotoGrad.
65
67
68
The negative transfer problem can be studied through the updates of the shared parameters 0. At each 89 training step, updated according to linear combination of task gradients, V = Σ. wa V Lı, which may suffer from two problems. First, magnitude differences of the gradients across tasks may lead a subset of tasks dominating the total gradient, and therefore to the model prioritizing them over the others. Second, conflicting directions of the gradients across tasks may lead to update
71

---

73 directions that do not improve any of the tasks. Figure 1 shows an example of poor direction updates
74 (left) as well as magnitude dominance (right).
75 In this work, we tackle negative transfer as a whole by homogenizing tasks gradients both in magnitude and direction. Note that homogenizing gradients with respect to 0 is equivalent homogenizing
77 gradients with respect to the shared feature 2 due to the chain rule, V = Vez VzL2. Thus,
78 from now on we focus on homogenizing the feature-level task gradients V VzLg.
3 RotoGrad
79
80
In this section we introduce RotoGrad, a novel algorithm that addresses the negative transfer problem 81 as whole. RotoGrad consists of two building blocks which, respectively, homogenize task-gradient
82 magnitudes and directions. Moreover, these blocks complement each other and provide convergence
83 guarantees of the network training. Next, we detail each of these building blocks and show how they 84 are combined towards an effective MTL learning process.
85
3.1 Gradient-magnitude homogenization
As discussed in Section 2, aim homogenize gradient magnitudes across tasks, as large magnitude 87 disparities can lead to subset of tasks dominating the learning process. Thus, the first goal of RotoGrad is to homogenize the magnitude of the gradients across tasks at each step of the training.
89 Let us denote the feature-level task gradient of the k-th task for the n-th datapoint, at iteration t, by Pn.k V Lhs: Vn.k). and its batch versions by G := [g 92. xR, where B is 91 the batch size. Then, equalizing gradient magnitudes amounts to finding weights wj that normalize and scale each gradient GR,, that is,
C
He GalGlll = |w, Vi
- wa GR
Ga CUR
Vk,
||
93 where UR | Note that, in the above expression, C' is free parameter that we need to select.
G. denotes the normalized task gradient and C is the target magnitude for all tasks.
94
In RotoGrad, we select C' such that all tasks converge similar rate. We motivate this choice by the fact that, by scaling all gradients, we change their individual step size, interfering with the convergence guarantees provided by their ipsshiiz-smmoothness (for an introduction non-convex optimization see, for example, [25]). Therefore, we seek for the value of C' providing the best step-size for those tasks that have converged the least up to iteration t. Specifically, we set C to be convex combination of the task-wise gradient magnitudes, C' := Σn or | |l. where the weights
96
97
98
99
100
101
01 .2: OKK measure the relative convergence of each task and sum up to one, that is,
||GxI//I|2|
06 LIGH///I
with GO being the initial gradient of the k-th task, i.c., the gradient at iteration 0 of the training.
103
As result, we obtain (haaerrparameter--fee approach that equalizes the gradient magnitude across tasks to encourage learning slow-converging tasks. Note that the resulting approach resembles Normalized Gradient Descent (NGD) [8] for single-task learning, which has been proved to quickly escape saddle points during optimization [24]. Thus, we expect similar behavior for RotoGrad, where slow-converging tasks will force guick-converging tasks to escape from saddle points.
104
105
107
108
The resulting training algorithm may however diverge as consequence of constantly oscillating between convergigg) tasks. For example, in scenarios where one task improves, there is always another task(s) that deteriorates. Fortunately, as shown in the following result (proof in Appendix A), such a phenomenon does not appear in the absence of conflicting gradients.
109
110
111
Proposition 3.1. Let G1, G.y..., B22....,G,, ,, be the task gradients with respect to Z as defined above. If 113 K = 2; or cos sim sim(G, G,) 2 0 pairwise; then there exists a small-enough step size E > 0 such that, for all tasks, we have that (h - - Un: σb);: Yx) < La (Z Φu)): Yx).
112
115 In other words, Proposition 3.1 shows that, when gradients do not conflict in direction with each other, following the feature-level gradient C C2 Up improves all bounde- task losses for
3

---

117
the given batch. This result, while restricted to the given batch and to the gradient with respect to the shared representation Z. still provides useful insights in favor of having as desideratum of an efficient MTI. pipeline the absence of conflicting gradients.
118
119
120
3.2
Gradient-direction homogenization
121
In the previous subsection, we have shown that avoiding conflicting gradients may not only be necessary to avoid negative transfer. but also to ensure the stability of the training. In this section we introduce the second building block of RotoGrad, an algorithm that homogenizes task- -raddiett directions. The main idea of this approach is to smoothly rotate the feature-space in order to reduce the gradient conflict between tasks-in following iterations-or the training by bringing (local) optima for different tasks closer to each other (in the parameter space). As result, it complements the previous magnitude-scaling approach and reduces the likelihood of the training to diverge.
122
123
124
125
126
127
128 In order to homogenize gradients, for each task k = ,,..,, K, RotoGrad introduces a matrix Ra so that, instead of optimizing L3 Lx(2) with z being the last shared representation, we optimize an equivalent loss function L3 (R As we are only interested in changing directions (not the gradient magnitudes), we choose R& E (O(d) to be rotation matrix' leading to per-task representations
129
130
131
т = 22 RotoGrad thus extends the standard MTL architecture by adding task-specific rotations before each head, as depicted in Figure 2.
132
133
134
Unlike all other network parameters, matrices R& do not seek to reduce their task's loss. Instead, these additional parameters are optimized to reduce the direction conflict of the gradients across tasks. To this end, for each task we optimize R2 to maximize the batch-wise cosine similarity or, equivalently, to minimize
135
136
137
c == - E In.k, Ca),
138 where In.k= == Vr, L& (wn,, Vn,,)) (which holds that Ir,k = R 9n. and vn is the target vector that we want all task gradients pointing towards. We set the target vector Ur to be the gradient we would have followed if all task gradients weighted the same, that is, Vn == K Σn Un,,k, where un,k is a row vector of the normalized batch gradient matrix UR. as defined before.
139
141
142
As result, in each training step of RotoGrad we simultaneously optimize the following two problems:
143
Network: minimize 0,(+)x
Σ Wy Lk.,
Rotation: minimize [R
Σ C
(4)
The above problem can be interpreted
Stackelberg game: two player- player-game in which leader and follower alternately make moves in order to minimize their respective losses, Lı and Ly, and the leader knows what will be the follower's response to their moves. Such an interpretation allows to derive simple guidelines to guarantee training convergence-than is, that the network loss does not oscillate as result of optimizing the two different objectives in Equation 4. Specifically, following Fiez et al. [10], we can ensure that problem 4 converges as long as the rotations' optimizer (leader) a slow-learner compared with the network optimizer (follower). That is, long as we make the rotations' learning rate decrease faster than that of the network, we know that RotoGrad will converge local optimum for both objectives. more extensive discussion can be found Appendix B.
145
146
147
148
149
150
151
152
153
3.3
RotoGrad: the full picture
154
After the two main building blocks of RotoGrad, we can now summarize the overall proposed approach in Algorithm 1. At each step, RotoGrad first homogenizes the gradient magnitudes such that there is no dominant task and the step size is set by the slow-converging tasks. Additionally, RotoGrad smoothly updates the rotation matrices- -usnn the local information given by the task gradients-te to seamlessly align task gradients in the following steps, thus reducing direction conflicts.
155
156
157
158
159
3.4
Practical considerations
160
In this section, we discuss the main practical considerations to account for when implementing RotoGrad and propose efficient solutions.
181
TTee special orthogonal group. SO(d), denotes the set of all (proper) rotation matrices of dimension d.

---

Algorithm Training step with RotoGrad
Input input samples X, task labels {Yx,, network's (RotoGrad's) learning rate 7) (Troto)
Output backbone (heads) parameters 0 (fØx D), RotoGrad's parameters {
1: compute shared feature Z - ()
2: for k = ..2...., K do
3: compute task-specific loss Li En, Lx (hg (Rxzni Br.kk
4: compute gradient of shared feature G. = VzLb
5: compute gradient of task-specific feature GR = Gg
> Treated as constant w.r.t. R
6: compute unitary gradients U2- G2/ Gx
7:
compute relative task convergence 0g ||
8: end for
9: make } sum up to one [ar, 02,...,
[a 022 ,,,,..,,,, (a,, axl//E On
10: compute shared magnitude C = 2 0n Gk
11: update backbone parameters 0
C Σx Un
12: compute target vector V
Σx Un
13: for k = 2, K do
compute RotoGrad's loss L = -, (R kn Un
m
15: update RotoGrad's 's parameters R₁ = Rsg- Nrooo V R. L
update head's parameters Φ = Φk v/ Li
17: end for
Unconstrained optimization. As previously discussed, parameters R& are defined rotation matrices, and thus the Rotation optimization in problem 4 is a constrained problem. While this would typically imply using expensive algorithms like Riemannian gradient descent [1], can leverage recent work on manifold parametrization [5] and, instead, apply unconstrained optimization methods by automatically² parametrizing R2 via exponential maps on the Lie algebra of SO
163
165
166
167
Memory efficiency and time complexity. Second, as we need one rotation matrix per task, we have to store d(d additional parameters. In practice, we only need Kd(d - 1)/2 parameters due to the aforementioned parametrization and, in most cases, this amounts to a small part of the total number of parameters. Moreover, described by Casado et al. [5], parametrizing Rg enables efficient computations compared with traditional methods, with a time complexity of ((d)) independently of the batch size. In our case, the time complexity is of ((Kd),, which scales better with respect the number of tasks than existing methods (for example, OKK²d) d for PCGrad [33]). Moreover, caching R3 in the forward pass and GPU parallelization can further reduce training time.
169
171
173
174
175
Scaling-up RotoGrad. Even though we can efficiently compute and optimize the rotation matrix RS. in some application domains, like computer vision, in which the size d of the shared representation is large, the time complexity for updating the rotation matrix may become comparable to the one of the network updates. In those cases, we propose to only rotate subspace of the feature space, that
176
177
179 is, rotate only m << d dimensions of z. Then, we can simply apply a transformation of the form T R 21:m Zm+1:d), where Za:b denotes the elements of with indexes o, 1,...,d b. While there exist other possible solutions, such as using diagonal rotation matrices Rg, we defer them
181
to future work.
183 4 Illustrative examples
In this section, we illustrate the behavior of RotoGrad in two synthetic scenarios, providing clean qualitative results about effect on the optimization process. Appendix C.I provides detailed description of the experimental setups.
185
187
To this end, we propose two different multi-task regression problems of the form
L(x) Li (x) + L2(x) = [(;; (z; + s (x) 0),
(5)
where yp is test function with single global optimum whose position is parametrized by the second argument, that is, both tasks are identical (and thus related) up to a translation. We use a single input
189
For example, Geotorch [4] makes this transparent to the user.
5

---

190
I22 and drop task- network parameters. As backbone, we take simple network of the form W2 max 2 b, 0) + b2 with b, Ro, b2 E R², and WT. W 10xx2
191
192
For the first experiment we choose a simple (avocado-shaped) convex objective function and, for the second one, opt for non-convex function with several local optima and single global optimum. Figure shows the training trajectories in the presence (and absence) of RotoGrad in both experiments, depicted as level plots in the space of and rk. respectively. We can observe that in the first experiment (Figure la), RotoGrad finds both ophima-which is in stark contrast to the vanilla case by rotating the feature space and matching the (unique) local optima of the tasks. Similarly, the second experiment (Figure 1b) shows that, as we have two symmetric tasks and a non-equidistant starting point, in the vanilla case the optimization is dominated by the task with an optimum closest to
198
194
195
196
197
198
199
200
the starting point. RotoGrad avoids this behavior by equalizing gradients and, by aligning gradients, able to find the optima of both functions.
201
Related Work
202
Understanding and improving the interaction between tasks is one of the most fundamental problems of MTL, since any improvement in this regard would translate to all MTL systems. Consequently, several approaches to address this problem have been adopted in the literature. Among the different lines of work, the one most related to the present work is gradient homogenization.
204
205
206
Gradient homogenization. Since the problem is two-fold, there are two main lines of work. On the one hand, we have task-weighting approaches that focus on alleviating magnitude differences. Similar to us, GradNorm [6] attempts to learn all tasks at similar rate, yet they propose to learn these weights as parameters. Instead, we provide a closed-form solution in Equation 1, and so does IMTL-G [18]. However, IMTL-G scales all task gradients such that all projections of G onto GR are cqual. MGDA [28], instead, adopts an iterative method based on the Frank- Wolfe algorithm in order A W2 G2 has minimum norm. On the other hand, recent works have started to put attention on the conflicting direction problem. Maninis et al. [22] first proposed adversarial training to make task gradients statistically indistinguishable
208
210
212
to find the set of weights } (with Σx 4 = 1) such that Σ
214
216 as part of a bigger image-tailored tailored architecture. More recently, PCGrad [33] proposed to drop the
217
projection of one task gradient onto another if they are in conflict, whereas GradDrop [7] randomly
218
drops elements of the task gradients based on a sign-purity score.
In the literature, nization, are complementary to our work and thus could be used along with RotoGrad. Next, we provide a brief overview of them.
also find other approaches which, while orthogonal to the gradient homoge-
220
222
prominent approach for MTL is task clustering, that is, selecting which tasks should be learned together. This approach dates back to the original task-clustering algorithm [31], but new work in this direction keeps coming out [29, 35]. Alternative approaches, for example, scale the loss of each
223
224
task differently based on different criteria such as task uncertainty [14], task prioritization [11], or similar loss magnitudes [18]. Moreover, while most models fall into the hard-parameter sharing umbrella, there exists other architectures in the literature. Soft-parameter sharing architectures [27], for example, do not have shared parameters but instead impose some kind of shared restrictions to the entire set of parameters. An interesting approach consists in letting the model itself learn which parts of the architecture should be used for each of the tasks [12, 23, 30, 32j. Other architectures, such as MTAN [19]. make use of task-specific attention to select relevant features for each task. Finally, problems triggered by the differences between task gradients (in magnitude and direction) have also
226
228
230
232
238
been studied in other domains like meta-learning [34] and continual learning [21].
6 Experiments
234
235
In this section we assess the performance of RotoGrad on a wide range of datasets and MTL architectures. First, we check the effect of the learning rates of the rotation and network updates on the stability of the learning process of RotoGrad. Then, with the goal of applying RotoGrad
236
237
238 in scenarios with extremely large sizes of z, we explore the effect of rotating a subspace of instead of the whole shared representation. Finally, we compare our approach with competing MTL solutions in the literature, showing that RotoGrad consistently outperforms all other methods. Refer
240
to Appendix C for more detailed description of the experimental setups and additional results.

---

242
Relative task improvement. Since MTL uses different metrics for different tasks, throughout this section we group results by means the relative task improvement, first introduced in [22]. Given task k. and the metrics obtained during test time by our model, Mg, and by baseline model, Sk- which consists of K networks trained on each task individually, the relative task improvement for the
243
245
246
k-th task is defined as
4 == 100 (-1)² Ms - Se
(6)
S
247
where 44 if Ms < Sy means that our model performs better than the baseline in the k-th task, and la 0 otherwise. We depict our results using different statistics of Ag such as its mean (avga At), maximum (max; As), and median (med, As) across tasks.
249
6.1
Training stability
Sum of digits
5.0
251 At the end of Section 3.2 we discussed that, by casting problem 4 as Stackelberg game, we have convergence guarantees when the rotation optimizer is the slow-learner. Next, we empirically show this necessary condition.
253
MS
3.5
255
Experimental setup. Similar to [28], we use a multi-task version of MNIST [16] where each image is composed of a left and right digit, and use as backbone a reduced version of LeNet [17] with light-weight heads. Besides the left- and right-digit classification proposed in [28], we consider three other quantities to predict: i) sum of digits; ii) parity of the digit product: and iii) number of active pixels. The idea here is to enforce all digit-related tasks to cooperate (positive transfer), while the (orthogonal) image-related task should not disrupt these learning dynamics. We use negative cross-entropy and accuracy for the left- and right-digit tasks, binary cross-entropy and fl-score for the parity task, and mean squared error (MSE) as loss and metric for both regression tasks.
3.0
2.5
257
0
1e-4
5-e4 1e-3
5e-3 1e-2 5e-2 1e-1
RotoGrad's learning rate
259
Figure 3: Test error the sum of digits task for different values of RotoGrad's learning rate.
261
263
265
266
267
Results. Figure shows the effect averaged over ten independent runs runs-in in terms of test error in the sum task, while the rest of tasks are shown in Appendix C.2-of changing the rotations' learning rate. We can observe that, the bigger the learning rate is in comparison to that of the network's 's parameters lower the learning rate, reaching a sweet-spot at half the network's 's learning rate (5e-4). For smaller values,
269
(Ie-3), the higher and more noisy the test error becomes. MSE keeps decreasing
271
the rotations' learning is too slow and results start resemble those of the vanilla case, in which no rotations are applied (leftmost box in Figure 3).
273
274
6.2
Rotating a subspace
275
Next, we evaluate the effect of subspace rotations as described at the end of Section 3.4, assessing the trade-off between avoiding negative transfer and size of the subspace considered by RotoGrad.
278
277
Experimental setup. We test RotoGrad on a 10-task classification problem on CIFARIO [15], using binary cross-entropy and fl-score as loss and metric, respectively. for all tasks. We use ResNet18 [13] without pre-training as backbone (d = 512), and linear layers with sigmoids as task-specific heads.
279
Results are summarized at the bottom part of Table 1. We can observe that rotating the entire space provides the best results, and they worsen as we decrease the size of R&. However, rotating only 64 features (12.5 % of the shared feature space) still yields better results than vanilla optimization.
281
283
6.3
Methods comparison
We now proceed to compare RotoGrad with the different existing approaches to gradient conflict (for both magnitude and direction) in different rerl-world datasets, showing how RotoGrad outperforms existing methods while being on par with existing methods in training time.
285
287
Experimental setup. order to provide fair comparisons among methods, all experiments use identical configurations and random initializations. For all methods we performed a hyper-paramete search and chose the best ones based on validation error. Our results are reported using the median and standard deviation computed over 5-10 random seeds. Further details can be found in Appendix C.I.
289
7

---

Task
Table 1: Task performance on CIFARIO for different competing methods (top) and RotoGrad with matrices R& of different sizes (bottom). Table shows median and standard deviation over five runs.
Task
Task
Task
Task 6
Method
av8a A 1
med; A. 1
max; Ax 1
Vanilla
2.58 +: 0.54
2.73 1.37
11.14 + 3.35
GradDrop
3.07 + 0.48
3.18 1.07
14.03 + 2.83
Task
Task
PCGrad
2.86
3.33
12.01 + 3.19
MGDA
-
-.448
3.67 0.98
Vanili
RatoGrad
GradDrap
PCGrad
GradNorm
0.95
8.82 2.22
IMTL-G-R
IMTL-G
2.23
1.95
1
Task
IMTL-GG+ R
3.02
4.38
12.76 +: 1.77
Task
RotoGrad 64
2.90 + 0.49
+.44
Figure 4: Task improvement (median over five runs) of different methods on CI- FARIO. RotoGrad outperforms competing methods on all tasks.
RotoGrad 128
2.97 1.08
12.64 2.14
3.56
RotoGrad 256
3.68
+ 0.68
3.29
2.18
14.01 3.22
RotoGrad 512
4.48
0.99
4.72
2.84
15.57
+ 3.99
291
MNIST and SVHN. We reuse
Table 2: Test performance (median and standard deviation) on two set of unrelated tasks, across ten different runs.
the experimental setting from Section 6.1-now with multi-
293
task versions of MNIST [16] and in order to evalu- ate how disruptive the orthogonal image-related task is for differ- methods. We observe in the results from Table 2 that the effect of the image-related task
MNIST
SVHN
295
SVHN [26]-in
Digits
Act Pix MSE 4
Digits aVgR A. t
Act Pix MSE І
Method
avgg 4
296
297
Single Vanilla
0.01 + 0.01
0.17 + 0.06 2.75 + 3.17
298
-2.51 + 3.01
0.11 + 0.01
5.14 + 0.83
299
GradDrop PCGrad
-2.51 : 1.73
0.13 0.02
5.68 +: 1.05
1.91 4: 0.86
300
-..12
0.12 0.02
5.50 4.75
2.26 0.85
301
is more disruptive in MNIST,
MGDA
99.97
+.99 0.022
0.66
302
in which MGDA utterly fails. methods (Grad- Drop and PCGrad) do not im- prove the vanilla results, whereas IMTL-G, GradNorm, and RotoGrad obtain the best results.
GradNorm
0.13
0.01
6.67 1.02
1.41
0.74
IMTL-G
2.77
0.07 0.01
5.81 0.85
2.47
1.65
303
Direction-awee
RotoGrad
2.12
0.08 0.02
6.08 0.48
1.61 1 2.72
304
305
306
307
CIFARIO. We reuse the setting in Section 6.2 and compare the different MTL methods using five different seeds. Results are shown Table and Figure Unlike the previous setting, scaling gradients is not enough to solve the problem. Among existing methods, both direction-awart solutions (PCGrad and GradDrop) improve over the vanilla case all the statisties, whereas most magnitude-
308
309
310
311
aware solutions substantially worsen task performance. In stark contrast, RotoGrad improves task performance across all ten tasks, as it can be observed both in Table and Figure 4. To further show that this is a consequence of gradient homogenization in terms of both magnitudes and directions, we introduced an extra- baseline, IMTL-G+RR, which applies IMTL-G the extended MTL architecture
312
313
314
(Figure 2), that is, with matrix Rs optimizing the k-th task loss (instead of Equation 3).
NvUv2. Now, we test all methods using NYUv2 [9] three different tasks: 13-class semantic segmentation; depth estimation; and surface normals. To speed up training, all images were resized to 288 384 resolution; and data augmentation was applied alleviate overfitting. As MTL architecture, we use SegNet [2] where the decoder is splitted into three convolutational heads. We use observe in Table that RotoGrad results in a consistent improvement over all tasks with respect to the vanilla case. MGDA obtains the best results in surface normals the expense of overlooking the other tasks, while GradDrop worsens all results and PCGrad obtains minor improvements in all tasks. GradNorm finds a trade-off solution instead, improving results in depth estimation and surface normals, yet with worse results in semantic segmentation. RotoGrad obtains the best results followed by IMTL-G and, more importantly, RotoGrad the only method resulting in a average positive task improvement across the three over training three single-task models independently. It is worth mentioning that, with only
317
318
319
320
the same setup as Liu et al. [19]. Like in previous experiments,
321
322
323
324
326
tasks over
8

---

Table 3: Results for different methods the NYUv2 dataset with a SegNet model. RotoGrad obtains the best performance in segmentation and depth tasks on all metrics, while significantly improving the results on normal surfaces with respect to the vanilla case.
Semantic
Depth
Surface Normal
Segmenation f
Estimation &
Angle Distance Median
Within 1
Method
mloU Pix Ace avg, Ag 1 |hs Err Rel Err
avga Au 1|
Mean
11.25
22.5
30
avgg A, 1
Hours
Single
0.38
0.63
0.59
0.23
24.76
18.99
30.11
57.81
69.90
11.37
Vanilla
0.37
0.64
-0.02
0.56
0.22
3.68
30.09
26.09
19.74
43.62
57.07
-27.26
3.45
Grop
0.37
0.63
-1.55
0.59
0.21
2.22
30.81
27.19
17.68
41.44
55.15
31.67
3.55
PCGrad
0.39
0.64
1.50
0.54
0.22
4.99
29.85
25.81
19.41
44.02
57.64
-66.68
3.51
MGDA
0.20
0.51
-32.75
0.73
0.28
-22333
24.98
19.02
30.57
57.61
69.41
-0.11
3.55
GradNorm
0.36
0.64
-1.74
0.55
0.23
3.31
25.80
20.30
28.22
54.91
67.21
-5 -5.25
3.54
IMTL-G
0.38
0.65
1.92
0.55
0.23
3.64
26.83
21 21.96
25.14
61.74
64.76
-11.67
3.60
RotoGrad
0.40
0.66
5.33
0.54
0.20
9.06
26.35
21.27
26.25
53.11
65.99
-8.99
3.85
Table 4: Task fl fl-score statistics and training hours in CelebA for all competing meth- ods and two different architectures/settinss. RotoGrad obtains the best performance in both setups with comparable training time as existing methods.
Convolutional (d 512)
ResNet18 (d = 2048)
task fc-rcores (%) f
task fl-scores (%) f
Method
ming
med, avgk std; |
Hours
ming
med,
avgk std, & Hours
Vanilla
1.62
54.74
58.69
24.18
4.06
15.45
61.52
61.25
22.09
1.49
GradDrop
3.94
55.80
58.62
23.98
4.42
4.46
63.52
63.61
21.79
1.60
PCGrad
2.69
60.30
59.83
23.85
17.03
17.23
61.82
62.74
20.84
5.90
GradNorm
1.83
52.17
54.68
24.94
11.02
14.43
64.10
63.51
21.20
3.59
IMTL-G
3.31
53.05
56.05
26.92
4.90
21.32
62.12
61.98
21.62
1.72
RotoGrad
9.11
62.31
62.45
22.14
11.00
25.72
63.84
65.17
18.99
6.90
328
three tasks, all methods trained in less than hours; and that this result consolidates RotoGrad's scalability, as we only rotate the first 1024 dimensions of z, out of a total of 7 millions.
329
330
CelebA. Last, we apply all methods to a 40-class multi-classificttion problem in CelebA [20] on
331
two different settings: one using a convolutional network as backbone (d = 512); and another using ResNet18 [13] as backboone (d 2048). Similar to CIFAR1O, we use binary cross-entropy and fl-score as loss and metric for all tasks. Even though we face two completely different architectures,
332
333
334
results in Table 4 show that RotoGrad convincingly outperforms all competing methods in all fl fl-score statistics, independently of the model. Furthermore, since this is a computationally demanding task
335
396
with 40 tasks in fact, we omit MGDA as takes several days to train- we also compare methods in terms of training time. On the one hand, GradDrop and IMTL-G produce little overhead compared with the vanilla case, as expected. On the other hand, GradNorm and PCGrad take, respectively, 2.5 and 4 times longer to train than the vanilla setting. More importantly, RotoGrad outperforms existing methods while staying on par with them in training time, rotating 50 % and 75 % of the shared feature z for the convolutional and residual backbones, respectively, which further demonstrates that RotoGrad can scale-up to real-world settings.
337
338
340
342
7
Conclusions
344
In this work, we have introduced RotoGrad, an algorithm that tackles negative transfer in MTL by homogenizing task gradients in terms of both magnitudes and directions. RotoGrad enforces similar convergence rate for all tasks, while at the same time smoothly rotates the shared representation differently for each task in order to avoid conflicting gradients. As result, RotoGrad leads to stable and accurate MTL. Our empirical results have shown the effectiveness of RotoGrad in many scenarios, staying on top of all competing methods in performance, while being on par in terms of computational complexity with those that better scale to complex networks.
346
348
350
351
We believe our work opens up interesting venues for future work. For example, it would be interesting to study alternative approaches to further scale up RotoGrad using, for example, diagonal-bles or sparse rotation matrices; to rotate the feature space in application domains with structured features (e.g., channel-wise rotations in images); and to combine different methods, for example, by scaling gradients using the direction-awann awareness of IMTL-G and the "favor slow-learners" policy of RotoGrad.
352
354

---

References
356
357
[1]
Pierre-Antoine Absil, Robert F. Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008
[2]
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. "SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation." In: IEEE Trans. Puttern Anal. Mach. Infell. 39.12 (2017), pp. 2481-2495. DOT: 10. 10.119//TPPMII 2016. 2644615.
380
382
[3]
Rich Caruana. "Multitask Learning: A Knowledge- Based Source of Inductive Bias." In: Machine Learning, Proceedings of the Tenth International Conference, University of Mas- sachusetts, Amherst, MA, USA, June 27-29, 1993. Ed. by Paul E. Utgoff. Morgan Kaufmann.
363
364
385
1993. pp. 41-48. DOI: 10. 1016/0978- 1- 55860 307 -0073330 3. 50012-5.
[4]
Optimization on Manifolds." In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada. Ed. by Hanna M. Wallach et al. 2019, pp. 9154-9664
Mario Lezcano Casado. "Trivializations for Gradient-Basse
367
369
370
[5]
Mario Lezcano Casado and David Martinez-Rubio. "Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group." In: Proceedings of Machine Learning Research 97 (2019). Ed. by Kamalika Chaudhuri and Ruslan Salakhutdinov,
372
373
pp. 3794-3803.
374
[6]
Zhao Chen al. "GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks." " In: Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018. Ed. by Jennifer G. Dy and Andreas Krause. Vol. 80. Proceedings of Machine Learning Research.
375
376
377
378
PMLR, 2018, pp. 793-802.
379
[7]
Zhao Chen et al. "Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout." In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
381
382
Fd. by Hugo Larochelle et al. 2020.
383
[8]
Jorge Cortés. "Finite-time convergent gradient flows with applications to network consensus." In: Autom. 42.11 (2006). pp. 1993-2000. DOI: 10. 1016/j j. www.aatticc.aan0 2006. 06. 015.
384
385
99
Camille Couprie et al. "Indoor Semantic Segmentation using depth information." In: /st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2013. Tanner Fiez, Benjamin Chasnov, and Lillian J. Ratliff. "Convergence of Learning Dynamics in Stackelberg Games." " In: CoRR abs/ 1901.010611111 (2019).
386
[10]
389
380
Michelle Guo et al. "Dynamic Task Prioritization for Multitask Learning." In: Computer Vision EECV 2018- 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XVI. Ed. by Vittorio Ferrari et al. Vol. 1 11220. Lecture Notes in Computer Science Springer, 2018, pp. 282-299. DOI: 10. 1007000077777 Pengsheng Guo, Chen-Yu Yu Lee, and Daniel Ulbricht. "Learning to Branch for Multi-Task Learning." In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event. Vol. 119. Proceedings of Machine Learning Research. PMLR, 2020, pp. 3854-3863,
391
382
393
030 01270-0X_17
394
[12]
396
398
[13]
Kaiming He et al. "Deep Residual Learning for Image Recognition." " In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, 2016, pp. 770-778. DOI: 10. 1109/ 1119/CVPR. 2016.90.
399
400
401
[14]
Alex Kendall, Yarin Gal, and Roberto Cipolla. "Multi-tasy learning using uncertainty to weigh losses for scene geometry and semantics." In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. pp. 7482-7491.
402
403
404
[15]
Alex Krizhevsky, Geoffrey Hinton, et al. "Learning multiple layers of features from tiny images." In: (2009).
406
[16]
Yann LeCun, Corinna Cortes, and CJ Burges. "MNIST handwritten digit database." In: ATT Labs [Online] 2 (2010).
407
408
[17]
Yann LeCun et al. "Gradient-based learning applied to document recognition." " In: Proceedings of the IEEE 86.11 (1998), pp. 2278-2278 23244.
409
410
[18] Liyang Liu et al. "Towards Impartial Multi-task Learning." " In: International Conference on Learning Representations. 2021. 2021.
10

---

412
[19]
Shikun Liu, Edward Johns, and Andrew J. Davison. "End-To-End Multi-Task Learning With Attention." In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation/ / IEEE, 2019, pp. 1871- 1880. DOT: 10. 1119/CVPR... 2019. 00197.
413
414
415
416
[20]
Ziwei Liu et al. "Deep Learning Face Attributes in the Wild." In: 2015 IEEE International Conference Computer Vision ICCV 2015, Santiago. Chile. December 7-13, 2015. IEEE Computer Society, 2015, pp. 3730-3738. DOI: 10. 1109/ /CCV///CC..... 2015.001.4450
418
419
[21]
David Lopez-Paz and Marc' Aurelio Ranzato. "Gradient Episodic Memory for Continual Learning." In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. Ed. by Isabelle Guyon et al. 2017, Pp. 6467-6476 6476.
420
421
422
423
[22]
PokitsK Maninis, Lija Radosavovic, and Iasonas Kokkinos. "Attentive Single- Tasking of Multiple Tasks." In: JEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation IEEE, 2019,
425
426
Pp. 1851-1860. DOI: 10. 1109/CVPR, 2019. .00195.
427
[23]
Ishan Misra et al. Stithh Networks for Multi-task Learning." In: 2016 IEEE Conference
428
on Computer Vision and Pattern Recognition. CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, 2016, pp. 3994-4003. DO1: 10. 110.111/9./R..22. 2016. 433.
429
430
[24]
Ryan W. Murray, Brian Swenson, and Soummya Kar. "Revisiting Normalized Gradient Descent: Fast Evasion of Saddle Points." In: IEEE Trans. Autom. Control. 64.11 (2019),
432
pp. 4818-4824 DOI: 10. 11w9//11..1122 2019 2914998.
433
[25]
Yurii E. Nesterov. Introductory Lectures on Convex Optimization A Basic Course. Vol. 87. Applied Optimization. Springer, 2004. DOI: 10.
434
1-419- 8853-9.
435
[66]
Yuval Netzer et al. "Reading digits in natural images with unsupervised feature learning." In: NeurlPS Workshop on Deep Learning and Unsupervised Feature Learning (2011).
436
[27]
Sebastian Ruder. " "An Overview of Multi-Task Learning in Deep Neural Networks." In: CoRR Abs//706.05099 (2017).
438
439
[28]
Ozan Sener and Vladlen Koltun. "Multi-Tare Learning as Multi-Objective Optimization." In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada. Ed. by Samy Bengio et al. 2018. pp. 525-536.
410
442
443
[29]
Trevor Standley et al. "Which Tasks Should Be Learned Together in Multi-task Learning?" In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event. Vol. 1 119. Proceedings of Machine Learning Research. PMLR, 2020, pp. 9120-9132.
445
447
[30]
Ximeng Sun et al. "AdaShare: Learning What To Share For Efficient Deep Multi Task Learn- ing." In: Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Ed. by
448
449
450
Hugo Larochelle et al. 2020.
[31] Sebastian Thrun and Joseph O'Sullivan. "Discovering Structure in Multiple Learning Tasks: The TC Algorithm." " In: Machine Learning, Proceedings of the Thirteenth International Conference (ICML '96), Bari, Italy, July 3-6, 1996. Ed. by Lorenza Saitta. Morgan Kaufmann, 1996. pp. 489-497.
452
454
455
[32]
Simon Vandenhende et al. "Branched Multi-Task Networks: Deciding what layers to share." In: 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020. BMVA Press, 2020.
456
457
[33]
Tianhe Yu et al. "Gradient Surgery for Multi-Task Learning." " In: Advances in Neural Informa- tion Processing Systems. Ed. by H. Larochelle et al. Vol. 33. Curran Associates, Inc., 2020, pp. 5824-5836.
459
[34]
Tianhe Yu et al. "Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Re- inforcement Learning." In: 3rd Annual Conference on Robot Learning, CoRL. 2019, Osaka, Japan, October 30- November 1, 2019, Proceedings. Ed. by Leslie Pack Kaclbling. Danica Kragic, and Komei Sugiura. Vol. 100. Proceedings of Machine Learning Research. PMLR, 2019. pp. 1094-1100.
462
463
484
465
11

---

486
[35]
Amir Roshan Zamir et al. "Taskonomy: Disentangling Task Transfer Learning." In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018. IEEE Computer Society, 2018, pp. 3712-3722. DOI: 10. 1 109//VPPP 00391.
487
468
469
Checklist
470
471
1. For all authors...
472
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] RotoGrad homogenizes both magnitudes (Equation 1)
478
474
and direction (Equation 4), and empirical results in Section 6 and Appendix C demon- strate our claims.
475
478
(b) Did you describe the limitations of your work? [Yes] In Section 6.1.
477
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
478
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
479
2. If you are including theoretical results...
481
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Proposi- tion 3.1's assumptions are stated in Appendix A, and those regarding RotoGrad's stability appear in Appendix B.
482
483
484
(b) Did you include complete proofs of all theoretical results? [Yes] Proof of Proposi- tion 3.1 appears in Appendix A. For the proofs related to Stackelberg games, which are not a direct contribution of our paper, please refer to [10].
405
487
3. If you ran experiments...
488
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We provide instructions and code to reproduce our experiments the supplemental material.
490
481
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Appendix C.I.
482
493
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [Yes] We provide statistics for most of our experiments computed over 5-10 independent runs. Due time complexity required by larger datasets on NYUv2 and CelebA, we only report the results for single random seed, but still compare the different methods using several performance metrics.
494
495
486
497
498
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] All details are provided in Appendix C.I.
500
501
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
502
(a) If your work uses existing assets, did you cite the creators? [Yes]
503
(b) Did you mention the license of the assets? Yes] We only use code from previous research and licence MIT, which we inherit and acknowledge in our extended version of the code.
504
505
506
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We release the code implementation to reproduce our experiments together with the supplementary material, and will make it publicly available after the paper acceptance. (d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] We only use publicly available datasets with no personal information. Moreover, our experiments only report statistics on the results.
508
509
510
511
512
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We only use publicly available and broadly used image datasets.
513
514
515
5. If you used crowdsourcing or conducted research with human subjects...
516
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
12

---

518
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
519
520
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "jVN7n9Dbiiy", "reviewer": "Reviewer_Kz4R", "summary": "The work proposes RotoGrad, a multitask layer that rotates a shared feature space in a deep neural network to align gradients and mitigate negative transfer effects. The proposed method works within models with multiple prediction heads and a shared backbone, and deals both with magnitude and directional imbalance between task gradients. Authors provide multiple experiments in the computer vision space to show efficacy of the proposed method against other multitask baselines. \n", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Overall:** The paper presents an interesting new method to treat multitask learning systems. If robust, this method can have general use to the community and be a powerful addition to the toolbox. However, there are many unanswered questions and I remained overall unconvinced that the method works without major additional experimentation, including detailed ablation studies, and justifications (too many to address within the short rebuttal period in my opinion). I do think the method shows promise though, so regardless of the decision after this review process I hope the authors continue to work on their method.  \n\n**Pros:**\n\n(P1) Method is clear and intuitive, and tackles a crucial problem within optimal multitask training. \n\n(P2) Results seem promising and beat other baselines on benchmarks within the main paper. \n\n(P3) Formulation of the method as a Stackelberg game is interesting and can perhaps provide general insight into these problems, as many modern MTL methods involve optimizing auxiliary losses.\n\n(P4) Results are given for multiple model structures and multiple datasets performing the same tasks. \n\n(P5) Code is provided for reproducibility. \n\n**Needs Comment/Revision (major):**\n\n(RM1) Why must the proposed method interact with the forward pass? There are some other methods (see for example Sinha et al. at https://arxiv.org/abs/1806.08028) that propose methods which only interact with the backwards pass and seem to also work well. Either theoretical justification or additional experiments are needed to justify why doing the aggressive thing of altering the forward pass feature space is necessary. \n\n(RM2) The toy example provided is unconvincing, as the only comparison is to vanilla SGD. It is rather puzzling that even the Rotograd layer within the toy example is optimized using RAdam, which I feel would give Rotograd a severe advantage. \n\n(RM3) When the proposed method outperforms other methods that’s great, but when all the other methods perform very poorly it raises questions of whether the other methods are implemented correctly. For instance, why is MGDA performing so poorly on the CIFAR/MGDA tasks when the original MGDA paper used that as a benchmark and actually performed quite well? These discrepancies with the prior results need to be addressed for the current results to be fully believed.\n\n(RM4) One strange detail is that GradNorm and GradDrop seem to have different compute overheads within the current implementations, at least within the CelebA experiments. However, as far as I can tell, GradNorm and GradDrop have essentially the same compute overhead (with the limiting factor in the compute overhead the same computation of dL_i/dw for each task loss L_i), so this discrepancy makes me concerned that the present implementations of baselines is not correct. Similarly, RotoGrad seems to be even slower than PCGrad, and PCGrad has to calculate dL_i/dw for all trainable weights while RotoGrad only needs to backpropagate to the feature layer, so this overhead is somewhat concerning as well.\n\n(RM5) Ablation studies are needed to see whether both the magnitude and directional components of Rotograd are both needed. \n\n(RM6) The magnitude part of the implementation seems to include the initial gradient magnitude as part of the normalized quantity. However, this is incredibly dependent on the network initialization. Did authors account for this in some way? \n\n(RM7) The digit classification task performances seem to be very dependent on the size of the RotoGrad layer. Looking at the appendix, the network structure for these tasks is very shallow, so how can the authors rule out the effect of the added network capacity within the RotoGrad layer?\n\n**Needs Comment/Revision (less major):**\n\n(RLM1) Proposition 3.1 is correct but is rather obvious - would authors consider rephrasing into nonmathematical language to reduce potential confusion? If two gradients have positive pairwise similarity, then following their sum (for small enough learning rate) will clearly reduce both losses because loss reduction is step_size*(grad_k dot grad_sum). This proposition has also been heavily treated in prior work as well like MGDA and PCGrad, so does not necessarily need independent treatment here and can just be stated.\n\n(RLM2) Authors claim that homogenizing gradients at one layer is equivalent to homogenizing gradients at the weight layers. This is a false statement, as chained gradient computations will cause even initially homogenized gradients at a feature layer to increasingly stray from each other. Authors must justify why they can ignore this effect.\n\n(RLM3) Figures need to be placed closer to the text where they’re referenced. \n\n(RLM4) Sometimes bolded numbers in tables are not the best results for that column. \n\n*Post rebuttal*: I thank the authors for their time and effort in crafting their response. A lot of my more technical comments were addressed. I am still very concerned about the extra capacity and don't feel these concerns are properly addressed by pointing out that one other baseline (IMTL-G) also has added capacity, and some of the new results actually may run counter to the proposed narrative - for example RotoGradNorm beating out RotoGrad. The method itself is well defined but I am still not sold on the scientific validity of the analysis. I will keep my rating a 4. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "More discussion could be provided regarding limitations and societal impact. In terms of the former, for example, there are plenty of benchmarks on which RotoGrad seems to not perform the best, so discussion of why needs to be more in detail. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "c6sIjq7QvEC", "reviewer": "Reviewer_3UmL", "summary": "This paper proposes RotoGrad, a multi-task optimisation strategy that normalises the gradient magnitude and (conflicting) direction in a unified manner, to avoid the negative transfer. The proposed method is intuitive, theoretically motivated by the two player-game, and hyper-parameter free (except for learning rate required for updating rotation parameters). \n\nThe paper extensively evaluates the proposed method comparing with other baselines, both in single domain MTL setting NYUv2, and in multi-domain MTL setting CIFAR-10, and CelebA, and have showed improved performances in all settings.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written, and well-motivated from the existing Stackelberg leadership model. I also appreciate the authors discuss the memory and time complexity in Section 3.4, and being honest about the current limitation on scaling issues in computer vision tasks, which typically with large representation dimensions, requiring extensive computing time for an exact solution without any approximation.\n\nGenerally, I like the idea and particularly for the part that introducing the gradient rotation, which I believe is novel and overlooked in previous multi-task optimization papers I know. Here are some other comments, which I hope the authors could clarify and hopefully could further improve the paper.\n\n-- Subspace rotation. As mentioned in the previous comment, for computing a direct solution requires a $d^2$ rotation matrix for each task with representation dimension $d$. The introduced rotation parameters can be neglectable if $d$ is small. However, if $d$ is large as typically in most computer vision tasks, it introduces more parameters compared to baseline methods, which leads to unfair comparison.  We thus would not know whether the improved performance is based on the design of this method, or simply because of the increased number of learnable parameters. In Table 1, applying the rotation only in the 64 subspace is having a close and worse performance compared to the baseline performance, this verifies my concern here. To be clear on the parameter size and/or FLOPs in each method would be essential for a fair comparison. Maybe adding a linear embedding to map to a smaller embedding size, or applying a block-wise rotation would alleviate this issue?\n\n-- Limitation on multi-task network design. From my understanding, the proposed RotoGrad requires a clear top-down multi-task network design with a **single** shared representation output $z$, which the task-specific rotation function needs to be applied on. It would be tricky, or impossible? to apply this strategy to any design of the network, say MTAN as discussed in the related work, or other popular multi-task network design: such as AdaShare, Cross-Stitch network, from which the shared network parameters are defined implicitly. In this way, the RotoGrad would be very hard to optimise, maybe requiring network-specific modification on choosing the appropriate $z$. This is not a serious issue though, if the improved performance solely on hard parameter sharing is significant enough that surpass the improvements brought by the design of the network itself. A paragraph discussing this issue would be helpful.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper studies the theoretical insights and optimisation strategy for multi-task learning, and thus having no societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "KxpuA2RwVcl", "reviewer": "Reviewer_wb8x", "summary": "The paper proposes a method to reduce the optimization difficulty for multi-task learning. The key idea is to homogenize task gradients in terms of both magnitudes and directions. Specifically, they rescale task gradients to maintain similar convergence rates over tasks. At the same time, they introduce new network parameter R_k for each task k and train it so that the angle between task-specific and average gradients are minimized when task-head parameters are rotated using R_k. The method improves baselines and outperforms existing methods on DIGITS, CIFAR10, NYUv2, and CelebA datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the key contribution of this paper is to introduce rotation matrices that reduces gradient conflicts by rotating the feature-space z in a way that matches their directions. In particular, the idea to use rotation matrices is reasonable and good in that they will preserve the relational information (e.g. distance) between features.\n\nThey claim that the novelty lies on the joint homogenization of gradients in terms of both magnitude and direction. However, IMTL [18] also jointly addressed loss magnitudes and gradient direction. Rather, I think novelty lies on the way they formulate the problem as a Stackelberg game. In other words, they optimize rotation matrices only to match gradient directions while optimizing the remaining network parameters to minimize the task loss. GradNorm used a similar strategy but they focused on adjusting gradient magnitudes. \n\nIn the experiments, their method improves baselines and performs better than other methods on four datasets and for two different backbone architectures. Additional analysis on training stability, effect of sub-dimension size for rotation, and training time comparison provides more comprehensive views.\nHowever, it seems one needs to carefully adjust the training parameters for the rotation matrix, depending on dataset. when I ran the submitted code for mnist, the median cosine similarity for the task 'density' oscillated largely.\n\n\nI think L130-131 is a bit misleading, because rotating the feature z is different from rotating its gradient g. Rotating z could result in the change of magnitudes in gradients. For example, consider a function f(x,y) = [relu(x),0]. A vector [x,y]=[1,0] and its rotation [x,y] = [-1,0] have different gradient magnitude.\n\nIt would be good to add ablation study to analyze the effect of magnitude and direction separately, i.e., magnitude-only and direction-only. Since each of them was explored in existing work, comparing each to corresponding other approaches will be also informative, e.g., magnitude-only to GradNorm and direction-only to PCGrad and GradDrop.\n\nOne way to demonstrate the training behavior will be to visualize the actual reduced gradient conflicts, possibly using angles, compared to the baseline. Also, comparing the method with baselines in the task loss curve will show that the method results in lower task loss.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "9u4Twtqo7Xx", "reviewer": "Reviewer_ab65", "summary": "The paper proposes a novel method for the multitask learning environment, which deals with the conflicting gradients problem (AKA negative transfer). The method comprises two parts for dealing with gradient magnitudes and directions. The latter part is quite novel, well-motivated (with a cartoon example from Fig. 1), and theoretically sound. The method is tested against a number of MTL benchmarks and is shown to produce better-performing models. Overall, the paper presents interest to the MTL community.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "L27: why is gradient cancellation seen as a problem? If a model cannot explain training data, conflicting gradients can be seen as reducing the variance of the solution, yielding a model which is \"equally\" bad wrt multiple objectives (as shown in Fig.1). In such a case, the proposed solution can be seen as an efficient way to increase model capacity in the right places (where the heads branch out), thus leading to a model with higher capacity (which is not bad per se but requires acknowledgment).\n\nL35: can prioritization of the \"lagging\" tasks be harmful, e.g., in cases where the task loss is noisy (e.g., due to extreme label noise)?\n\nThe usage of a task-specific rotation matrix is interesting; however, could it be replaced with a simpler learned construct, such as a task-conditional batch norm, keeping the alternating training scheme from Alg.1 intact?\n\nThe usage of matrix exponential for parameterizing the rotation matrix is interesting; however, in the light of the argument in L151, do the gradients of the unconstrained parameters of R require any treatment (gradient magnitude/LR adjustment)?\n\nOne unclear aspect associated with matrices R is whether the preceding activations are the output of a linear layer or a nonlinearity. If activations are nonlinear and given that such a matrix R constitutes an additional linear layer, it would be interesting to see extended single-task baselines with matrix R added in the right place (where it would otherwise branch out from the backbone in MTL).\n\nAn ablation study of the magnitude and direction counterparts, particularly where another more traditional method can be applied in place of one of the proposed magnitude components, is missing. The closest I could find in Table 1 reporting IMTL-G+$R_k$\n\n================= Post-rebuttal comment\n\nAfter reading all reviews and responses, and discussing the common concerns with the other reviewers, one weak point of this work remains the increased capacity of the network (regardless of how the added parameters are optimized). For this reason, I chose to keep my original rating of WA.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work introduces an approach to minimize interference in multitask learning RotoGrad. 

After carefully looking over the reviews and rebuttal, I do find the proposed solution interesting and the experimentation section comprehensive (especially with the additional results provided by the authors during the discussion period). 
I do not feel that the concerns of extra parameters problematic in this scenario. 

In general if there is a weakness of this work that I would highlight is that I feel more analysis could have been done to understand the behavior of the rotation matrices and the scaling of the updates done. For example, would the rotation matrices converge before the multitask problem converge to some fix task-specific rotation ? I would have been interested in any form of visualization that could have helped to understand what these matrices do, how they interfere with the baseline model, how they change the learned representation etc. The same for the scaling of the updates. I feel the focus is mostly on looking at the improved performance (which to be fair is the standard way of approaching this) but not as much on understanding how the proposed method works and what impact it has. 

That said, the results are reasonable, and I do think this method will be of interest to the community. 
After a long deliberation, I feel the paper in its current form is not exactly ready for NeurIPS (though is really a very borderline case). I think the additional results the authors provided greatly increase the paper. I would have liked a bit more analysis (or ablation) in understanding the interplay between the added capacity. What the rotation matrices really do. Anything that gives additional understanding of the dynamics of the method would have probably pushed the paper over. I strongly encourage the authors to improve the manuscript and resubmit/
Decision: Reject