Gradient Surgery for Multi-Task Learning
Authors: Tianhe Yu
Saurabh Kumar
Abhishek Gupta
Karol Hausman
Sergey Levine
Chelsea Finn
Venue:  (2020)
Keywords: multi-task learning
deep learning

Abstract: While deep learning and deep reinforcement learning systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge, particularly as these algorithms learn individual tasks from scratch. Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efficient learning. However, the multi-task setting presents a number of optimization challenges, making it difficult to realize large efficiency gains compared to learning tasks independently. The reasons why multi-task learning is so challenging compared to single task learning are not fully understood. Motivated by the insight that gradient interference causes optimization challenges, we develop a simple and general approach for avoiding interference between gradients from different tasks, by altering the gradients through a technique we refer to as “gradient surgery”. We propose a form of gradient surgery that projects the gradient of a task onto the normal plane of the gradient of any other task that has a conflicting gradient. On a series of challenging multi-task supervised and multi-task reinforcement learning problems, we find that this approach leads to substantial gains in efficiency and performance.  Further, it can be effectively combined with previously-proposed multi-task architectures for enhanced performance in a model-agnostic way.

Under review as a conference paper at ICLR 2020
GRADIENT SURGERY FOR MULTI-TASK LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
While deep learning and deep reinforcement learning systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge, particularly as these algorithms learn individual tasks from scratch. Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efficient learning. However, the multi-task setting presents a number of optimiza- tion challenges, making it difficult to realize large efficiency gains compared to learning tasks independently. The reasons why multi-task learning is so challenging compared to single task learning are not fully understood. Motivated by the insight that gradient interference causes optimization challenges. we develop simple and general approach for avoiding interference between gradients from different tasks, by altering the gradients through a technique we refer to as "gradient surgery". We propose a form of gradient surgery that projects the gradient of a task onto the normal plane of the gradient of any other task that has conflicting gradient. On a series of challenging multi-task supervised and multi-task reinforcement learning problems, we find that this approach leads to substantial gains in efficiency and performance. Further, it can be effectively combined with hyrooooosy multi-task architectures for enhanced performance in a model-agnostic way.
INTRODUCTION
While deep learning and deep reinforcement learning (RL) have shown considerable promise in enabling systems to perform complex tasks, the data requirements of current methods make it difficult learn breadth of capabilities particularly when all tasks are learned individually from scratch, A natural approach to such multi-task learning problems is to train single network on all tasks jointly, with the aim of discovering shared structure across the tasks in a way that achieves greater efficiency and performance than solving the tasks individually. However, learning multiple tasks all at once results in difficult optimization problem, sometimes leading to worse overall performance and data efficiency compared to learning tasks individually (Parisotto et al. 2015 Rusu et al.) 2016a). These optimization challenges are so prevalent that multiple multi-task RL algorithms have considered using independent training subroutine of the algorithm before distilling the independent models into a multi-tasking model (Levine et al.) 2016 Parisotto et al. 20155] Rusu et al., [20 Ghosh et al., 2017, Teh et al.) 2017), producing multi-task model but losing out the efficiency gains over independent training. If we could tackle the optimization challenges of multi-task learning effectively, we may be able to actually realize the hypothesized benefits of multi-task learning without the cost in final performance.
While there has been a significant amount of research in multi-task learning (Caruana 1997: Ruder 2017), the optimization challenges are not well understood. Prior work has described varying learning speeds of different tasks (Chen et al. 2017) and plateaus in the optimization landscape (Schaul et al. 2019) as potential causes, while a range of other works have focused on the model architecture (Misra et al. 2016b; Liu et al., 2018). In this work, we instead hypothesize that the central optimization issue in multi-task learning arises from gradients from different tasks conflicting with one another. In particular, we define two gradients to be conflicting if they point away from one another (i.c., have a negative cosine similarity). As concrete example, consider the 2D optimization landscapes of two task objectives shown in Figure T The optimization landscape of each task consists of a deep valley, as has been characterized of neural network optimization landscapes in the past (Goodfellow et al.) 2014). When considering the combined optimization landscape for multiple tasks, SGD produces gradients that struggle to efficiently find the optimum. This occurs due to gradient thrashing

---

Under review as a conference paper at ICLR 2020
Multi-Tast Objective
Task Objective
Task 2 Objective
Adam
Adam PCGrad
(N)
(b)
cc)
(d)
Figure I: Visualization of PCGrad's effect a 2D multi-task optimization problem. A multi-task objective individual task objectives that comprise the multi-tase objective. (d) Trajectory of gradient updates on the multi-task objective using the Adam optimizer. The gradient vectors of the two tasks at the end of the trajectory are indicated by blue and red arrows, where the relative lengths are log (e)le.(e) Trajectory of gradient updates the multi-task objective using Adam with PCGrad. For (d) and (e), the optimization trajectory goes from black yellow.
landscape. (b) & (c) Contour plots
phenomenon, where the gradient of one task destabilizes optimization in the valley. We observe this in Figure dd) when the optimization reaches the deep valley task I, but is prevented from traversing the valley to optimum. In Section 6.2 find experimentally that this thrashing phenomenon also occurs in a neural network multi task learning problem.
The contribution of this work is method for mitigating gradient interference by altering the gradients directly, i.e. by performing "gradient surgery". If two gradients are conflicting, we alter the gradients by projecting each onto the normal plane of the other, preventing the interfering components the gradient from being applied to the network. We refer to this particular form of gradient surgery as projecting conflicting gradients (PCGrad), PCGrad is model-agnostic. requiring only a single modification to the application of gradients. Hence, it is easy to apply to a range of supervised learning and multi-tase reinforcement learning, and
problem settings, including multi-tash, supeerei can also be readily combined with other multi-task learning approaches, such as those that modify architecture. We evaluate PCGrad on multi-task CIFAR classification, multi-objective scene understanding, challenging multi-task RI domain, and goal-conditioned RI. Across the board, we find PCGrad leads to significant improvements in terms of data efficiency, optimization speed, and final performance compared prior approaches. Further, multi-task supervised learning tasks, PCGrad can be successfully combined with prior state-of-the-ae methods for multi-ta tasti learning for greater performance.
2 PRELIMINARIES
The goal of multi-task learning is to find parameters 0 of a model fo that achieve high average performance across the training tasks drawn from a distribution of tasks p(T). More formally, we aim to solve the problem: min (T) [&s (Jo)), where L, is a loss function for the i-th task T. that
want minimize. To obtain model that solves specific task from the task distribution p( (T), we define a task-conditioned model fo (yl jr, 2;), with input x, output y, and encoding 2; for task T, which could be provided one-hot vector in any other form.
3 MULTI-TASK LEARNING VIA GRADIENT SURGERY
While the multi-task problem can in principle be solved by simply applying standard single-task algorithm with a suitable task identifier provided to the model or a simple multi-head or multi-output model, a number of prior works (Parisotto et al.) 2015 Rusu et al., [016aa] [Sener & Koltun| 2018) have found this learning problem to be difficult, especially in the reinforcement learning setting. We hypothesize that one of the main challenges of multi-tass task learning and thrashing gradients, and find that this can significantly impede learning progress, especially when combined with iterative data collection. We identify possible causes for this problem and propose simple and general approach to mitigate it.
be characterized by conflicting
3.1 THRASHING GRADIENTS IN MULTI- MUSKI-TASK OPTIMIZATION LANDSCAPES
We hypothesize that key optimization issue in multi-tas talti-task learning arises when gradients from multiple tasks are conflict with one amother, i.c. when gradients point away from one another. More specifically, hypothesize that such conflict may lead to gradient thrashing. Concretely, gradient
2

---

Under review as a conference paper at ICLR 2020
gi
projag:
gi
gi
g)
8)
conflicting
Ho1 ROn- conflicting
(a)
(b)
(c)
Figure 2: Visual depiction of conflicting gradients and PCGrad. In (a), we see that tasks A and B have conflicting gradient directions, which can lead to destructive interference and unstable learning. In (b), we illustrate the PCGrad algorithm in cases where gradients conflicting. PCGrad projects the gradient of task onto the normal vector task B's gradient. In (c), we show that tasks with non con-confficting gradients are not altered under PCGrad, thereby keeping tasks with constructive interference.
thrashing refers to the phenomenon where a large gradient for one task changes the parameter vectors in way that substantially decreases performance another task. Since worse performance typically leads to larger gradients, this results in alternating gradient directions, where, at the next iteration, the second task will have large gradients that dominate and reduce performance on the former task. This issue can be particularly pronounced for neural network optimization, since neural network loss landscapes are known resemble long narrow valleys (Goodfellow et al.) 2014), where the gradient perpendicular to the direction of the valley will be small.
We aim to study this hypothesis through two toy examples. First, consider the two-dimensional optimization landscape illustrated in Fig. Th. where the landscape for each task objective corresponds to deep and curved valley (Fig. IT and Te). The optima of this multi-task objective correspond to where the two valleys meet. More details on the optimization landscape are in Appendix B We observe that the gradient thrashing hypothesis consistent with what we observe when running Adam (Kingma & Bal 2014) on this landscape in Fig. D1. where observe that Adam does not traverse one valley towards the other, preventing it from reaching an optimum.
We also aim to detect if similar phenomenon occurs in multi-task learning with neural network with thousands of parameters on a toy regression problem. To measure the extent of gradient thrashing, we plot the cosine similarity between the gradients of two tasks throughout the beginning of learning in Fig. (left.. We indeed observe significant level of gradient thrashing at every iteration, where the cosine similarity varies between -0.75 and 0.75 very high frequency.
Motivated by these observations, we develop an algorithm that aims to alleviate the optimization challenges caused by gradient thrashing by preventing such gradient conflict between tasks.
3.2 PCGRAD: PROJECTING CONFLICTING GRADIENTS
We aim to prevent gradient thrashing by directly altering the gradients themselves, i.e. through "gradient surgery."
maximally effective and maximally applicable, we must perform surgery in way that still allows for positive interactions between the task gradients and does not introduce any assumptions on the form of the model.
We start by first detecting whether two gradients are in conflict, by measuring whether they point away from one another. More concretely, we characterize two tasks as conflicting for the current parameter setting if they yield negative cosine similarity between their respective gradients. The goal of PCGrad is to modify the gradients for each task so as to minimize negative conflict with other task gradients, which will in turn mitigate gradient thrashing.
To deconfiict gradients during optimization, PCGrad adopts a simple procedure: if the gradients between two tasks are in conflict, i.e. their cosine similarity is negative, we project the gradient from one task onto the normal plane of the gradient of the other task. This amounts to removing the conflicting component of the gradient for the task, thereby reducing the amount of destructive gradient interference between tasks. A pictorial description of this idea is shown in Fig. 2 Suppose the gradient for task T, is g,, and the gradient for task T, is g,. PCGrad proceeds as follows: (1) First, it determines whether g; conflicts with g; by computing the cosine similarity between vectors gi and g,, where negative values indicate conflicting gradients. (2) the cosine similarity negative, gig,
we replace g: by its projection onto the normal plane of g;: gi = g: IR, || gj. If the gradients are not in conflict, i.c. cosine similarity is non negativee the original gradient g: remains unaltered. (3) PCGrad repeats this process across all of the other tasks sampled in random order from the current.
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 PCGrad Update Rule
Require: Current model parameters
1: Sample mini-batch of tasks B = (Tx} p({)
2: for T, ~ B in sequence do
3: Compute gradient gi
as g: (ff((ff)
4: for T unifzenly B in random order do
5: Compute gradient g; of task Tj as gj (oo)(fo)
6:
Compute cosine similarity between g, as g, as co05) (0:5)=
8. 85
le Ilyy
7:
if cos( < 0 then W
8:
g: g. 1 ||,
Subtract the projection of ge onlo gj
9: end
10: end for
11: Store g. =
12:
end for
13:
return update Ao = =; g.
batch T, V j * i, resulting in the gradient nro that is applied for task T, We perform the same procedure for all tasks in the batch to obtain their respective gradients. The full update procedure is described in Algorithm [ and a discussion on using a random task order is included in Appendix D
This procedure, while simple to implement, ensures that the gradients that we apply for each task per batch interfere minimally with the other tasks in the batch, mitigating the thrashing gradient problem, producing variant on standard first-order gradient descent in the multi-objective setting, In practice, the PCGrad gradient surgery method can be combined with any gradient-based optimizer, including commonly used methods such as SGD with momentum and Adam (Kingma & Bal 2014). by simply passing the computed update to the respective optimizer instead of the original gradient. Our experimental results verify the hypothesis that this procedure reduces the problem of thrashing gradients, and find that, as result, learning progress is substantially improved.
Finally, we analyze the convergence of this procedure Theorem the two-task setting, to ensure that the procedure is sensible under the standard assumptions in optimization.
Theorem 1. Consider two task loss functions L1 1 + R and Lg RR - R which are convex and differentiable. For all 9 E IR"", ler L(0) (0) L0))), iee. L is a multi-task objective. Let o be the angle between V L1 (0) and V (0). Suppose L is differentiable and that its gradient is Lipschitz continuous with constant L L> 0, i.e. we have [[V](()) VL(02)ll2 S L||0, 02112. any 0,, Oz. Then, the PCGrud update rule with step size S will converge to either (1) location in the optimization landscape where cos(o) -1 or (2) the optimal value L(
Proof. See Appendix A
Theorem 1 states that application of the PCGrad update in the two-task setting with a convex and Lipschitz multi- task loss function L leads to convergence to either the minimizer L or potentially sub-optimal objective value. A sub-optimal solution occurs when the cosine similarity between the gradients of the two tasks is -1, i.e. the gradients directly conflict, leading to zero gradient after applying PCGrad. However, in practice, since we are using SGD, which is noisy estimate of the true batch gradients, the cosine similarity between the gradients of two tasks in a minibatch is unlikely to be 1, thus avoiding this scenario.
THE PRACTICAL OPERATIONS OF PCGRAD
We apply PCGrad to both supervised learning and reinforcement learning problem settings with multiple tasks or goals. In this section, we discuss the practical instantiations of PCGrad in those settings. Further implementation details are included in Section
4.1 MULTI-TASE SUPERVISED LEARNING
In multi-task supervised learning, each task To p(T) has corresponding training dataset D; consisting of N; labeled training examples, i.e. D; {xx, ) The objective for each task
4

---

Under review as a conference paper at ICLR 2020
in this supervised setting is then defined L.(fe) B( 1- log fo(y/2, 21)], where 2; is one-hot encoding of task T,
At each training step, we randomly sample batch of data points B from the whole dataset U, Dj and then group the sampled data with the same task encoding into small batches denoted as B, for each T; represented in B. We denote the set of tasks appearing in B as By. After sampling, precompute the gradient of each task in By as
Veli(fe) (a Vg log fo(y)2,2))
Given the set of precomputed gradients (oL)(fo), we also precompute the cosine similarity between all pairs of the gradients in the set. Using the pre-computed gradients and their similarities, we can obtain the PCGrad update by following Algorithm without re-computing task gradients nor backpropagating into the network.
Since the PCGrad procedure is only modifying the gradients of shared parameters in the optimization step, it is model-agnostic and can be readily applied to any architecture designed for supervised multi-task learning. In Section 6 we combine PCGrad with two state-of-the-ar architectures for multi-task learning, which leads to noticeable improvement over their original performance.
4.2 MULTI-TASK AND GOAL-CONDITIONEE REINFORCEMENT LEARNING
For multi-task reinforcement learning, PCGrad can be readily applied to policy gradient methods by directly updating the computed policy gradient of each task, following Algorithm [ analogous the supervised learning setting. For actor-critic algorithms, it is also straightforward to apply PCGrad: simply replace the task gradients for both the actor and the critic by their gradients computed via PCGrad. Hence, PCGrad can be readily incorporated into a variety of model-free RL algorithms. When applying PCGrad to goal-conditioned RI., we represent p(T) as distribution of goals and let 24 be the encoding of goal. Similar to the multi-task supervised learning setting discussed above, PCGrad may be combined with various architectures designed for multi-task and goal-conditioned
RL (Fernando et al.) 2017 Devin et al. 2016). where PCGrad operates on the gradients of shared parameters, leaving task speci-peiific parameters untouched.
In our experiments, we apply PCGrad to the soft actor-critic (SAC) algorithm (Haarnoja et al. 2018). a recently proposed off-policy actor-critic algorithm that has shown significant gains sample efficiency and asymptotic performance across many different domains. In SAC, we employ Q- learning style gradient to compute the gradient of the Q-function network, Qo a, 2,), often known gradient to compute the gradient of the policy network To(a|s, 2:), often known as the actor. For sampling, we instantiate a set of replay buffers (Dihr,sp(T) Training and data collection are alternated throughout training. During a data collection step, the policy Hy on all the tasks T ~ p(T) to collect an equal number paths for each task and store the paths of each task T; into the corresponding replay buffer D, At each training step, we sample an equal amount of data from each replay buffer D; to form a stratified batch. For each task Te ~ p(T). the parameters of the critic are optimized to minimize the soft Bellman residual:
as the critic, and reparametteiizaiionssyy
Q (Ø) = E(o,... (Q a,, 2;) (r(se, O2 2;) + 7V (
(2)
(3(++11, 24) = Eac [Q (141 2.) a log no ( 18411, x]]]
(3)
where y is the discount factor, & are the delayed parameters, and a is learnable temperature that automatically adjusts the weight of the entropy term. For each task Te ~ p(T), the parameters of the policy ne are trained to minimize the following objective
e (0) = Ea,ND. [ [Barno((a))aaaa) [a log Te(aelst, 2.) - Qs az x1]l
(4)
We compute V (0) and V Algorithm
J (0) (0) for all To ~ p(T) and apply PCGrad to both following
In the context of SAC specifically, further study how the temperature should be adjusted. If we use a single learnable temperature for adjusting entropy of the multi-task policy no s, 2;), SAC may stop exploring once all easier tasks are solved, leading poor performance tasks that
5

---

Under review as a conference paper at ICLR 2020
harder or require more exploration. To address this issue, propose learn the temperature per-task basis, i.c. using a parametrized model to represent ag (2,) (which we abbreviate as PA for per-task alpha). This allows the method to control the entropy of (als, 24) per-task. We optimize the parameters of oxp (z;) using the same constrained optimization framework as in Haarnoja et al. (2018).
5 RELATED WORK
Algorithms for multi-task learning typically consider how to train single model that can solve a
variety of different tasks (Caruana 1997: Bakker & Heskes] 2003; Ruder (017). 'he multi-task
formulation has been applied to many different settings, including supervised learning (Zhang ct al.
2014 Long & Wang 2015 Yang & Hospedales] 2016 [eener& & Koltun| 2018: Zamir et al. 2018)
(Espeholt et al. 2018 Wilson et al.; 2007). p. as well as many different domains, such vision (Bilen & Vedaldi 2016 Misra al.) 2016a Kok
and cewtoorcemen--earniinn
2017; et al.
2018
Zamir et al.] 2018), language (Collobert & Weston 2008
Dong et al. 2015 McCann et al. 2018
Hausman Radford et al.] 2019) and roboties (Riedmiller al. 2018 Wulfmeier al., Hausmann et al.
2018). While multi-task learning has the promise of accelerating acquisition of large task repertoires, in practice presents a challenging optimization problem, which has been tackled in several ways in
prior work.
A number of architectural solutions have been proposed to the multi-task learning problem based on multiple modules or paths (Fernando et al. 2017 Devin et al. 2016 Misra et al. 2016bb Rusu jtt al. [0166] [Rosenbaum 2018| Vandenhende et al. 2019 | attention-based architectures (Liu et al. 2018: Maninis et al.) 2019). Our work is agnostic to the model architecture and can be combined with prior architectural approaches a complementary fashion. A different set of multi-task learning approaches aim to decompose the problem into multiple local problems, often corresponding to each task, that are significantly easier to learn, akin to divide and conquer algorithms (Levine al. [016 Rusu et al. 2016a Parisotto et al. 2015 Tch et al. 2017] Ghosh et al. 2017, Czarnecki et al. 2019). Eventually, the local models are combined into a single, multi-task policy using different distillation techniques (outlined in (Hinton al. 2015 Czarnecki cet al. 2019)). In contrast to these methods, we propose a simple and cogent scheme for multi-task learning that allows us learn the tasks simultaneously using a single, shared model without the need for network distillation.
Rosenbaum et al. 2018). 8). or using
Similarly to our work, a number of prior approaches have observed the difficulty of optimization in the multi-task learning setting (Hessel et al. 2219 [ Chen et al. [2018] Kendall et al., [201 Schaul al. 2019). Our work, in contrast to many of these optimization schemes, suggests that the challenge in multi-task learning may be attributed to the problem of gradient thrashing, which we address directly by introducing a simple and practical algorithm that de-confiicts gradients from different tasks. Prior work (Sener & Koltun 2018) alternatively proposes gradient-based multi-objective optimization problem for multi-task learning to address the problem of optimizing possibly conflicting objectives. As noted in Alg 2 in (Sener& & Koltun 2018), it learns constant scaling factor for per-task gradient to avoid conflicting, while our method corrects both the scaling factor and the direction of per-task gradient, which can more effectively deconflict gradients. Prior work has also used the cosine similarity between gradients to define when an auxiliary task might be useful for single-task learning (Du et al., 22018). We similarly use cosine similarity between gradients to determine if the gradients between a pair of tasks are in conflict. Unlike Du et al. (2018), we use this measure of gradient conflict as part of gradient surgery in the context of multi-task learning applications.
A number of works in continual learning have studied how to make gradient updates that do not adversely affect other tasks by projecting the gradients into a space that do not conflict with pre- vious tasks (Lopez-Paz & Ranzato 2017: CChaudhry et al., 2018). Those methods focus on the continual learning setting, and either need to solve for the gradient projections using quadratic programming (Lopez-Paz & Ranzato (017). or only projecting the gradient onto the normal plane of the average of the gradients of past tasks (Chaudhry et al. 2018). In contrast, our work focuses on multi-task learning, does not require solving any QP, and iteratively projects the gradients of each task onto the normal plane of the gradients of each of the other tasks instead of averaging. Finally, our method is distinct from and solves a different problem than the projected gradient method (Calamai & Moré, 1987). which is an approach for constrained optimization that projects gradients onto the constraint manifold.

---

Under review as a conference paper at ICLR 2020
sweep
3. basketball
4, into hole
5. off faucet
push
pull lever dial with
faucet
stick
46. hand insert
10.
11. pull handle side
12. assemble
13. pull with stick
14. pick out ss clsassemble of hole
16. place ento shelf
17. push mug
18. press handle side
coffee
nut
47. close box
20. plate
21. slide
22. press button wall
23. press handle
24, pull handle
25, soccer
26. retrieve 27. retrieve
48. lock door
hammer
plate side
plate side
plate
28. close drawer
29. 30. reach button
31 press button top wel
32. reach
33. insert peg side
34, push
35. push with wall
36, pick & wi wal
49. unlock door
with wall
37. press button
38. pick & place
39. pull
440. unplug peg
41. close window
42. open window
43. open door
44. close door
45. open drawer
50. pick bin
mug
Figure 3: We show visualization of 50 tasks used in MT50 from Meta-Worke Yu et al. 2019), which we use for multi-task RL experiments. MT10 is subset of the total 50 tasks, which includes reach, push, pick & place, open drawer, close drawer, open door, press button top, open window, close window, and insert peg inside.
EXPERIMENTS
The goal of our experiments to study the following questions: (1) Are conflicting gradients major factor in making optimization for multi-task learning challenging? (2) Does PCGrad make the optimization problems easier for various multi-task learning problems including supervised, reinforcement, and goal-conditioned reinforcement learning settings across different task families? (3) Can PCGrad be combined with other multi-tar task learning approaches further improve performance?
6.1 EXPERIMENTAL SETUP
To evaluate our method experimentally, we consider both multi-task supervised learning and multi-task reinforcement learning problem setup. For supervised learning, we first consider the MultiMNIST dataset (Sener & Koltun| 2018), which contains two tasks: classifying the digit on the top left and on the bottom right in overlaid image. Beyond digit classification, we also use the CIFAR-I00 dataset (Krizhevsky et al. 2009) where each of the 20 label superclasses are treated as distinct tasks, following Rosenbaum al. (2018), We also conduct experiments the NYUv2 dataset (Silberman et al. 2012). which consists of RGB-D indoor scene images. Following Liu et al. (2018), evaluate method on tasks: 13-clas semantic segmentation, depth estimation, and surface normal prediction. In the case of multi-task reinforcement learning, we evaluate our algorithm on the recently proposed Meta-Word benchmark (Yu al.) 2019). This benchmark includes variety of simulated robotic manipulation tasks contained in a shared, table-top environment with simulated Sawyer arm (visualized the "Push" environment Fig. 3. In particular, we use the multi-task benchmarks MT10 and MT50, which consists of the 10 tasks and 50 tasks respectively depicted in Fig. 3 that require diverse strategies solve them, which makes them difficult optimize jointly with single policy. Note that 1T is a subset of MT50. To evaluate goal-conditioned RL scenarios, we consider goal-conditioned robotic pushing with a Sawyer robot. This domain representative of challenges in learning goal-conditioned policies over wide distribution of goals. For details on the experimental set-up and model architectures see Appendix
6.2 ANALYSIS OF CONFLICTING GRADIENTS
To answer question (1), we consider a simple regression problem, where each task is regressing the input to the output of sine function. The amplitude and the phase of each task are varied. We construct 10 tasks with the amplitude uniformly sampled from the range [0, 5) and the phase uniformly sampled from the range (0, r). The input is also uniformly sampled from the range (0,5)
7

---

Under review as a conference paper at ICLR 2020
Gradient cosine similarity
Loss curve
Cos simianto cask os bank
Adam
Adam
Adams PCGrd
Adam PCGrod
0.50
Sog soy mun to
260
6de
Number of grad ent staps
Number of gradiaat staps
Figure 4: An analysis of the gradients during the first 1000 updates of training, regression problem. Left: The cosine similarity between the gradients of of the 10 tasks (selected arbitrarily. and fixed throughout this plot). We observe a substantial amount of thrashing with standard Adam training, while Adam with PCGrad reduces the thrashing and leads to more closely aligned updates. Right: Adam with PCGrad improves performance compared standard Adam.
toy 10-task sinusoid
and is concatenated with the one-hot task encoding. For training, we use 3-layer fully-connected neural network with 100 hidden units.
We compare the performance of the network trained with Adam and the network trained with Adam with PCGrad-modified gradients while plotting the cosine similarity between a pair of tasks during training shown in Figure The plot the left in Figure demonstratess that the cosine similarity of Adam gradients between a pair of tasks has high variance, which leads to the gradient thrashing problem, while the cosine similarity of the gradient projected by PCGrad yields positive values diminishing the conflicting-gradieet with PCGrad leads to faster learning over Adam, which implies that gradient thrashing indeed problem in multi-task optimization and reducing it can result in considerable performance boost.
problem. As shown in the plot on the right in Figure 4 Adam
6.3 MULTI-TAST AND MULTI- OBJEITVBE SUPERVISED LEARNING WITH PCGRAD
To answer question (3), we perform experiments on three standard multi-task supervised learning datasets: MultiMNIST, multi-task CIFAR- 100 and NYUv2. We include the results on MultiMNIST in Appendix
For CIFAR-100, we follow (Rosenbaum et al.) 2018) to treat 20 coarse labels in the dataset as distinct tasks and create dataset with 20 tasks and 2500 training instances as well as 500 test instances per task. We combine PCGrad with a powerful multi-task learning architecture, routing networks (Rosenbaum et al. 2018 2019). by simply projecting gradients of the shared parameters in routing networks. As shown in Table [ applying PCGrad to single network achieves 71 % classification accuracy, which outperforms most of the prior methods such as independent training and cross-stitch (Misra et al. 2016b). Though routing networks achieve better performance than PCGrad on its own, PCGrad is complementary to routing networks and combining PCGrad with routing networks leads to 2.8% absolute improvement in test accuracy averaged over runs.
We also combine PCGrad with another state-of-art multi-task learning algorithm, MTAN (Liu et al. 2018), and evaluate the performance on more challenging indoor scene dataset, NYUv2, which contains 3 tasks as described in Section [..] We compare MTAN with PCGrad to list of methods mentioned in Section 6.1 where each method is trained with three different weighting schemes as in (Liu et al., 2018), equal weighting, weight uncertainty (Kendall et al.) 201 8a), and DWA (Liu al. [018)) We only run MTAN with PCGrad with weight uncertainty as we find weight uncertainty as the most effective scheme for training MTAN. The results comparing Cross-Stitch, MTAN and MTAN + PCGrad are presented Table 2 while the full comparison can be found in Table in the
Appendix E.3 MTAN with PCGrad is able to achieve the best scores in 8 out of the categories where there are categories per task.
Our multi-task supervised learning results demonstrate that PCGrad can be seamlessly combined with state-of-art multi-task learning architectures and further improve their results on established supervised multi-task learning benchmarks.
6.4 MULTI-TASK REINFORCEMENT LEARNING
To answer question (2), we test all methods on 10 and 50 manipulation tasks respectively shown in Figure At each data collection step, we collect 600 samples for each task, and at each training step,
8

---

Under review as a conference paper at ICLR 2020
% accuracy
task specific-I-fe 1-fc [Rosenbaum al. 2018
42
task specific-all-fè [Rosenbaum et al. 20188)
cross stitch-all-fe (Misra ct al. 2016b)
53
routing-all-fed all-fc+ WPL (Rosenbaum ct al.
2019
74.7
independent
67.7
PCGrad (ours) routing-all-fe- WPL. PCGrad (ours)
71
77.5
Table 1: CIFAR-IOK multi-task results. We apply PCGrad to the routing networks and achieve a significant
improvement in classfication accuracy.
Segmentation
Depth
Surface Normal
NP.
Architecture
Weighting
(ligher Better)
(Lower Better)
Angle Distance (Lower Better)
Within t (Higher Better)
mloU
Pis Acc
Abs Err
Rel Fir
Mean
Median
11.25
22.5
30
Equal Weights
14.71
50.23
0.6481
0.2871
33.56
28.58
20.08
40.54
$1.97
A33
Cross Stitch
Uncert. Weights"
15.69
52.60
0.6277
0.2702
32.69
27.26
21.63
42.84
54.45
T
16.11
53.19
0.2611
26.91
21.81
43.14
54.92
Equal Weights
17.72
55.32
0.5906
0.2577
31.44
25.37
23.17
45.65
57.48
MTANT
Uncert. Weights"
17.67
55.61
0.5927
0.2592
31,25
25.57
22.99
45.83
57.67
DWAT T 2
17.15
54.97
0.5956
0.2569
31.60
25.46
22.48
44.86
57.24
1.77 MTAN'+ PCGrad (ours)
Uncert. Weights"
20.17
[6.65
0.5904
0.2467
30.01
24.83
22.28
46.12
[5.777
Table We present results on three tasks on the NYUv2 dataset: 13-class semantic segmentation, depth estimation, and surface normal prediction results. #P shows the total number of network parameters. We highlight the best performing combination of multi-task architecture and weighting in bold. The top validation
scores for each task annotated with boxes. The symbols indicate prior methods: ": (Kendall al. (0188a)
Liu et al. 2018 : Misra et al. 2016b). Performance other methods as reported in (Liu et al 2018
MT20
MI50
Coal-Conditioned Pushing
1.31
SIC. M.cd
MACHM
Sucress Rates
Distance to the ghal
8.40
0:45
0.3
Success
0.2
615
39:0
2000
2003
4003 5000
5000
10000
35300
20006 610
560
2000
15.00 thousand CIV stops
2201
ZA0 J0CB
Number thousand onv steps
Number thousand emy steps
Figure 5: Learning curve on MT10, MT50 and goal-conditioned pushing. PCGrad outperforms the other methods in the three settings in terms of both success rates / average distance to the goal and data efficiency.
sample 128 datapoints per task from corresponding replay buffers. The results are shown in the two plots on the left in Figure We measure success according to the metrics used in the Meta- Meta-World benchmark where the reported the success rates averaged across tasks. For all methods, we apply PA as discussed in Section to learn a separate alpha term per task as the task encoding in MT10 and MT50 is just one-hot encoding. PCGrad combined with SAC learns all tasks with the best data efficiency and successfully solves all of the 10 tasks in MT10 and about 70% of the 50 tasks in MT50. Training a single SAC policy and a multi-head policy turns out be unable acquire half of the skills in both MT10 and MT50, suggesting that eliminating gradient interference across tasks can significantly boost performance of multi-task RL. Training independent SAC agents is able to eventually solve all tasks in MT and 70% of the tasks in MT50, but requires about 2 millions and 15 millions more samples than PCGrad with SAC MT10 and MT50 respectively, implying that applying PCGrad can result in leveraging shared structure among tasks that expedites multi-task learning.
As noted by Yu et al. 22119), these tasks involve fairly distinct behavior motions, which makes learning all of them with a single policy challenging as demonstrated by poor baseline performance. The ability to learn these tasks together opens the door for number of interesting extensions to meta-lcarning, goal conditioned RL and generalization to novel task families. We present the results of PCGrad goal-conditioned RI. in the following subsection.

---

Under review as a conference paper at ICLR 2020
We also provide ablation study on the importance of correcting the gradient direction and scaling the gradient magnitudes in PCGrad. We construct two variants of PCGrad: (1) only applying the gradient direction corrected with PCGrad while keeping the gradient magnitude unchanged and (2) only applying the gradient magnitude computed by PCGrad while keeping the gradient direction unchanged. As shown in the plot on the left in Figure both variants perform worse than PCGrad and the variant where we only vary the gradient magnitudes is much worse than PCGrad. We also compare PCGrad to a prior method GradNorm (Chen et al. 2018). which scales the magnitude of gradients of all the tasks. As shown in the plot on the right in Figure 6 PCGrad significantly outperforms GradNorm. We also notice that the variant of PCGrad where only the gradient magnitudes change gets comparable results to GradNorm, which suggests that its important to modify both the gradient directions and magnitudes to eliminate interference and achieve good multi-task learning results.
MT10
MT10
SAC SaefCaa
SAC-PCOnat- tuut
Success Rates
2/2
Success Retes
0.2-
0.4
0.1
5.
0.0
3200
2000
3000
4000
5000
3000
2000
Number of thousand env steps
Number of thousand env steps
Figure 6: Ablation study only using the magnitude and the direction of the gradients modified by PCGrad (left) and comparison between PCGrad and GradNorm (Chen et al. 2018) (right). PCGrad outperforms both ablations and GradNorm with large margin, indicating the importance of modifying both the gradient directions and magnitudes in multi-task learning.
6.5 GOAL-CONDITION
REINFORCEMENT LEARNING
For our goal-conditioned RL evaluation, we use the robot-pushing environment described in Sec. 6. where the goals are represented as the concatenations of the initial positions of the puck to be pushed and the its goal location, both of which apply PA as discussed in Section 4 to predict the temperature for entropy term given the goal. We summarize the results in the plot on the right in Figure PCGrad with SAC and PA achieves the best performance in terms of average distance to the goal position, while PCGrad with SAC improves over the baseline and a vanilla SAC agent is struggling to successfully accomplish the task. This suggests that PCGrad is able to ease the RL optimization problem also when the task distribution is continuous.
uniformly sampled (details in Appendix (..)). We also
CONCLUSION
In this work, we identified one of the major challenges in multi-task optimization: conflicting gradients across tasks. We proposed a simple algorithm (PCGrad) to mitigate the challenge of conflicting gradients via "gradient surgery". PCGrad provides simple way to project gradients to be orthogonal in a multi-task setting, which substantially improves optimization performance, since the task gradients are prevented from negating each other. We provide some simple didactic examples and analysis of how this procedure works in simple settings, and subsequently show significant improvement in optimization for variety of multi-task supervised learning and reinforcement learning problems. We show that, once some of the optimization challenges of multi-task learning are alleviated by PCGrad, we can obtain the hypothesized benefits in efficiency and asymptotic performance that are believed to be possible in multi-task settings.
While we studied multi-task supervised learning and multi-task reinforcement learning in this work. we suspect the problem of conflicting gradients to be prevalent in a range of other settings and applications, such as meta-learning, continual learning, multi-goal imitation learning (Codevilla et al. 2018), and multi-task problems in natural language processing applications (McCann et al. 2018). Due to its simplicity and model-agnostic nature, we expect that applying PCGrad in these domains to be a promising avenue for future investigation. Further, the general idea of gradient surgery may be an important ingredient for alleviating a broader class of optimization challenges in deep learning, such as the challenges in the stability challenges in two-player games (Roth et al.) 2017) and multi-agent optimizations (Nedic & Ozdaglar 2009). We believe this work to be a step towards simple yet general techniques for addressing some of these challenges.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(12)2441-2445, 2017.
Bart Bakker and Tom Heskes. Task clustering and gating for bayesian multitask learning. J. Mach. Learn. Res., 2003.
Hakan Bilen and Andrea Vedaldi. Integrated perception with recurrent multi-task neural networks. In Advances in neural information processing systems, pp. 235-243, 2016.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Paul H Calamai and Jorge Moré. Projected gradient methods for linearly constrained problems. Mathematical programming, 39(1)933-116, 1987.
Rich Caruana. Multitask learning. Machine Learning, 1997.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient 2018.
lifelong learning with a-gem. arXiv preprint
Zhao Chen, Vijay Badrinarayanan, Chen-Yr Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient nor- malization for adaptive loss balancing in deep multitask networks. arXiv preprint 17X1v:/711.02257, 2017.
Zhao Chen, Vijay Badrinarayanan, Chen-Y. Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of the International Conference on Machine Learning, 2018.
Felipe Codevilla, Matthias Millerr, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to- end driving via conditional imitation learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-9. IEEE, 2018.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167.60 ACM, 2008
Wojciech M. Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant M. Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS, 2019.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. CoRR, (11///009007700 2016.
Daxiang Dong, Hua WU, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computa- tional Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pp. 1722-1732, 2015.
Yunshu Du, Wojeiech M. Czarnecki, Siddhant M. Jayakumar, Razvan Pascanu, and Balaji Lak- shminarayanan. Adapting auxiliary losses using gradient similarity. CoRR, abs/18122 111.02224, 2018.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, 2018
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, fabs//701.087340 2017. http:ttppp/axxl..ooog//oo/
001770111.77.4 08734
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning. CoRR, abs/1711.099 2017.
11

---

Under review as a conference paper at ICLR 2020
Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. problemss arXiv preprint arXiv:1412.6544 2014.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy
maximum entropy deep reinforcement learning with stochastic actor. International Conference on Machine Learning. 2018.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. 2018.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3796-3803, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1rXiv:1503.0253 2015.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 201 8a.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. CVPR, 2018b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6991 2014.
lasonas Kokkinos Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6129-6138, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-337, 2016. 2016.
Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. CoRR, abs/1803. 107.00044, 2018.
Mingsheng Long and Jianmin Wang. Learning multiple tasks with deep relationship networks. arXiv preprint arXiv:1506.021/1 2, 2015.
David Lopez-Paz and Mare Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Kevis-Kokits, Maninis, Ilija Radosavovic, and Iasonas Kokkinos. Attentive single-tasking of multiple tasks. CoRR, abs/ 10//1994408188 88, 2019.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint 18Xiv: 2018.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994-4003, 6a.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Conference on Computer Vision and Pattern Recognition, CVPR, 2016b.
Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control, 54(1 5441):48, 2009.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement leamnnng arXiv preprint arXiv: 1./66664, 2015.
12

---

Under review as a conference paper at ICLR 2020
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch. arXiv preprint arXiv: 10567, 2018.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. International Conference on Learning Representations (ICLR), 2018
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprini 1904. /2774, 2019.
Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In Advances in neural information processing systems, pp. 2018-2028, 2017.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint
arXiv:1706. 05098,000800 2017.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçchre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In 4th International Conference on Learning Representations, ICLR, 2016a.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1ee.1166606771 2016b.
Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: source of plateaus in deep reinforcement learning. arXiv preprint 19X4vv/904..145:: 2019.
Ozan Sener and Vladlen Koltun. Multi-task learning as multiiobjecttive optimization. In Advances in Neural Information Processing Systems, 2018
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision, pp. 746-760. Springer, 2012.
Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4496-4506, 2017.
Simon Vandenhende, Bert De Brabandere, and Luc Van Gool. Branched multi-task networks: Deciding what layers to share. CoRR, abs/1904 0000/999,,,22,0, 2019.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pp. 1015-1022. ACM, 2007.
Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Neunert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Regularized hierarchical policies for compositional transfer robotics. arXiv preprint arXiv:1906.1 1228, 2019.
Yongxin Yang and Timothy M Hospedales. Trace norm regularised deep multi-task learning. arXiv preprint arXiv: 160.000800. 2016.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta-reinforccement learning. 2019.
arXiv preprint arXiv:
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3712-3722, 2018
13

---

Under review as a conference paper at ICLR 2020
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-tast task learning. In European conference on computer vision, pp. 94-108. Springer, 2014.
A PROOF OF THEOREM
Proof. We will use the shorthand || to denote the L2-norm and VL VaL, where is parameter vector. Let gi VL1, g2 v C2, and á be the angle between g1 and g2-
At each PCGrad update, we have two cases: cos( 2 0 or cos( < 0).
If cos( () 2 O, then we apply the standard gradient descent update using S 1 which leads to strict decrease the objective function value L(Ø) unless VL(Ø) = O, which occurs only when 0 0 (Boyd & Vandenberghe 2004).
In the case that cos(@) O, we proceed as follows:
Our assumption that V C is Lipschitz continuous with constant L. implies that V22(0) 1 is negative semidefinite matrix. Using this fact, we can perform a quadratic expansion of C around C()) and obtain the following inequality:
C(O+) - (()) + VL(O) - + [² - 0112
2
1
S [()) + VL(0)" 0) 2
10 - B112
Now, we can plug in the PCGrad update by letting 0+ (VL((0) then get:
$1.82
g1
18a
g2). We
C(O+) < ((V()))=(()))
+
81
82
81
+
81 g2
|
gg
2 (
gg
81
g2
g1
82
1g
gg
(Expanding, using the identity VC = g1 + g2)
(gi 82)²
(g) g2)²
-()) + +-1(11g11
|g2
+
2g1
g2 +
||
g2
2
2 g1 +
Lt2
g2
81
82
|g
g2
(Expanding further and re-arranging terms)
= = - (t
Lt
(
g1
2
12
(gi
82)
[gi g2
2
1g
g2
[g
g2
-L/² g2
81
(22
gg
g2
(Using the identity cos(ø)
g1 g2
gg 82
= C - (t - Lt [1
2 cos? (o))
ga ||
-L - (o)) g1 |
cos(o)
(Note that cos(Ø) < 0 so the final term is non-negative)
14

---

Under review as a conference paper at ICLR 2020
Using S 1. know that - 3 -1 and Lt t.
Plugging this into the last expression above, we can conclude the following:
[()+) - (()) 2 [(11 - cos cos)
1
cos (g)))
- 1(1 cos (0)) gi 82 (o)
1
((0)
2 t(1
cos (6)) g1 + 2
g2 cos(() +
gg |
2
1
[(0)
f(1 cos (6)) g1
2g1
B2 + +|
2
1
C(0)
f(1 cos (0)) 82
2
((0)
t(1 cos (Z))
2
will always be positive unless VL(()) 0. This inequality implies that the objective function value strictly decreases with each iteration where cos(Ø) -1.
cos(1) -1, then 1/(1
Hence repeatedly applying PCGrad process either reach the optimal value L()) L( ) or cos(Φ) = 1, which case {((1 (o) =0. 0. Note that this result only holds when we choose to be small enough, i.e. 1 1 L
B 2D OPTIMIZATION LANDSCAPE DETAILS
To produce the 2D optimization visualizations in Figure R² and the following task loss functions:
we used a parameter vector 0 = [01, 02]
( (0) 20 log (max( 550 tanh(02)), 0.000005)
(0) = 25 log max 56 tanh(02) 2), 0.000005)
The multi-task objective is L(O) (i + (0)). We initialized 0 [0.5, -3 and performed 500,000 gradient updates to minimize L using the Adam optimizer with learning rate 0.001. We compared using Adam for each update to using Adam in conjunction with the PCGrad method presented in Section 3.2
C EXPERIMENTAL RESULTS ON MULTIMNIST
Following the same set-up in Sener & Koltun (2018), for each image, sample different one uniformly at random. Then we put one of the image on the top left and the other one on the bottom right. The two tasks in the multi-task learning problem and bottom right (task-R) respectively. We construct such 60K examples. We combine PCGrad with the same backbone architecture used in (Sener Koltun 2018) and compare its performance |enner & Koltun (2018) by running the open-sourced code provided in (Sener & Koltun 2018). As shown in Table our method results 13% and 0.55% improvement over Sener & Koltun (2018) left and right digit accuracy respectively.
classify the digits the top left (task-L)
left digit
right digit
Sener & Koltun (2018)
96.45
94.95
PCGrad (ours)
96..58
95..50
Table 3: MultMNNIST results. PCGrad achieves improvements over Sener & Koltun (001 in both left and right digit classfication accuracy.
D ABLATION STUDY ON THE TASK ORDER
As stated on line 4 in Algorithm | we sample the tasks from the batch and randomly shuffle the order of the tasks before performing the update steps in PCGrad. With random shuffling, make
15

---

Under review as a conference paper at ICLR 2020
PCGrad symmetric w.r.t. the task order in expectation. In Figure 7 random task order achieves better performance between PCGrad with a fixed task order in the setting of MT50 where the number of tasks is large and the conflicting gradient phenomenon is much more likely to happen.
we observe that PCGrad with
MT50
A
PMG-RR+P lours) PAC-PA fheed (ours)
0.5
0.5
Success
0.2
0.1
6000 GD0c 10200 Number thousand env steps
2000
4000
Figure 7: Ablation study on using fixed task order during PCGrad. PCGrad with random task order does significantly better PCGrad with fixed task order MT50 benchmark.
E EXPERIMENT DETAILS
E. DETAILED EXPERIMENT SET-UP
For our CIFAR- 100 multi multi-task experiment, we adopt the architecture used in Rosenbaum et al. (2019), which is convolutional neural network that consists of convolutional layers with 160 3 filters each layer and 2 fully connected layers with 320 hidden units. As for experiments on the NYUv2 dataset, we follow Liu et al. (2018) to use SegNet (Badrinarayanan al.) 2017) as the backbone architecture.
Our reinforcement learning experiments all use the SAC (Haarnoja et al. 2018) algorithm as the base algorithm, where the actor and the critic are represented as 6-layer fully-connected feedforward neural networks for all methods. The numbers of hidden units of each layer of the neural networks are 160, 300 and 200 for MT10, MT50 and goal-conditioned RL respectively.
We use five algorithms as baselines in the CIFAR- 100 multi-task experiment: task specific-1-EF (Rosenbaum et al. 2018): convolutional neural network shared across tasks except that each task has separate last fully-connected layer, task specific-1-fe (Rosenbaum et al. 2018) all the convolutional layers shared across tasks with separate fully-connected layers for each task, cross stitch-all-fc (Misra et al. (0016b): one convolutional neural network per task along with cross-stitch units to share features across tasks, routing-all-fe + WPL (Rosenbaum et al., (019): network that employs trainable router trained with multi-agent RL algorithm (WPL) to select trainable functions for each task, independent: training separate neural networks for each task.
For comparisons on the NYUv2 dataset, we consider 5 baselines: Single Task, One Task: the vanilla SegNet used for single-task training, Single Task, STAN (Liu et al.) (2018): the single-task version of MTAN as mentioned below, Multi-Task Split, Wide Deep (Liu et al. 2018): the standard SegNet shared for all three tasks except that each task has a separate last layer for final task specifee prediction with two variants Wide and Deep specified in Liu et al. (2018), Multi-Task Dense: shared network followed by separate task task-specific networks, Multi-Task Task Cross-Stitch (Misra et al.. 2016b); similar to the baseline used in CIFAR- 100 experiment but with SegNet as the backbone, MTAN (Liu et al., 2018): a shared network with a soft-attention module for each task.
On the multi-task and goal-conditioned RL domain, apply PCGrad to the vanilla SAC algorithm with task encoding as part of the input to the actor and the critic as described in Section and compare method to the vanilla SAC without PCGrad and training actors and critics for each task individually (Independent).
E.2 GOAL-CONDITIONED EXPERIMENT DETAILS
We use the pushing environment from the Meta-World benchmark (Yu et al., 2019) as shown in Figure 3 In this environment, the table spans from [-0.4, 0.2] to (0.4, 1.0) in the 2D space. To
16

---

Under review as a conference paper at ICLR 2020
construct the goals, sample the intial positions of the puck from the range [-0.2, 0.6] to [0.2. 0.7] on the table and the goal positions from the range [-0.2, 0.85] to [0.2, 0.95 on the table. The goal is represented as concatenation of the initial puck position and the goal position. Since in the goal-conditioned setting, the task distribution is continuous, we sample a minibatch of 9 goals and 128 samples per goal at each training iteration and also sample 600 samples per goal in the minibatch at each data collection step.
E.3 FULL NYUv2 RESULTS
We provide the full comparison on the NYUv2 dataset in Table
Sogmentation
Dopch
Surface Normal
Type
#P.
Architecture
Weighting
(Higher Better)
(Lower Better)
Angle Distance
(Lower Better)
Within 2 (Higher Better)
mloU Pix Acc
Abs Err
Rel
Meun Median
11.25
22.5
30
Single Task
One Task
n.a.
15.10
51.54
0,7508
0.3266
31.76
25.51
22.12
45.33
57.13
4.56 STAN
15.73
0.2891
32.09
21.49
Equal Weights
15.89
51,19
0.6494
0,2804
33.69
28.91
18.54
39.91
52.02
1,75 Split, Wide
Uncert. Weights
15.86
51.12
0.6040
0.2570
32.33
26.62
21.68
43.59
55.36
DWA
32.34
20.69
42.73
Equal Weights
13.03
41.47
0.7836
0.3326
38.28
36.55
9.50
27. 11
39.63
2. Split, Deep
Uncert. Weights
14.53
43.69
0.7705
0.3340
35.14
32.13
14.69
34.52
46.94
DWA
13.63
0.3227
36.41
34.12
12.82
31.12
43.48
Multi Task
Equal Weights
0.6488
33.58
4,95 Dense
Uncert. Weights
16.48
54.40
0.6282
0.2761
31.68
25.68
21.73
44.58
56.65
DWA
16.15
54.35
0.6059
0.2593
32.44
27.40
20.53
42.76
54.27
Equal Weights
0.6481
33.56
20.08
40.54
51.97
A3 Cross-Stitch"
Uncert. Weights"
15.69
$2.60
0.6277
0.2702
32.69
27.26
21 21.63
42.84
54.45
DWA1 1WWA1.0 T =
16.11
53.19
0,5922
0.2611
32.34
26.91
21.81
43.14
54.92
Equal Weights
17.72
0.5906
0.2577
31.44
25.37
[23.17
45.65
1,77 M
Uncert. Weights"
17.67
55.61
0.5927
0.2592
31,25
25.57
22.99
45.83
57.67
DWA T' =
17.15
54.97
0.5956
0.2569
31.60
25.46
22.48
44,86
57.24
1,77 MTAN' PCGrrad (ours) Uncert. Weights"
20.17
36.65
0.5904
0.2467
30.01
24.83
22.28
46.12
58.77
Table 4: We present the full results three tasks on the NYUv2 dataset: 13-class semantic segmen- tation, depth estimation, and surface normal prediction results. #P shows the total number of network parameters. We highlight the best performing combination of multi-task architecture and weighting in bold. The top validation scores for each task are annotated with boxes. The symbols indicate prior methods: ": (Kendall al. 2018a). 1; (Liu al. 2018). & (Misra et al. 2016b). Performance of other methods taken from
(Liu et al. 2018).
17


Reviews:
[{"review_id": "rJgFulCl9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes as solution to manage the case where gradients are conflicting in gradient-based Multi-Task Learning (MTL), pointing to different directions. They propose a simple “gradient surgery” technique that alters the gradients by projecting a conflicting gradient on the normal vector of the other one, in order to mitigate the effect. The method is generic in the sense that it can be directly applied to various gradient-based architectures easily.\n\nThe paper is well written and easy to follow. However, the whole proposal relies on the assumption that conflicting gradients are common and harmful for MTL. For simple convex models, like the model used in the theorem, I get that this can be an issue. But using a convex model for MTL seems no that common. And as far as I know, MTL is not used commonly with simple models, it is rather common is with deep neural networks, where a common part of the network (i.e. representation) is shared among the tasks, while we have a distinct head, of one or few layers, for each task. With such a setting, we expect to have enough capacity to model the various tasks, such that the neural network model should be able to model both tasks independently if they are in complete contradiction. In such case, over the training, conflicting gradients over some neurons with be a transient phenomenon, with the neurons specializing on one or the other task, or just be disabled. In practice, common elements will be shared, while contradictory elements will be in the task-specific part of the network. Said otherwise, I think that conflicting gradients may appear on some neurons over some data, but will be mostly a stochastic phenomenon which is not necessarily harmful on the long run, much like the stochasticity of picking a sequence of data from different classes in SGD.\n\nI may be not totally right in speculating in such way on what is going on with conflicting gradients in MTL. My point is to show that the whole paper is based on assumptions that are not verified. I would need to be convinced that we are tackling a real problem, not such an idea of something that may happen but in practice is quite rare or not necessarily that harmful. The experiments over the toy problem on trashing gradient (Sec. 3.1 and Fig. 1) is interesting to illustrate the problem, but show the issue in a 2D setting, where the model has very little degree of freedom. With deep networks, I don’t think that such constrained search space is common.\n\nAnother issue with the paper is the lack of comparison with other approaches for MTL. A conjecture I have on the performance of the approach is that it may dampen the gradient for the loss of the different tasks, and their amplitude, to get to an effect similar to what GradNorm is doing. In fact, all results reported are on a base method and one with PCGrad. Comparison with other methods to handle MTL is required in my opinion, in particular with GradNorm, which may have similar effects than the current one. If PCGrad and GradNorm achieve similar results, my guess is that the issue is not with conflicting gradients, but rather bad scaling of the losses when they are put together.\n\n** Update ** : After reading the authors' comments and other reviews, I would maintain a \"weak reject\", although if there was a \"borderline\" choice, I would have used it. The authors provide reasonable answer to my request, although I think the paper could have been stronger in term of experimentations. Also, the other reviews and comments on existing work make it clear to me that the positioning to other related work was incomplete. I think the proposal and paper is correct and interesting, so still below the acceptance threshold for ICLR.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byl3R3e3tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a simple rule to ignore the so-called conflicting gradients in addressing multi-task learning problems. The underlying idea is very straight-forward if two gradients are contradictory (the angle between them is > \\pi) then one should not be considered.\n\nI have some questions here. Let's assume the gradients passing to PCGrad are g1, g2, and g3 (in that order).\n\n1- In PCGrad, we start with g1 and will keep g2, or g3 if their direction complies with g1, isn't it better to keep the gradient that minimizes the loss more (or say has a bigger norm) instead? Maybe one should first pick which gradient is more important and then use that to start PCGrad.\n\n2- One may argue that if <g1,g2> <= 0, <g1,g3> <= 0, PCGrad will ignore both g2 and g3, hence, intuitively it should converge slowly and probably not generalizable. Can you comment on why I should not worry about this?\n\n3- Is it possible to generalize PCGrad to work with more than two gradient vectors? One imagines that if it can work with more than 2 gradients, maybe a better and more robust algorithm can be achieved.\n\nBased on the above, I believe that the paper in its current form has not completely studied the problem and hence I am giving the paper a weak reject score at this stage.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1ljM4ehtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method to boost multi-task learning performance by editing gradient to remove conflicts between tasks. The main idea is to use cosine similarity to 1) determine if two task gradients conflict and 2) to project one conflicting gradient to the normal plane of the other, thereby removing the conflict at the expense of disturbing the other gradient to some extent. Experiments are presented for classification and other computer vision tasks along with reinforcement learning problems.\n\nOverall, I really liked this paper. The explications are clear, the visualizations provided help the understanding (especially Fig.1), and results are compelling. I definitely value the way the method is straightforwardly presented: the underlying idea is simple yet strong. There are, however, a few elements precluding me to pick a higher rating, which I describe in details here.\n\nFirst, there are a lot of similarities with the MTL method of Sener and Koltun 2018. In particular, I do not agree with the statement that \"[this] work, in contrast to many of these optimization schemes [incl. Sener and Koltun], suggests that the challenge in multi-task learning can be attributed to the problem of gradient trashing, which we address directly by introducing a practical algorithm that de-conflicts gradients from different tasks.\" MTL has the concept of \"common descent direction\" and, as Fig.1 of relevant paper suggests, it does \"de-conflict\" the gradients. Sure, the wording is not the same, but the idea is there nonetheless.\nTo be clear, this is not to say that PCGrad has no merits. I do find it simpler and more elegant (although the latter is a subjective assessment). But I think the similarities should have been discussed in greater details, and the performance compared with Sener and Koltun on at least one problem.\n\nSecond, the experiments make it difficult to see the performances of PCGrad alone. Indeed, it is always combined with another multi-task approach/algorithm (MTAN, WPL, SAC+PA, etc.). Providing these results is not incorrect in itself, but it makes it difficult grasp what PCGrad can do alone. Is it worth using only in conjunction with other approaches, or could someone consider using it in a standalone manner? Is PCGrad more a \"gradient fine-tuner\" than a comprehensive solution for multi-task leearning? The experiments presented, although significant, do not answer these questions.\n\nThird, I am unsure about the value of Theorem 1 and its proof in Sec. 3.2. It assumes too much (e.g., 2 convex tasks) to be of any use in practice. Also, leading to a minimizer of L does not forcibly mean leading to a good solution, depending on the definition of L1 and L2.\n\nThere is also one element I'm unsure about. Looking at Algorithm 1, it looks like the order of the tasks may have an effect. Indeed, since it is g_i that is modified at line 8 and not g_j, the last element of B will always remain unaltered (because all other task gradients will have previously been modified to avoid gradient conflict with it). The second to last element of B will be altered, but only due to potential conflicts with the last, and so on, up to the first which can potentially be altered by all others. As an effect, some gradients will always be significantly more altered than others, which can have an impact (at least theoretical) on the learning process. Algorithm 1 thus looks like a greedy approach. Nothing bad in itself (and some ad hoc adjustments, like shuffling B at each update, could very well fix this), but I think this deserves more discussion.\n\nFinally, a more generic comment: the naming of the method should remain the same through the paper. The abstract/intro/Fig.1 refer to the technique as \"gradient surgery\", while the explanations and experiments talk about PCGrad. Also, in the introduction \"plateuas\" -> \"plateaus\".\n\nIn summary, I think this is a good paper, presenting a straightforward and useful idea for multi-task learning. However, the related work is not always well described, the experiments lack important comparisons, and the practical effects of PCGrad should be explained in more details instead of focusing on a proof for a convex case.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for improving optimization in multi-task learning settings by minimizing the interference of gradients belonging to different tasks. 

While the idea is simple and well-motivated, the reviewers felt that the problem is still not studied adequately. The proofs are useful, but there is still a gap when it comes to practicality.

The rebuttal clarified some of the concerns, but still there is a feeling that (a) the main assumptions for the method need to be demonstrated in a more convincing way, e.g. by boosting the experiments as suggested with other MTL methods (b) by placing the paper better in the current literature and minimizing the gap between proofs/underlying assumptions and practical usefulness. 

Decision: Reject