Recon: Reducing Conflicting Gradients From the Root For Multi-Task Learning
Authors: Guangyuan SHI
Qimai Li
Wenlong Zhang
Jiaxin Chen
Xiao-Ming Wu
Venue: ICLR 2023 poster (2023)
Keywords: Multi-task Learning
Conflicting Gradients

Abstract: A fundamental challenge for multi-task learning is that different tasks may conflict with each other when they are solved jointly, and a cause of this phenomenon is conflicting gradients during optimization. Recent works attempt to mitigate the influence of conflicting gradients by directly altering the gradients based on some criteria. However, our empirical study shows that ``gradient surgery'' cannot effectively reduce the occurrence of conflicting gradients. In this paper, we take a different approach to reduce conflicting gradients from the root. In essence, we investigate the task gradients w.r.t. each shared network layer, select the layers with high conflict scores, and turn them to task-specific layers. Our experiments show that such a simple approach can greatly reduce the occurrence of conflicting gradients in the remaining shared layers and achieve better performance, with only a slight increase in model parameters in many cases. Our approach can be easily applied to improve various state-of-the-art methods including gradient manipulation methods and branched architecture search methods. Given a network architecture (e.g., ResNet18), it only needs to search for the conflict layers once, and the network can be modified to be used with different methods on the same or even different datasets to gain performance improvement. The source code is available at https://github.com/moukamisama/Recon.

Published as a conference paper at ICLR 2023
RECON: REDUCING CONFLICTING GRADIENTS FROM THE ROOT FOR MULTI-TASK LEARNING
Guangyuan Shi, Qimai Li, Wenlong Zhang, Jiaxin Chen, Xiao-Ming Wu Department of Computing, The Hong Kong Polytechnic University, Hong Kong S.A.R., China (guang-yuan. shi, qee-mai. li,, wen ong. z connect......... po....ollluttl. jiax. chen@connect. polyu. hk, xiao-ming: wuêpol yu. .hu
ABSTRACT
A fundamental challenge for multi-task learning is that different tasks may con- flict with each other when they are solved jointly, and a cause of this phenomenon is conflicting gradients during optimization. Recent works attempt mitigate the influence of conflicting gradients by directly altering the gradients based on some criteria. However, our empirical study shows that "gradient surgery" cannot ef- fectively reduce the occurrence of conflicting gradients. In this paper, we take a different approach to reduce conflicting gradients from the root. In essence, we investigate the task gradients w.r.t. each shared network layer, select the layers with high conflict scores, and turn them lask-specific layers. Our experiments show that such a simple approach can greatly reduce the occurrence of conflict- ing gradients in the remaining shared layers and achieve better performance, with only slight increase in model parameters in many cases. Our approach can be easily applied to improve various sfate-of-thh-ar methods including gradient ma- nipulation methods and branched architecture search methods. Given network architecture (e.g.. ResNet18), 8), it only needs to search for the conflict layers once, and the network can be modified to be used with different methods on the same or even different datasets to gain performance improvement. The source code is available at https:: hightthhhh.... com/ moukami sama/ /Recon)
INTRODUCTION
Multi-task learning (MTL) is learning paradigm in which multiple different but correlated tasks are jointly trained with a shared model (Caruana| 1997). in the hope of achieving better performance with an overall smaller model size than learning each task independently. By discovering shared structures across tasks and leveraging domain-speciio
training signals of related tasks, MTL can achieve efficiency and effectiveness. Indeed, MTL has been successfully applied in many domains including natural language processing (Hashimoto et al. (017), reinforcement learning (Parisotto) al. 2016 D' Bramm al. 2020) and computer vision (Vandenhende et al. 2021).
major challenge for multi-task learning is negative transfer (Ruder) 2017). which refers to the performance drop on a task caused by the learning of other tasks, resulting in worse overall perfor- mance than learning them separately. This is caused by task confficts, i.c., tasks compete with each other and unrelated information of individual tasks may impede the learning of common structures. From the optimization point of view, a cause of negative transfer is conflicting gradients (Yu et al.) 2020), which refers to two task gradients pointing away from each other and the update one task will have negative effect on the other. Conflicting gradients make it difficult to optimize the multi- task objective, since task gradients with larger magnitude may dominate the update vector, making the optimizer prioritize some tasks over others and struggle to converge to desirable solution.
Prior works address task/gradient conflicts mainly by balancing the tasks via task reweighting or gradient manipulation. Task reweighting methods adaptively re-weight the loss functions by ho- moscedastic uncertainty (Kendall et al. 2018), balancing the pace at which tasks are learned Chen et al. (2018); Liu et al.) (2019). or learning loss weight parameter (Liu et al. 2021b). Gradient manipulation methods reduce the influence of conflicting gradients by directly altering the gradients
based different criteria (Sener & Koltun 2018 Yu al. 2020 Chen al. 2020 Liu et al.

---

Published as a conference paper at ICLR 2023
2021a) rotating the shared features (Javaloy & Valera 2022). While these methods have demon- strated effectivennes in different scenarios, in our empirical study, we find that they cannot reduce the occurrence of conflicting gradients (see Sec. 3.3 for more discussion).
We propose a different approach to reduce conflicting gradients for MTL. Specifically, we investi- gate layer-wise conflicting gradients, i.e., the task gradients w.r.t. each shared network layer. We first train the network with a regular MTL algorithm (e.g., joint-training) for a number of iterations, com- pute the conflict scores for all shared layers, and select those with highest conflict scores (indicating severe conflicts). We then set the selected shared layers task-specific and train the modified network from scratch. As demonstrated by comprehensive experiments and analysis, our simple approach Recon has the following key advantages: (1) Recon can greatly reduce conflicting gradients with only a slight increase in model parameters (less than 1% in some cases) and lead to significantly bet- ter performance. (2) Recon can be easily applied to improve various gradient manipulation methods and branched architecture search methods. Given network architecture, it only needs to search for the conflict layers once, and the network can be modified to be used with different methods and even on different datasets to gain performance improvement.
(3) Recon can achieve better performance than branched architecture search methods with much smaller model.
2 RELATED WORKS
In this section, we briefly review related works in multi-task learning in four categories: tasks clus- tering, architecture design, architecture search, and task balancing. Tasks clustering methods mainly focus on identifying which tasks should be learned together (Thrun & O' D'ullivan 1996
Zamir et al.
2018 Standley et al. 2020: Shen et al., 2021 Fifty et al. 2021).
Architecture design methods include hard parameter sharing methods (Kokkinos) 2017 Long al. 2017 Bragman et al. 2019). which learn a shared feature extractor and task sask-specificc decoders, and
soft parameters sharing methods (Misra et al., [016 |Ruder al., 2019 Gao et al.) 2019 2020 Liu
jet al. 2019). where some parameters of each task are assigned to do cross-task talk via a sharing mechanism. Compared with soft parameters sharing methods, our approach Recon has much better scalability when dealing with a large number of tasks.
Instead of designing a fixed network structure, some methods (Rosenbaum et al. 2018 Meyerson & Mükkuulanee 2018: Yang et al.) (2020) propose to dynamically self oorgaoree the network for differ- ent tasks. Among them, branched architecture search (Guo et al.) 2020; Bruggemann et al. 2020) methods are more related to our work. They propose an automated architecture search algorithm to build tree-structured network by learning where to branch. In contrast, our method Recon decides which layers to be shared across tasks by considering the severity of layer- wise conflicting gradients, resulting in a more compact architecture with lower time cost and better performance.
Another line of research is task bulancing methods. To address task/gradient conflicts, some meth- ods attempt to re-weight the multi-task loss function using homoscedastic uncertainty (Kendall et al.) 2018), task prioritization (Guo et al.] 2018), similar learning pace (Liu et al.) 2019: 202 GradNorm (Chen et al. (018)) learns task weights by dynamically tuning gradient magnitudes. MGDA (Sener & Koltun, (018)) find the weights by minimizing the norm of the weighted sum of task gradients. To reduce the influence of conflicting gradients, PCGrad (Yu et al.) (020)) projects each gradient onto the normal plane of another gradient and uses the average of projected gradi- ents for update. Graddrop (Chen et al. (0200) randomly drops some elements of gradients based on element-wise conflict. CAGrad (Liu et al.) 2021a) ensures convergence age loss across tasks by gradient manipulation. RotoGrad (Javaloy & Valera 2022) re-weights task gradients and rotates the shared feature space. Instead of manipulating gradients, our method Recon leverages gradient information to modify network structure to mitigate task conflicts from the root.
minimum of the aver-
PILOT STUDY: TASK CONFLICTS IN MULTI-TASK LEARNING
3.1 MULTI-TASK LEARNING: PROBLEM DEFINITION
Multi talki learning (MTL) aims to learn a set of correlated tasks (T) simultaneously. For each task Tú the empirical loss function is L: (0,n, 0z), where Ozh are parameters shared among all tasks
2

---

Published as a conference paper at ICLR 2023
0.35
Joint-train
Gradorop
0.25
PCGrad
0.20
CAGrad
6.10
0.65
[1.00.0.0]]
(0.03.0.02)
[0.02,0.01]
10.01:0.00]
10.00, 4001 6401, 0.022 14.02, 0.003 (0.03. 1100] cosoy
Figure 1: The distributions of gradient conflicts (in terms of cos thÿ)) of the joint-training baseline and state-of-the-art gradient manipulation methods on wer---aashhnnniiiin benchmark.
and 0g are task-specific parameters. The goal is to find optimal parameters 0 = {Oah 0,, 0g. .0 to achieve high performance across all tasks. Formally, it aims to minimize multi-task objective:
g = arg min
Σ wi C, (0.h.)))))) 0:).
where ia are pre-defined or dynamically computed weights for different tasks. A popular choice to use the average loss (i.e., equal weights). However, optimizing the multi-task objective is difficult, and a known cause conflicting gradients.
3.2 CONFLICTING GRADIENTS
Let gi Va, (.shhh 0z) denote the gradient of task T; w.r.t. the shared parameters Osh (i.e., a vector
of the partial derivatives of Li w.r.t. 0sh) ) and g Vo, C. 0:) denote the gradient w.r.t. the task-specific parameters 0;. small change of 4sb the direction of negative gi lab + 0sh - agn with sufficiently small step size a. The effect of this change on the performance of another task T measured by:
AC, (0.h ag,, 0,) - (Psh (itah.0) 0,) = aai g, + o(a).
(2)
where the second equality is obtained by first order Taylor approximation. Likewise, the effect of small update of 6sh in the direction of the negative gradient of task T, (i.e., -g,) on the performance of task Ti is AL; = -0 ag. gi + o()). Notably, the model update for task T; considered have a negative effect on task T, when g: gj O, since increases the loss of task TT. and vice versa.
formal definition of conflicting gradients is given as follows Yu et al. 2020).
Definition (Conflicting Gradients). The gradients gi and g + j) said to be conflicting with each other if cos Φi < 0, where Φiş is the angle between gi and gj-
As shown in Yu et (020), conflicts in gradient pose serious challenges for optimizing the multi- task objective (Eq, 1p. Using the average gradient (i.e., + Σ g:) for gradient decent may hurt the performance of individual tasks, especially when there is large difference in gradient magnitudes, which will make the optimizer struggle to converge to desirable solution.
3.3 GRADIENT SURGERY CANNOT EFFECTIVELY REDUCE CONFLICTING GRADIENTS
To mitigate the influence of confficting gradients, several methods (Yu et al. 2020 Chen al. 2020 Liu et al. 2021a) have been proposed to perform "gradient surgery". Instead of following the average gradient direction, they alter conflicting gradients based on some criteria and use the modified gradients for model update. We conduct pilot study to investigate whether gradient ma- nipulation can effectively reduce the occurrence of conflicting gradients. For each training iteration, we first calculate the task gradients of all tasks w.r.t. the shared parameters (i.e., for any task i) and compute the conflict angle between any two task gradients g: and g, in terms of coseij. We then count and draw the distribution of coso1; in all training iterations. We provide the statistics of the joint-training baseline (i.e., training all tasks jointly with equal loss weights and all parameters shared) and several state-of-the-a gradient manipulation methods including GradDrop (Chen al. 2020), PCGrad (Yu et al.) 2020), CAGrad (Liu et al.) 2021a), and MGDA (Sener & Koltun 2018) on
(Lin al. 2019). CityScapes, NYUv2, and PASCAL-Contes datasets.

---

Published as a conference paper at ICLR 2023
Tumad tadk spocilie
2ald
(a) Joint-train
(b) PCGrad
(c) Recon
(d) Recon
Figure 2: Illustration of the differences between trantnrrainigg gradient manipulation, and our ap- proach. (a) In joint-training, the update vector (in green) is the average gradient 2 (g: + gj). Due to the confiict between gi and gj. the update vector dominated by gi (in red). (b) PCGrad (Yu er al. 2020) projects each gradient onto the normal plane of the other one and uses the average of the projected gradients (indicated by dashed grey arrows) as the update vector (in green). As such, the update vector is less dominated by g:- (c) Our approach Recon finds the parameters contributing most (c.g., 0g) to gradient confficts and turns them into task specific ones. In effect, it performs an orthographicccoordinatt projection of conflicting gradients to the space of the rest parameters (e.g., 8 and 02) such that the projected gradients g, fix and fix are better aligned. (d) Illustration of Recon turning shared layer with high conflict score task-sp layers.
The results are provided in Fig. [ Fig. Fig 6 that gradient manipulation methods can only slightly reduce the occurrence of conflicting gradients (compared to coint-training! some cases, and in some other cases they even increase
Fig.
Table 6 and Tables 1 It can be seen
OUR APPROACH: REDUCING CONFLICTING GRADIENTS FROM THE RoOT
Our pilot study shows that adjusting gradients for model update cannot effectively prevent the currence of conflicting gradients in MTI., which suggests that the root causes of this phenomenon may be closely related to the nature of different tasks and the way how model parameters shared among them. Therefore, to mitigate task conflicts for MTL, in this paper, we take different ap- proach reduce the occurrence of conflicting gradients from the root.
4.1 RECON: REMOVING LAYER-WEL WISE CONFLICTING GRADIENTS
Our approach is extremely simple and intuitive. We first identify the shared network layers where conflicts occur most frequently and then turn them into task task-specific parameters. Suppose the shared model parameters lzu, are composed of n layers, i.e., 4a = {0!h) Jk=1- where ((k) is the 1thh shared layer. Let g! denote the gradient of task TO w.r.t. the tth shared layer 0(A)) i.e. g is vector of the partial derivatives of L, w.r.t. the parameters of g(A) Let di denote the angle between g!!))
(k) We define layer-wise conflicting gradients and S-conflict score follows.
and g)
Definition (Layer-wise Conflicting Gradients). The gradients g (k) and g) (i + j) are said be conflicting with each other if cos d O.
Definition 3 (S-Conflict Score). For any -1 < S O, the S-conflict score for the kth shared layer the number of different pairs (i.j)(i # s.t. cos o S. denoted (k)
indicates the severity of conflicts, and setting S smaller means we care about cases of more severe conflicts. The S-conflict score (k) indicates the occurrence of conflicting gradients at severity level for the tt shared layer. If (k) = (?). it means that for any two different tasks, there a conflict in their gradients w.rt. the tth shared layer. By computing S-conflict scores, we can identify the shared layers where conflicts occur most frequently.
We describe our method Recon in Algorithm compute S-conflict scores for each shared layer (k)) in every iteration, denoted by {8 )1=1- Then,
First, we train the network for iterations and [
4

---

Published as a conference paper at ICLR 2023
Algorithm 1; Recon: Removing Layer-wise Conflicting Gradients
Input:
Model parameters 0, learning rate a, a set of tasks (Ti) number of iterations / for computing conflict scores, conflict severity level S, number of selected layers K. // Train the network and compute conflict scores for all layers for iteration i= 2, / for i 1.2...... T' do
Compute the gradients of task T, w.r.t. all shared layers, i.e., {g) (x))) }k=1
end
Calculate the S-conflict scores for all shared layers in the current iteration, i.c., {s, in Update 0 with toint-training any gradient manipulation method;
1k=
end
// Set layers with top conflict scores task-specific For each layer k, calculate the sum of S-conflict scores in all iterations, i.e., s(A) Select the top K layers with highest g(x) and set them spec-fie;
EV=1 8; (K)
// Train the modifies network from scratch for iteration == 2, do
Update 0 with joint-training any gradient manipulation method; end
Output: Model parameters 0.
8 (k) and find the layers with highest (k) scores. Next, we these layers to be task-specific and train the modified network from scratch. We demonstrate the effectiveness of Recon by theoretical analysis in Sec. 4.2 and comprehensive ex- periments in Sec.5l The results show that Recon can effectively reduce the occurrence of conflicting gradients in the remaining shared layers and lead to substantial improvements over state-of-the-art
we sum up the scores in all iterations, i.e.,
4.2 THEORETICAL ANALYSIS
Here, we provide a theoretical analysis of Recon. Let 0sh = {0ax J where 0ix are the remaining shared parameters, and 0ef are those that will be turned to task-specific parameters off oof ,.. 0gf- Notice that oet ,5f,.... ... 04 will all be initialized with get Therefore, after applying Recon, the model parameters are 0, = (Oax out..., 04, 01, 0,2 J. An one-step gradient update of 0, is:
......,,,,
@fx gfix
wag. fix
0 off ag!
0: = 0:h ag
T
where my are weight parameters, g! V Li. g different methods such as joint-training, MGDA Senner & Koltun (2018). PCGrad Yu et al. (020))
Vor L. and g
Bnx L.- Notice that
and CAGrad Liu et al. (2021a) choose different w; dynamically.
Without applying Recon, the model parameters are 0 = {ftix she 6tS ent update of / given by
0%) An one-step gradi-
T
T
fix sh
gfx sh
O
w.g.
gef sh = get
a
w.g
8 = 8:0 ag!.
i= 1,....,,7 T. (4)
After the one-step updates, the loss functions with the updated parameters Ú, and Ò respectively are:
T
T
((O,) = Σ c. 1=1
(ftx (0f(
0; 0.t0) and, C()) = Σ C.
(0th get 0::))
(5)
1=1
where L; is the loss function of task T. Denote the set of indices of the layers turned task-specific
by P. then gel (0(#) },A IP. Assume that Σ We 1, then have the following theorem.
5

---

Published as a conference paper at ICLR 2023
Table I: Multi-task learning results Multi-FashionnMNISTT dataset. All experiments repeated over 3 random seeds and the mean values are reported. Am% denotes the average relative improve- ment of all tasks. #P denotes model size (MB). The grey cell color indicates that Recon improves the result of the base model. The best average result is marked in bold.
Steglerack
CeatDeop
Table Multi-task learning results on CelebA dataset. All experiments are repeated over 3 random seeds and the mean values are reported. Am% denotes the average relative improvement of all tasks. #P denotes model size (MB). The grey cell color indicates that Recon improves the result of the base model. The best average result is marked in bold.
Method
Single-tusk
Joins Irain
w/ Recon
CAGrad
w/ Recon
Graddrop
w/ Recom
PCGrra
w/ Recon
Average Error
8.38
8.33
8.22
8.31
8.23
8.33
8.20
8.64
8.36
Am% f
0.55
1,92
0.79
1,74
0.23
2.13
-3.14
0.24
#P
1706.03
43.26
68.03
43.26
68.03
43.26
68.03
43.26
68.03
Theorem 4.1. Assume that C is differentiable and for any two different tasks Tr and Tj. it satisfies
cos Oij lBB
(k)
(kc
|1B, (8) 1,
Vk e P
then for any sufficiently small learning rate a > 0,
[(0,) < L(0).
The theorem indicates that a single gradient update on the model parameters of Recon achieves lower loss than that on the original model parameters. The proof is provided in Appendix A
5
EXPERIMENTS
In this section, we conduct extensive experiments to evaluate our approach Recon for multi-task leamning and demonstrate its effectiveness, efficiency and generality.
5.1 EXPERIMENTAL SETUP
Datasets. We evaluate Recon on 4 multi-task datasets, namely Multi-Fashion+MNNER (Lin al., 2019). CityScapes (Cordts et al. 2016), NYUv2 (Couprie et al. 2013). PASCAL-Context (Mot- taghi et al. 2014). and CelebA (Liu et al. 2015). The tasks of each dataset are deseribed as follows. contains two image classification tasks. Each image consists of an item from FaShionMNIST and an item from MNIST. 2) CityScapes contains vision tasks: 7-class mantic segmentation and depth estimation. 3) NYUv2 contains 3 tasks: 13-class semantic segmen- tation, depth estimation and normal prediction. 4) PASCAL-Contex consists tasks: semantic segmentation, human parts segmentation and saliency estimation, surface normal estimation, and edge detection. 5) CelebA contains 40 binary classification tasks.
1) Muuli-FFahhon+MNNT
The baselines include 1) single-task learning ssingle-task): training all tasks indepen- dently; 2) joint-training (ointt-train) training all tasks together with equal loss weights and all parameters shared; 3) gradient manipulation methods: MGDA (Sener & Koltun, 2018). PCGrad Yuu
Baselines.
al. 2020), GradDrop (Chen al.) 2020). CAGrad (Liu al. 2021aj, RotoGrad (Javaloy Valera 2022): 4) branched architecture search methods: BMTAS (Bruggemann et al., 2020): 5) Architec-
ture design methods: Cross-Stitch (Misra et al. 2016). MMoE (Ma et al., 2018)) Following Liu al. (221a)), we implement Cross-Stitch based SegNet (Badrinarayanan
al. 2017). For fair comparison, all methods use same configurations and random seeds, We run all experiments 3 times with different random seeds. More experimental details provided in Appendix B
Relative task improvement. Following Maninis et al. (2019), we compute the relative task im- provement with respect to the single-task baseline for each task. Given a task T,, the relative task

---

Published as a conference paper at ICLR 2023
Table 3: Multi-task learning results CityScapes dataset. All experiments are repeated over 3 random seeds and the mean values are reported. Am% denotes the average rel- ative improvement of all tasks. #P denotes the model size (MB). The grey cell color indicates that Recon improves the result of the base model. The best average result is marked in bold.
10
-26
30
15
20
25
30
(a)
Segmentation
Depth
Method (Higher Better) Pix Ace
(Lower Better) Rel Err
Am% f
#P.
mloU
Abs Err
Single-task
74.36
93.22
0.0128
29.98
190.59
Cross-Stitch
74.05
93.17
0.0162
116.66
-79.04
190.59
RotoGrad
73.38
92.97
0.0147
82.31
-7.81
103.43
Joint-train
74.13
93.13
0.0166
116.00
-79.32
95.43
Recon
93.21
43.18
108.44
(b)
MGDA
70.74
92.19
0.0130
47.00
-16.22
95.43
w/ Recon
71.01
92.17
0.0129
33.41
-4.46
108.44
Figure 3: The performance of CAGrad combined with Recon the Multi-Fashion+MNIIST bench- mark with (a) different number of se- lected layers K (b) different severity
Graddrop
74.08
93.08
0.0173
115.79
-80.48
95.43
Recon
41.37
PCGrad
73.98
93.08
0.02
114.50
-78.39
95.43
w/ Recon
74.18
93.14
0.0136
46.02
-14.92
108.44
CAGrad
73.81
93.02
0.0153
88.29
-53.81
95.43
w/ Recon
93.10
0.0130
38.27
-7.38
108.44
value for computing conflict scores.
improvement is AMT, = x (M)) where M,, S, refer to metrics for the 7thh cri- terion obtained by objective model and single-task model respectively. 4 1 if a lower value for the criterion is better and 0 otherwise. The average relative task improvement is Am 7 Σ}=1 Amry.
5.2 COMPARISON WITH THE STATE-OF-THE-ART
Recon improves the performance of all base models. The main results on Paalii-Fashion-MNIITT and CelebA, Scapes, PASCAL-Comtext, and NYUv2, are presented in Table Table Table Table 4 and Table 5 respectively. (1) Compared to gradient manipulation methods, Recon consis- tently improves their performance in most evaluation metrics, and achieve comparable performance on the rest of evaluation metrics. (2) Compared with branched architecture search methods and architecture design methods, Recon can further improve the performance Besides, Recon combined with other gradient manipulation methods with small model size can achieve better results than branched architecture search methods with much bigger models.
BMTAS and MMoE.
Small increases in model parameters only changes a small portion of shared parameters to task-specific. As shown in Table 15 Re- 1.42%, 1.46%, 12.77%, 0.26%,
lead to good performance gains. Note that Recon
con increases the model size by 0.52% 57.25%. Recon 9.80% shared parameters to task-specific on Mulii-Fashonn-MNNISS CelebA, CityScapes, NYUv2 and PASCAL-Contex respectively. The results suggest that the gradient conflicts in a small portion (less than 13%) of shared parameters impede the training of the model for multi-task learning.
Recon is compatible with various neural network architectures. We use ResNet18 Multi- Mashion+MNISTT. SegNet (Badrinarayanan et al., 2017) ) on CityScapes, MTAN (Liu et al. 2019) on NYUv2, and MobileNetV2 (Sandler et al. 2018 on PASCAL-Comtext. Recon improves the per- formance of baselines with different neural network architectures, including the architecture search method BMTAS (Bruggemann et 2020) which finds tree-like structure for multi-task learning-
Only one search of conflict layers is needed for the same network architecture. An interesting observation from our experiments is that network architecture seems to be the deciding factor for the conflict layers found by Recon. With the same network architecture (e.g., ResNet18), the found conflict layers are quite consistent w.r.t. (1) different training stages (e.g., the first 25% iterations,
the middle or last ones) (see Table 12 and Table 13 and discussion in Appendix C (2) different
MTL methods (c.g., joint-training or gradient manipulation methods) (see Table [4 and discussion in Appendix C and (3) different datasets (see Table and T6 discussion in Appendix C).

---

Published as a conference paper at ICLR 2023
Table 4: Multi-task learning results PASCAL-Context dataset with 4-task setting. All exper- iments are repeated over 3 random seeds and the mean values are reported. Am% denotes the average relative improvement of all tasks. #P denotes the model size (MB). The grey cell color indicates Recon improves the result of the base model. The best average result is marked in bold.
SemSeg
PartSeg
saliency
Surface Normal
Angle Distance
Within 1" (Higher Better) 11.25
Method
(Higher Better)
(Lower Better)
(Higher Better)
(Lower Better)
Am%
WP.
mloU
Pix Acc
mloU
Pix Ace
mloU
Mean
Median
22.5
Single-task
65.00
90.53
59.59
92.61
65.61
14.55
12.36
46.51
81.29
30.09
Joint-train
64.06
90.45
57.91
92.17
62.71
16.40
14.23
39.38
75.93
-4.82
8.04
w/ Recon
64.73
90.50
59.00
92.44
66.17
14.99
12.68
44.82
80.11
.0.66
10.20
MGDA
86.62
77.82
w/ Recon
55.82
87.73
56,31
91.67
64.91
15.12
12.88
44,36
79.81
-4.7 -4.14
10.20
POGrad
63.91
90.45
58.01
19
63.09
16.34
14.19
39.62
76.06
-4.59
8.04
w/ Recon
65.02
90:45
59.22
92:46
66. 14
14.95
12.73
44.96
80.22
-0.35
10.20
Graddrop
64.14
90.34
57.62
92.12
62.64
16.46
14.28
39.29
75.71
-5.00
8.04
Recon
64.48
90:45
59.08
66.23
14.94
12.72
45.03
80.25
-0.63
10.20
CAGrad
63.37
90.17
57.49
92.07
64.16
16.30
14.12
39.80
76.23
-4.37
8.04
Recon
90:40
59.27
92.47
65.67
12.71
80.33
10.20
BMTAS
64.89
90:44
58.87
92.
63.42
13.66
13.44
42.29
78.14
-2.89
15.18
Recon
64.78
90:46
59.96
92.58
65.96
14.74
12:57
45.62
80.84
-0.19
16.83
Table 5: Multi-tat task learning results on NYUv2 dataset with MTAN as backbone. All experiments are repeated over 3 random seeds and the mean values are reported. Am% denotes the average relative improvement of all tasks. #P denotes the model size (MB). The grey cell color indicates that Recon improves the result of the base model. The best average result is marked in bold.
Segmentation
Depth
Surface Normal
Angle Distance (Lower Better)
Within 1 (Higher Better)
Method
(Higher Better)
(Lower Better)
Am% f
#P.
mloU
Pix Acc
Abs Err
Rel Er
Mean
Median
11.25
22.5
30
Single-task
38.67
64.27
0.2788
24.87
18.99
30.43
57.81
285.88
Cmss-Stitch
40.45
66.15
0.5051
0.2134
27.58
23.00
24.69
49.47
62.36
4.16
285.88
Joint-train
39.48
65.23
0.5491
0.2235
27.87
23.76
22.68
47.91
61.58
0.75
168.72
w/ Recon
39.54
65.20
0.5312
0,2234
26.55
21.40
26.53
52.60
65.31
4.14
169.59
MGDA 29.28
60.30
0.6027
0,2515
24.89
19.32
29.85
57.18
69.38
-2.26
168.72
w/ Recon
32.82
61,26
0.5884
0,2295
25.17
19.72
28.18
56.49
68.96
0.53
169.59
Graddrop
38.70
64.97
0.5565
0.2333
27.41
23.00
23.79
49.45
62.87
0.49
168.72
w/ Recon
40.14
66.08
0.5265
0,2241
26.51
21.45
26.51
52.48
65.26
4.67
169.59
PCGrad
38.55
65.07
0.54
0.23
26.90
22.05
24.98
51.36
64.41
2.02
168.72
w/ Recon
38.61
65.48
0.5350
0.2271
26.31
21.11
26.90
53.21
65.95
3.87
169.59
CAGrud w/ Recon
39.89
66.47
0.5496
0.2281
26.36
21.47
25.50
52.68
65.90
3.74
168.72
39.92
66.07
0.5320
0.2200
25.80
20.59
27.60
54.31
67.05
5,80
169.59
Hence, in our experiments, we only search for the conflict layers once with the joint-training baseline in the first 25% training iterations and modify the network improve various methods the same dataset. We also find that the conflict layers found on one dataset can be used to modify the network to be directly applied on another dataset to gain performance improvement.
5.3 ABLATION STUDY AND ANALYSIS
Recon greatly reduces the occurrence of confticting gradients. In Fig. Fndd Table we compare the distribution of cos diy before and after applying Recon on Multi- Pashion-MNiSIST (the results on other datasets are provided in Appendix C). It can be seen that Recon greatly reduces the numbers of gradient pairs with severe conflicts (cos dij E (-0.01, 1D) by at least 67% and up to 79% when compared with foint-training, while gradient manipulation methods only slightly reduce the percentage and some even increases it. Similar observations can be made from Tables
Randomly selecting conflict layers does not work. To show that the performance gain of Re- con comes from selecting the layers with most severe conflicts instead of merely increasing model parameters, we further compare Recon with the following two baselines. RSL.: randomly selecting
8

---

Published as a conference paper at ICLR 2023
Sabke
KO
Keed
Figure 4: The distribution of gradient conflicts (in terms of cos oxy) of baselines and baselines with Recon on Maltii-FashoonnMNNN dataset.
Table 6: The distribution of gradient conflicts (in terms of cos ).y)) w.n.t the shared parameters on Multi-Fashion+MNIST dataset. "Reduction" means the percentage of conflicting gradients in the interval of (-0.01, - 1.0] reduced by the model compared with joint-training. The grey cell color indicates Recon greatly reduces the conflicting gradients (more than 50%). In contrast, gradient manipulation methods only slightly decrease their occurrence, and some method even increases it.
c06-9%
Jonn trann
w/ RSL..
wi RSP
Reccn
MGDD
w/ Recon
Gracdrop
w/ Recoo
PCCead
Recos
C.Mitdd
Recon
58.53
57.75
(0. 0.01]
22:35
37.57
52.36
38.76
2.87
10,00.
-1.00
0.25
Reduction
52.94
80.13
7,80
7,01
67.20
72.21
0.05
75:32
Table 7: Comparison of Recon with RSL and RSP. PD: performance drop compared to Recon.
CAGirad
POGira
Seed
Recon
Task2
AP.
Acct
PD
Asc?
PD
Acct
PD
Acct
PD
6
577 97.061
11.68
64.39
25.26
73.02
97.43
0,87
65.57
24.31
73,02
97.11
1.18
87.61
2.04
83.63
94.92
3,39
87.31
2.46
83.63
76.33
96.93
58.16
97.81
0.47
58.28
1.36
51.96
97.63
0.68
88.55
1.22
51.96
2
$1.18
17.10
76.55
13.09
47,50
$8:71
9.59
$4.51
5.27
47.50
43.42
89.77
43,42
same number of layers as Recon and set them task-specific. RSP: randomly selecting similar amount of parameters as Recon and set them task-specific. The results in Table / show that both RSL and RSP lead to significant performance drops, which verifies the effectiveness of the selection strategy of Recon. We compare Recon with the baselines that selects the first or last K layers in Appendix C
Ablation study on hyperparameters. We study the influence of the conflict severity S and the number of selected layers К on the performance of CAGrad w/ Recon on Multi-Fashion+-MNN As shown in Fig. small K leads to a significant performance drop, which indicates that there still some shared network layers suffering from severe gradient conflicts, while a large K will not lead to further performance improvement since severe conflicts have been resolved. For the conflict severity S, we find that a high value of S (e.g., 0.0) leads to performance drops since it includes many gradient pairs with small conflicts, while some of them are helpful for learning common structures and should not be removed. In the meantime, a too small S (c.g., -0.15) also leads to performance degradation because it ignores too many gradient pairs with large conflicts, which may be detrimental to learning. While K and S are sensitive, we may only need to tune them once for given network architecture, as discussed in Sec. 5.2
CONCLUSION
We have proposed a very simple yet effective approach, namely Recon, to reduce the occurrence of conflicting gradients for multi-task learning. By considering layer-wise gradient conflicts and iden- tifying the shared layers with severe conflicts and setting them task-specific, Recon can significantly reduce the occurrence of severe conflicting gradients and boost the performance of existing meth- ods with only a reasonable increase in model parameters. We have demonstrated the effectiveness, efficiency, and generality of Recon via extensive experiments and analysis.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors would like thank Lingzi Jin for checking the proof of Theorem A.1 and the anony- mous reviewers for their insightful and helpful comments.
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions pattern analysis and machine intelligence, 39(12):2481-2495, 2017.
Felix JS Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel Alexander, and Jorge Cardoso. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolttion kernels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1385-1394, 2019.
David Bruggemann, Menelaos Kanakis, Stamatios Georgoulis, and Luc Van Gool. Automated search for resource-efficient branched multi-task networks. British Machine Vision Conference (BMVC), 2020.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pp. 794-803. PMLR, 2018.
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039-2050. 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2016.
Camille Couprie, Clément Farabet, Laurent Najman, and Yann LeCun. Indoor semantic segmenta- tion using depth information. CoRR, abs/1301 13572, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 248 255. Ieee, 2009.
Carlo D' Eramoo, Davide Tateo, Andrea Bonarini, Marcello Restelli, Jan Peters, et al. Sharing knowl- edge in multi-task deep reinforcement learning. International Conference on Learning Repre- sentations, pp. 1-11 1. OpenReview. net, 2020.
Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identify- ing task groupings for multi-task learning. Advances in Neural Information Processing Systems, 14277503-27515 2021.
Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille. Nddr-enn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3205-3214, 2019.
Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 1154-11552, 2020.
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task priori- tization for multitask learning. In Proceedings of the European conference computer vision (ECCV), pp. 270-287, 2018.
Pengsheng Guo, Chen-V Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference Machine Learning, pp. 3854-3863, PMLR, 2020.
10

---

Published as a conference paper at ICLR 2023
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many task model: Growing a neural network for multiple nlp tasks. Empirical Methods in Natural Language Processing (EMNLP), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 770-778, 2016.
Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In International Conference on Learning Representations, 2022.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 7482-7491, 2018.
lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 6129-6138, 2017.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. Advances in Neural Information Processing Systems, 32, 2019.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34: 18778-1188,80, 2021a.
Liyang Liu, Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Repre- sentations, 2021b.
Shikun Liu, Edward Johns, and Andrew Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 1871-1880, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Philip S Yu. Learning multiple tasks with multilinear relationship networks. Advances in Neural Information Processing Systems, 30, 2017.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relation- ships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1930-1939, 2018.
Kevis-Koking Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos. Attentive single-tasking of multi- ple tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 1851-1860, 2019.
Elliot Meyerson and Risto Mikkuulanen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. International Conference on Learning Representations, 2018.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3994 4003, 2016.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmen- tation in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 891-898, 2014.
Emilio Parisotto, Lei Jimmy Ba, and Ruslan Salakbutdinov. Actor-mimice Deep multitask and transfer reinforcement learning. In International Conference on Learning Representations, 2016.

---

Published as a conference paper at ICLR 2023
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In International Conference on Learning Represen- tations, 2018.
Scbastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:/706.05096, 2017.
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi-task ar- chitecture learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4822-4829, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 4510-4520, 2018.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in Neural Information Processing Systems, 31, 2018.
Jiayi Shen, Xiantong Zhen, Marcel Worring, and Ling Shao. Variational multi-task learning with gumhel-sofftax priors. Advances Neural Information Processing Systems, 34:21031-21042, 2021.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pp. 9120-9132. PMLR, 2020.
Sebastian Thrun and Joseph O'Sullivan.
Discovering structure in multiple learning tasks: The to algorithm. In International Conference on Machine Learning, volume 96, pp. 489-497, 1996.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey, IEEE transactions pattem analysis and machine intelligence, 2021.
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems, 33:4767-4777, 2020.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances Neural Information Processing Systems, 33:5824-588 5836, 2020.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3712-3722, 2018
12

---

Published as a conference paper at ICLR 2023
A PROOF OF THEOREM A.1
Theorem A.1. Assume that differentiable and for any two different tasks Ti and T5. it satisfies
cos @ (k) lgg
(2)
< ||g) (8) I.
vk
(8)
then for any sufficiently small learning rate 0,
[(,,) < C()).
(9)
Proof. We consider the first order Taylor approximation of Lj. For normal update, we have
Ls 60xx dct 0/-)
=L;
(0gh dct 0!:) + (f ( T fix
gi
(10)
(0sh e gi (0
p g + o(a).
(11)
For Recon update, we have
Ls (8xx 8 010)
=L,
(0th gct 0!0)
+
(ffxx
( 18
(12)
(0s) g
gi
of
(0
0 gi + o(a).
(13)
The difference between the two loss functions after the update
L (8xx 0. 0.x) C. ( (B9)s @ct. 03-)) ( 0ch
T
gi
+
+
(14)
T
T
a
gi
W
of
gi +
(15)
j=1
T
Wij
(g. - 8gth
B, + +(aa)
(16)
j=1
a Σ
Wy (1g
oft of + o(aa).
(17)
Assume, without loss of generality, that Ilg.f
+ 0, then
|
1|2
g)
cfT
cf
Σ
g:
(k)
g: Bj
(R) (8)
(18)
kEP
Σ
g:
(k)
(1
8i
(k)
(k)
8)
(k)
(19)
kEP
0.
(20)
Hence, the above difference is negative, if a is sufficiently small. As such, the difference between the multi-task loss functions is also negative, if a is sufficiently small.
((0))- - C(O) Σ
L. @ax Oct, .0,0))
--E, C, (ōgh Oct 010) <
(21)
B EXPERIMENTAL SETUP
B. MULTT-FASHION-MNIST
Model. We adopt ResNet18 (He et al., 2016) without pre-training the backbone and modify the dimension of the output features to 100 for the last linear layer. For the task-specific heads, we define two linear layers followed by ReL.U function.
13

---

Published as a conference paper at ICLR 2023
Tasks, losses, and metrics. Each task is classification problem with 10 classes and the cross-entropy loss as the classification loss. For evaluation, we use the classification accuracy as the metric for each task.
Model hype ayperparameters. We train the model for 120 epochs with the batch size of 256. We adopt SGD with initial learning rate of 0.1 and decay the learning rate by 0.1 at the 60th and 90th epoch.
Baseline hyperparameters. For CAGrad, we set a = 0.2. For BMTAS, we set the resource loss weight 1.0, and we search the architecture for 100 epochs. For RotoGrad, which is equal to the dimension of shared features and set the learning rate of rotation parameters learning rate of the neural networks. For MMoE, the initial learning of expert networks and gates are 0.1 and le - 3 respectively.
set Rg 100
Recon hy perparameters. We use CAGrad train the model for 30 epochs and compute the conflict score of each shared layer. We set S = -0.1 for computing the scores. We select 25 layers with the highest conflict scores and turn them into task specifec layers.
B.2 CITYSCAPES
Model. We adopt SegNet (Badrinarayanan et al. (017) as the backbone where the decoder is split into two convolutional heads.
Model hyperparameters.
train the model for 200 epochs with batch size of 8. We adopt Adam with an initial learning rate of 5e 5 and decay the learning rate by 0.5 at the 100th epoch.
Baselines hyperparameters. For CAGrad, we set 0.2. For RotoGrad, we set Rg 1024 and set the learning rate of rotation parameters 10 times less than the learning rate of the neural networks.
Recon hyperparameters. We use joint-train to train the model for 40 epochs and compute the conflict score of each shared layer. We set S = 0.0 for computing the scores. We select 39 layers with the highest conflict scores and turn them into task-specific layers.
B.3 NYUv2
Model. We adopt MTAN (Liu al. 2019 the SegNet combined with task-specific attention modules on the encoder.
Model hyperparameters. We train the model for 200 epochs with the batch size of 2. We adopt Adam with an initial learning rate of le - 4 and decay the learning rate by 0.5 at the th epoch.
Baseline hyperparameters. For CAGrad, we set a 0.4 similar with Liu al. (021a).
Recon hyperparameters. We use joint-train to train the model for 40 epochs and compute the conflict score of each shared layer. We S -0.02 for computing the scores. We select 22 layers with the highest conflict scores and turn them into task-specift layers.
B.4 PASCAL-CONTENT
Model. Following Bruggemann et al. (2020), we employ MobileNetv2 Sandler et al. (2018) as the backbone with a reduced design of the ASPP module (R-ASPP) (Sandler et al. 2018). pre-train the model ImageNet (Deng al. 2009).
Model hy hyperparameters. We train the model for 130 epochs with the batch size of 6. We adopt Adam with an initial learning rate of le 4 and decay the learning rate by 0.1 the 70th and th0th epoch.
Baselines hyperparameters. For CAGrad, we weight to 0.1, and we search the architecture for 130 epochs.
0.1. For BMTAS, set the resoure loss
Recon hy pyerpaaraees We joint-train to train the model for 40 epochs and compute the conflict score of each shared layer. We set S = -0.02 for computing the scores. We select 85 layers with the highest conflict scores and turn them into task-specific layers.
14

---

Published as a conference paper at ICLR 2023
B.5 CELEBA
Model. Following Sener & Koltun 2018), we use ResNer18 (e et al. 2016) as the backbone network. We pre-tram the model on ImageNet (Deng et al. 2009).
Model hyperparameters. We train the model for 5 epochs. We adopt Adam with an initial learning rate 5e and decay the learning rate by 0.5 the 3 epoch.
Baselines hyperparameters. For CAGrad, we set a = 0.1.
Recon hyperparameters. We use joint-train train the model for 2 epochs and compute the conflict score of each shared layer. We set S = -0.05. We select 25 layers with the highest conflict scores and turn them into task-specific layers.
C ADDITIONAL ABLATION STUDY
The distribution of gradient conflicts. In addition to the statistics on Mulii-Fashion-MNNITT, we further show the distributions gradient conflicts of various baselines CityScapes, NYUv2, and PASCAL-Context in Fig 5 Fig [ and Fig 7 respectively. We compare the distributions with those
of baselines w/ Recon on the three datasets Fig. Fig. and Fig. 10 respectively. The detailed
statistics are provided in Tables 10
Graddrop
PCGrad
0.20
Probability
0.15
9.10
0.00
[1.06.0.08
(0.08.0.06]
(0.06,0.04)
(0.04.0.52]
(0.02.0.00:
(0.00, 0.02] 00,02, 0.04] 10.04, 0.06] (0.06, 0.08) 10.08. 1.008. cosO:
Figure 5: The distributions of gradient conflicts (in terms of Φit) of the joint-training baseline and state-of-the-art gradient manipulation methods on CityScapes dataset.
Graddrop
PCGrad
0.4
Probability
0.3
0.2
0.1
[1.00.0.08]
[0.08.0.06]
(0.06.0.04)
10.04.0.02)
(0.02.0.00)
(0.00, 0.02| (0.02, 0.04) 10.04 0.00] (0.06. 0.08] (0.05, 1.00]
Figure 6: The distributions of gradient conflicts (in terms of cos di;)) of the joint-training baseline and state-of-the-ee gradient manipulation methods on NYUv2 dataset.
Joint-train
Graddrop
PCGrad
CAGrad
MGDA
probablity
0.20
0.15
0.05
0.88
(6.08/0.06]
(0.06.0.04)
10.64|p.20]
(0.00 c.00) cosé,
(6.00, -n.021 (4.02 -0.64 -6.044 (0.06, -2.08) 1-0.88 (+0.08.-1.00) -1.00]
Figure 7: The distributions of gradient conflicts (in terms of cos diy)) of the joint-training baseline and state-of-the-art gradient manipulation methods on PASCAL-Contex dataset.
15

---

Published as a conference paper at ICLR 2023
mah Ia Direma mht
MSBA
Grasimpp
PGönd Poone Reain
Cidie Choness Rocon
Probabiity
Recon
24
Seasemp Rectn
Probaabiity
65
25
64
AA
8.4
6.0
6.3
6:
codn man
coo 1.
colat 4.900
1.00
cosg..
co54y
000
Figure 8: The distribution of gradient conflicts (in terms cos Dsy) w.r.t. the shared parameters on CityScapes. RSL: randomly selecting same number of layers Recon and them specific. RSP: randomly selecting similar amount of parameters as Recon and set them task-specific.
6.7
MSBS
Seatomp
37
PoGme
37
Chöne
Orobabaltt tt
1.
Graddrop Rlccn
PCSiud
2.
6.5
25
25
Recen
1A
2A
86.
13
23
Eaz
12
6.
61
24
01
taaz aad
on ctep
eam dayy
cum Jaa
au
daad
pusc
coog
coopy
Figure 9: The distribution of gradient conflicts (in terms of cos diy) of baselines and baselines with Recon NYUv2. RSL: randomly selecting same number of layers Recon and set them task- specific. RSP: randomly selecting similar amount of parameters as Recon and set them task saek-spec.fic.
PCSuxd
CAOuu
Wi AZL
Pretekiity
K.5
15
Preteatiitty
25
Pretabliis
45
DIEE NY Reces
42
stodd
nosee s1a0
stued
dlod sboet
coust ause
case acedd
Labse aust
Lase, sod
cold
Local
Figure 10: with Recon on PASCAL-Context. RSL: randomly selecting same number of layers Recon and set them task-specific. RSP: randomly selecting similar amount of parameters as Recon and set them task-specific.
The distribution gradient conflicts (in terms of cos Φity of baselines and baselines
Table 8: The distribution of gradient conflicts (in terms of cos drj)) w.r.t. the shared parameters on CityScapes dataset. "Reduction" means the percentage of conflicting gradients the interval of (-0.02, - 1.0] reduced by the model compared with joint-training. The grey cell color indicates Recon greatly reduces the conflicting gradients (more than 50%). In contrast, gradient manipulation methods only moderately decrease their occurrence (MGDA deceases it by 22%), and methods even increase it.
Groddrap
PCCiod
CAGiod
[1.0.
59.55
53.16
58.29
73.62
63.9
78.27
59.56
73.82
59.85
74.52
50.79
74.5t
(0, .0.02]
10.14
9.01
10.77
20.13
12.51
12.54
9.61
19.75
9.58
19.43
11.13
19.77
(4.02
(4004. .0.06
6.45
5.09
6.48
0:34
5.39
2.23
6.49
1.05
6.24
0.96
6.05
0.89
(4006, .0.08
4.79
4.53
4.61
0.14
3.29
0.85
4.76
0.16
4.41
0.15
4.06
0.13
(0.88
Reduction (8)
-24.83
-2.11
79.41
22.11
69.70
-1.72
78.78
-0.89
80:05
7.36
81.22
16

---

Published as a conference paper at ICLR 2023
Table 9: The distribution of gradient conflicts (in terms of cos (i9) w.I.t the shared parameters on NYUv2 dataset. "Reduction" means the percentage of conflicting gradients in the interval of (-0.04, -1.0] reduced by the model compared with joint-training. The grey cell color indicates Recon greatly reduces the conflicting gradients (more than 50%). In contrast, gradient manipulation methods only slightly decrease their occurrence, and some methods even increase
eas 0g
Joint-train
w/ RSL.
w/ RSP
w/ Recon
MGDA
w/ Recon
Groddrap
w/ Recon
PCGimd
w/ Racon
CAGind
w/ Recon
[1.0.00
61,96
52.61
59.70
73,99
61,28
74.08
62.93
75.35
63.25
75.54
61.95
74:49
(0, 0.02)
3.53
(4002 .0.44
3.63
3.60
3.41
7.07
2.77
7.21
3.70
6.71
3.62
6.65
3.30
6.96
(4004, .0.06
3.43
2.89
3.45
2:71
3.21
2.71
(4.06, 0..0)
3,30
2.94
1.03
1.25
3.05
(0.08, -100
24,05
33.31
27.37
0.76
27.53
0.87
22.92
0.70
23.20
0.90
24.88
0.63
Reduction
-31.06
84.35
3.34
85.47
3.37
-1.98
85.76
Table 10: The distribution of gradient conflicts (in terms of cos Øij) w.r.t the shared parameters on PASCAL-Context dataset. "Reduction" means the percentage of conflicting gradients in the interval of (-0.02, - 1.00 reduced by the model compared with joint-training. The grey cell color indicates Recon greatly reduces the conflicting gradients (more than 50%). In contrast, gradient manipulation methods only slightly decrease their occurrence, and some methods even increase it.
Joint-train
w/ RSL.
RSP
Recon
MGDA
w/ Recon
Gruddrop
Rocon
PCGml
w/ Rocon
CAGinal
Recon
59.20
60.99
(0, -000]]
9.66
21.01
18.25
23.57
8.51
33:53
9.66
23.41
9.83
23.61
9:95
24.04
(-002. -0.04]
7,90
9.91
9.10
7.65
7.27
2.04
7,89
7.83
7.90
7.65
8.03
7.53
(4.04-4)) -0.06]
3.05
2.71
5.01
(-0.66 00.08]
4.16
1.32
1.79
1.07
4.35
0.17
4,21
1.12
4.13
1.10
4.23
1.04
11.16
1.30
10.97
10.88
Reduction (5)
46.41
41.31
57.21
98
-0.24
$5.90
0.07
Table 11: Multi-task learning results on Multi-Fashion-MMIIST dataset. LSK refers to turning the fist K layers into task-specific layers. FSK refers to turning the last K layers into task-specific layers. PD denotes the performance drop compared with Recon.
CAGrad
PCGrad
LSK
FSK
w/ Recon
Task
Task2
Task
Task2
#P.
#P.
Acct
PD
Acct
PD
Acct
PD
Acct
PD
97.63
0.66
89.14
0.50
84.17
97.63
0.65
88,98
84.17
98.21
0.07
89.15
0.50
48.90
98.19
0.09
89.51
0.13
48.90
d
98.28
89.65
0
43.42
98.30
0
89.77
43.42
Selecting the first K layers and the last K Layers as conflict layers does not work. To fur- ther support the conclusion that the selection of parameters with higher probability of conflicting gradients contributes most to the performance gain rather than the increase in model capacity. We compare Recon with two baselines: (1) Select the first K neural network layers and turn them into task-specific layers. (2) Select the last K neural network layers and turn them into task-specific layers. The multi-task learning results on the
MNIST benchmark are presented in Table IT] The results show that if we directly turn the top or the bottom of the neural network into task-specific parameters, it still will lead to performance degradation compared to Recon.
Recon finds similar layers in different training stages. Recon ranks the network layers according to the computed S-conflict scores. The ranking result can be represented as a layer permutation, denoted as and πt (I) is the position of layer l. The similarity between two rankings π, and πΙ can be measured as:
d( ( n
(1) k. (,()), [L|
(22)
where L denotes the set of neural network layers. In Table we measure the differences in rankings obtained in different training stages (e.g., in the first 25% iterations the second 25% iterations)
17

---

Published as a conference paper at ICLR 2023
Table 12:
The distance between the layer permutations (rankings) obtained in different training
stages on Multi-Fashion+MNNDD dataset. "Iter." denotes iterations.
Training Stage
1st 25% Iter.
2nd 25% Iter.
3rd 25% Iter. 4th 25% Iter.
All Iter.
1st 25% Iter.
0
2nd 25% Iter.
2,39
0
3rd 25% Iter.
1.85
2.14
0
4th 25% Iter.
1.95
2.24
0.68
All Iter.
1.36
1.95
0.82
0.97
Performance of the networks modified by Recon with conflict layers found in different training stages of joint-training CityScapes dataset. Am% denotes the average relative improve- ment of all tasks. #P denotes the model size (MB). The best result is marked in bold.
Table 13:
Segmentation
Depth
Model
(Higher Better)
(Lower Better)
Am
HP.
mloU
Pix Acc
Err
Rel Err
Single-task
74.36
93.22
0.0128
29.98
190.59
1st 25% Iterations
74.17
93.21
0.0136
43.18
-12663
108.439
2nd 25% Iterations
74,20
93.19
0.0135
42.45
-11.83
108.440
3rd 25%
Iterations
74.80
93.19
0.0136
41.34
-10.90
109.567
4th 25% Iterations
74.80
93.19
0.0136
41.34
-10.90
109.567
All Iterations
74.80
93.19
0.0136
41.34
-10.90
109.567
on Multi-Fashin Malli-PPashonnMNISTT by Eq. 22 The small distances (less than 2.4) indicate that the layers found in different training stages quite similar. In Table [3] compare the performance of the networks modified by Recon with conflict layers found in different training stages on CityScapes. It can be seen that the results of the last three rows are the same, which is because the layers found in the 3rd 25% iterations, 4th 25% iterations, and all iterations are exactly the same (the rankings may be slightly different though). The layers found in the later stages lead slightly better performance than those found in the early stages (i.e., 1st 25% iterations and 2nd 25% iterations), indicating the conflict scores in early iterations might be a little noisy. However, since the performance gaps are acceptably small, to save time,
the initial 25% training iterations to find conffict layers.
Table The distance between the layer permutations (rankings) obtained by Recon with different methods on Mulii-FashionMMNISTT dataset.
Method
Joint-train
CAGrad
PCGrad
Gradrop
MGDA
Joint-train
CAGrad
1.07
0
PCGrad
0.78
1.17
0
Gradrop 0.59
0.83
0.68
MGDA
1.71
1.32
1.90
1.56
Recon finds similar layers with different MTL methods. In Table 14 we measure the differences in layer permutations (rankings) obtained by Recon with different methods (e.g., CAGrad and PC- by Eq. 22 The small distances (less than 1.9) indicate that the
Grad) on Multi-Fashion layers found by Recon with different methods are quite similar. Therefore, in our experiments, only use joint-training to search for the conflict layers once, and directly apply the modified network to improve different gradient manipulation methods as shown in Tables
The conflict layers found by Recon with the same architecture are transferable between dif- ferent datasets. We conduct experiments with three different architectures: ResNet18, SegNet, and MTAN. (1) For Resnet18, we find that the layers found by Recon on CelebA and those found on Multi-Fashion+ are exactly the same. (2) For SegNet, find that 95% layers (38 out of 40)
18

---

Published as a conference paper at ICLR 2023
Multi-task learning results on NYUv2 dataset with SegNet as backbone. Recon" de- notes setting the layers found on CityScapes task-specific. Am% denotes the average relative improvement of all tasks. #P denotes the model size (MB). The grey cell color indicates that Recon or Recon" improves the result of the base model.
Table 15:
Segmentation
Depth
Surface Normal
Angle Distance
Within t
(Higher Better)
(Lower Better)
Method
(Lower Better)
(Higher Better)
Am?
mloU
Pix Ace
Abs Err
Rel
Mcan
Median
11.25
22.5
30
Single-task
38.67
64.27
0.6881
0,2788
24.8683
18.9919
30.43
57.81
69.7
285.88
Joint-train
38.62
65,36
0.5378
0,2273
29.92
25.82
20.79
44.29
57.36
-1.62
95.58
w/ Recon
40.68
66.12
0.5786
0.2558
26.72
21.41
26.58
52.58
65,20
2.15
139.59
Recon'
38.81
63.69
0.5637
0,2413
26.75
21.73
26.16
51.80
64.64
1.59
121.59
MGDA
25.71
57.72
0.6033
0.2358
24.53
18.65
31.22
58.46
70.21
-2.15
95.58
w/ Recon
36.64
62.36
0.5613
0.2255
24.66
18.66
31.30
58.47
70.16
5.37
139.59
Recon*
36.85
63.51
0.5760
0.2362
24.89
18.96
30.53
57.94
69.82
4.34
121.59
Graddrop
39.01
66.13
0,5462
0.2296
29.72
25.51
19.87
44,68
58.12
-1.52
95.58
w/ Recon
39.78
65.63
0.5460
0.2280
26.42
21.16
26.89
53.16
65.84
4.45
139.59
Recon"
39.97
65.71
0,2261
26.52
21.37
26.65
52.65
65.46
4.21
121.59
PCGrad 40.01
65.77
0,5349
0.2227
28.53
24.08
22.33
47.42
60.69
1,43
95.58
w/ Revon
40.03
65.92
0.5523
0,2384
26.24
20.89
27.30
53.66
66.25
4.19
139.59
Recon'
39.93
65.46
0.5494
0.2315
26.82
21.70
26.34
52.04
64.74
3.53
121.59
CAGrad
38.87
66.54
0.5331
0,2289
25.85
20.60
27.50
54.41
67.10
5.60
95.58
w/ Recon
40.68
66.12
0,5372
0.2266
25.44
19.87
28.96
56.00
68.28
6.99
139.59
Recon"
39.97
65.92
0.5298
0.2273
25.56
20.11
28.69
55.37
67.75
6.47
121.59
Table 16: Multi-task learning results on CityScapes dataset with MTAN as backbone. Recon de- notes setting the layers found NYUv2 to task-specific. Am% denotes the average relative improvement of all tasks. #P denotes the model size (MB). The grey cell color indicates that Recon Recon" improves the result of the base model.
Segmentation
Depth
Method
(Higher Better)
(Lower Better)
Am
#P.
mloU
Pix Acc
Abs Emr
Rel Err
Single-task
73.74
93.05
0.0129
27.71
190.58
Joint-train
75.35
93.55
0.0169
45.64
-23.26
157.19
w/ Recon
75.72
93.74
0.0130
40.90
-11.36
196.32
w/ Recon"
76.32
93.76
0.0132
46.40
-16.44
159.19
MGDA
70.46
91.75
0.0224
34.33
-26.02
157.19
w/ Recon
72.23
92.60
0.0122
26.93
1.37
196.32
w/ Recon
70.83
92.14
0.0125
25.69
1.31
159.19
Graddrop
75.19
93.53
0.0168
46.35
-23.90
157.19
w/ Recon
75.60
93.72
0.0127
38.55
-8.71
196.32
w/ Recon
76.49
93.82
0.0129
47.54
-16.81
159.19
PCGrad
75.64
93.54
0.02
43.53
-23.60
157.19
w/ Recon
75.89
93.71
0.0129
40.05
-10.55
196.32
w/ Recon"
76.24
93.69
0.0128
45.24
-14.66
159.19
CAGrad
75.26
93.50
0.0176
44.23
-2..40
157.19
Recon
75.65
93.71
0.0125
36.23
-6.15
196.32
w/ Recon
76.25
93.74
0.0123
40.05
-8.99
159.19
19

---

Published as a conference paper at ICLR 2023
found NYUv2 are identical to those found on CityScapes. On NYUv2, compare the perfor- mance of using confict layers found on NYUv2 (baselines w/ Recon) to that using conflict layers found CityScapes (i.e., baselines w/ Recon"), with attention), we find that 68% layers (17 out of 25) found on CityScapes are identical to those found NYUv2. On CityScapes, CityScapes (baselines w/ Recon) to that of using conflict layers found on NYUv2 (i.e., baselines w/ Recon"), shown in Table [6 The results show that the conflict layers found on one dataset can be used to modify the network to be directly used on another dataset to consistently improve the performance of various baselines, while searching for the conflict layers again on the new dataset may lead to better performance.
shown in Table D3 (3) For MTAN (SegNet
compare the performance of using conflict layers found
2.3
2.2
Recon
Joint-train
1.9
CAGrad
1.8
Graddrop
1.7
PCGrad
1.6
1.5
@12
1.1
1.0
0.9
0.8
0.7
05
04
03
02
0.1
0.0
4 6 10 12111116 22
262246622 32 48
Number of tasks
Figure 11: Comparison of running time (one iteration, excludes data fetching) on CelebA dataset.
Analysis running time. We evaluate how Recon scales with the number tasks on CelebA dataset, by comparing the running time of one iteration used by Recon in computing gradient conflict scores (the most consummonnmnng part of Recon) to that of the baselines. The results in Fig. show that Recon is fast other gradient manipulation methods such as CAGrad (Liu et al. 102 and Graddrop (Chen et al. 2020), but much slower than joint-training especially when the number of tasks is large, which natural since Recon needs to compute pariwise cosine similarity of task gradients. However, since Recon only needs to search for the conflict layers once for a given network architecture, discussed above, the running time is not problem.
20


Reviews:
[{"review_id": "vh97T2QG4Z", "reviewer": "Reviewer_Tqjk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles conflicting gradients in multi-task learning (MTL). Specifically, the idea is to identify the shared layers of the multi-task model that exhibit large degrees of conflicting gradients, and replace these layers with task-specific parameters. The reported experiments suggest that only a small number of additional task parameters are necessary to improve performance, and reduce the occurrence of conflicting gradients in the remaining shared layers.\n\nThe experiments cover four multi-task benchmarks: multi-fashion+MNIST, CityScapes, NYUv2, and PASCAL-Context. The main finding is that the proposed “Recon” MLT approach improves the performance of all base models. Notably, the performance increase can be obtained with only a small increase in the total number of parameters (1.42%, 12.77%, 0.26%, and 9.8%, respectively for each benchmark).\n\nFurther analysis is provided in the way of an ablation study, showing that Recon significantly reduces the occurrence of conflicting gradients. Additionally, a theoretical analysis is presented, showing that a single gradient update on the model parameters of Recon achieves lower loss than on the original model parameters.\n", "summary_of_the_review": "Overall, this paper describes a simple idea to improve MTL by introducing a relatively small number of task-specific parameters, in order to reduce the impact of gradient conflict. The experiments are fairly comprehensive.", "strength_and_weaknesses": "While the approach is straightforward, the simplicity and effectiveness of the proposed Recon method are appealing.\n\nA potential concern with resolving conflicting gradients at training time is that they may, in some cases, provide a regularisation benefit that leads to positive transfer on held-out data. I would have liked to see more discussion on this issue.\n\nThe experiments are limited in two respects: (1) they focus only on vision problems; and (2) three out of four multi-task settings involve semantic segmentation. I would have liked to see more diversity of benchmarks as well as benchmarks with larger numbers of tasks.\n\nA downside of the proposed approach is that the resulting model is not actually multi-task anymore, as it contains task-specific weights, which could be problematic in some settings for deployment.\n\nFinally, the hyperparameters (S, K) appear to be quite sensitive, which increases the cost of the approach as they require fine-tuning for the different domains / datasets.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "aRIaGNNhOHC", "reviewer": "Reviewer_DXpE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-task learning method to reduce conflict gradients during multi-task optimisation. The proposed method, name Recon is very simple in design and intuitive – it first computes the cosine similarity score for all combinations of task pairs in each shared layer, and sets this shared layer to be task-specific if its cosine similarity score is negative for multiple task pairs. As such, the proposed method can be combined with any gradient-based multi-task optimisation methods and can be applied on any multi-task architecture. The paper verifies this design on a range of multi-task benchmarks with different architectures and different optimisation methods.", "summary_of_the_review": "The paper proposes a simple multi-task optimisation method: to make task-shared layers to be task-specific if the gradient conflict is severe. The paper in general is very well written with comprehensive empirical and theoretical support. ", "strength_and_weaknesses": "Strength. \n+ The paper is very well written and clearly motivated, supported with a wide range of experiments.\n+ The proposed method is general in design and can be attached to any optimisation methods and any architecture.\n+  The proposed method can achieve smaller loss supported with theoretical analysis.\n\nLimitations.\n+ Scalability and Running Time. Clearly, one inevitable problem for the proposed method is to compute cosine similarity for all combination of task pairs which might be unscalable when the number of training tasks is large. Though I agree that the newly initialised task-specific parameters might be small compared to the total network parameters, but the running time might be much slower. It would be clearer if the authors could the list the increased running time for such method and how it scales with increasing number of tasks. One paragraph that related to this calming that “requires extra 25% of training iterations” is clearly not sufficient and definitely not fast considering the fact that we only 2 tasks.\n\n+ Conflict from Architecture Design. It’s very interesting to see that the authors found out that the conflict layers are consistent across multiple methods. This observation makes me wonder that the conflict itself was mainly due to the multi-task architecture design. As such, I am wondering whether these conflicts would reduce if we re-train the model from scratch after we set these conflict layers to be task-specific. If so, the running time would not be a problem since we only need to run this method once.  Just to be clear, the consistency is from different methods on the same dataset or different methods on different datasets as well?\n+ Severity criteria. Is this hyper-parameter needs to be tuned for each dataset as the paper lists the percentage of increased parameters vary quite a lot from different datasets, which will make the method a bit less general.\n+ Minor visualisation suggestion. Please add the attached method before (w/ Recon) or have a midrule to separate other methods. This would make the comparison much clearer.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IkixExqly_", "reviewer": "Reviewer_5G4Q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a new perspective on solving the conflicting gradients problem in MTL. It checks the conflicting level at layer scale, and turns the layers with high conflict scores into task-specific ones. Some theoretical analysis and experimental verifications are provided. ", "summary_of_the_review": "The paper is in general interesting. But in the current form, some parts of this paper are questionable or not clear. The authors need to clarify two main points raised in the review. \n\n(1) The claim on the effects  of gradient manipulation to conflicting gradients. \n\n(2) What does \"set these layers task-specific\" mean?\n\n(3) More experiments or discussions are needed to see whether the proposed applies to typical MTL structures such as MMoE.\n\nIf the concerns are well addressed, I will raise my score.", "strength_and_weaknesses": "Strengths:\n\n+ The perspective on identifying conflicting gradients at layer scale is interesting. The proposed method that turn layers with high conflicts to task-specific ones seems to work well.\n+ The overall structure of this paper is easy to follow.\n\nWeaknesses:\n+ I am not convinced by the claim that \"gradient manipulation can not reduce the occurrence of conflicting gradients\". In my practice, I find these techniques can alleviate the conflicting gradients. Your results are only based on one dataset. It's surely not enough. Moreover, how do you calculate the conflicting angles? I think Figure 1 and Section 3.3 need more in-depth explanations. Otherwise, this claim seems to be exaggerated.\n+ The key algorithmic design \"turn the layers into task-specific ones\" is hard to figure what its exact meaning. I struggled to understand this point very concretely. Some figures with network structures may help better illustrate this clearly. \n+ The applicability of this method to other commonly used MTL structure such as MMoE is not provided.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BPMTCi7TPI7", "reviewer": "Reviewer_2d79", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to mitigate the negative transfer, i.e., the conflicting gradients, of different tasks on the shared layers. After showing that the gradient surgery methods cannot reduce the occurrence of conflict by investigating the gradient angle distributions of different tasks, the authors propose to split those layers with the most conflicting gradients. \n\nSpecifically, the authors train the network for some initial steps to calculate an S score to indicate the gradient conflict, where for a specific layer, its S score is the number of tasks whose gradient angle is larger than a threshold. Then, the shared layer with a higher S score (i.e., higher gradients conflict) is split for T times for each of T tasks. Finally, the new network with split task-specific layers is continued to train until convergence. Theoretical analysis is also provided to guarantee that applying the proposed Recon reduces the training loss.\n\nThe experiments are conducted on Multi-Fashion+MNIST, CityScapes, NYUv2, and PASCAL-Context with ablations validating the promising performance.", "summary_of_the_review": "Overall I think this is a good paper with clear (empirical) problem analysis and concise solution. For those further possible improvements, please see the Weakness section.", "strength_and_weaknesses": "Strength:\n\n1. Important problem with the pilot study probing existing issues and the concise solution addressing them.\n2. The theoretical analysis is also carried out that guarantees a lower training loss.\n3. This paper is well-organized and easy to follow.\n\nWeaknesses:\n\n1. In order to avoid , there is another category of MTL methods aims to avoid the negative transfer/gradient conflict by designing proper network architectures. For example, [1-4] perform layerwise feature fusing between tasks, which do not consider splitting from a shared network, but starting with several individual networks to merge (fuse features). Can the authors discuss, and preferably compare with, those methods?\n2. Would it help to control the starting iteration step when calculating (i.e., summing up) the s-scores? In other words, currently s^(k) = \\sum_{i=1}^{I} s_i^(k), would it help if i does not start with 1 as the s-scores of the very initial iterations can be noisy?\n3. This could be a minor issue: there exists a much larger MTL dataset, i.e., Taskonomy [5], to validate the proposed method.\n\n[1] Misra et al., Cross-stitch networks for multi-task learning. CVPR 2016.\n\n[2] Gao et al., NDDR-CNN: Layerwise feature fusing in multi-task CNNs by neural discriminative dimensionality reduction. CVPR 2019.\n\n[3] Ruder et al., Latent multi-task architecture learning. AAAI 2019.\n\n[4] Gao et al., MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning. CVPR 2020.\n\n[5] Zamir et al., Taskonomy: Disentangling task transfer learning. CVPR 2019.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster