IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING

1Multi-task Curriculum Learning
Based on Gradient Similarity
Hiroaki Igarashi1

1

DENSO CORPORATIONTokyo, Japan

2

DENSO IT Laboratory
Tokyo, Japan

3

Tokyo Institute of Technology
Tokyo, Japanhiroaki.igarashi.j3m@jp.denso.com

Kenichi Yoneji1
kenichi.yoneji.j3v@jp.denso.com

Kohta Ishikawa2
ishikawa.kohta@core.d-itlab.co.jp

Rei Kawakami3∗
reikawa@sc.e.titech.ac.jp

Teppei Suzuki2
suzuki.teppei@core.d-itlab.co.jp

Shingo Yashima2
yashima.shingo@core.d-itlab.co.jp

Ikuro Sato2,3
sato.ikuro@core.d-itlab.co.jp

Abstract
Intensive studies on multi-task learning (MTL) with deep neural networks have showncases where both test error and computational cost can be reduced compared to singletask learning. However, several studies have argued that a naive implementation of MTL
often degrades test performance due to gradient conflict, in which task-wise gradients
have a negative inner product. These studies also invented ways to modify the gradientsand eliminate the conflict. One concern about these methods is that the obtained solution is no longer optimal for the original objective due to the modification. In this paper,
we propose a multi-task curriculum learning based on gradient similarity (MCLGS) to
mitigate the negative impact of gradient conflicts while retaining the original objective
toward the end of the training. We adopt a simple curriculum strategy that gives moreweights to minibatches exhibiting fewer gradient conflicts in the early stage of training.
We experimentally confirmed that MCLGS outperforms existing MTL methods, such as
MGDA, PCGrad, GradDrop, and CAGrad, on BDD100K and NYUv2 datasets.

1

Introduction

Robotics applications, such as autonomous driving (AD) and advanced driver-assistance systems (ADAS), require multiple perceptional tasks [5, 7, 28], for example, object detection© 2022. The copyright of this document resides with its authors.
It may be distributed unchanged freely in print or electronic forms.
∗ Work done while at DENSO IT Laboratory.

2

IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING(a) An overview of MCLGS(b) Weighting gradient in MCLGS

Figure 1: (a) An overview of MCLGS; g̃i and g̃ j denote the i-th and j-th task-wise gradientson a shared network among tasks (shown as pink blocks). In MCLGS, a pair of task-wise
gradients (g̃i , g̃ j ) are weighted by the weight wti, j (shown in purple) and used to update themodel parameters θ . (b) The design of the weight wti, j . This depends on the amount of gradient conflict measured by the cosine similarity between task-wise gradients and the training
step t. It is designed such that the weight becomes higher than 1 to encourage training if
the cosine similarity is high; otherwise, it will be lower than 1 to suppress training in theearly stage of training (top in (b)). As learning progresses, wti, j will not depend on cosine
similarity and will always be around 1, which is consistent with the naive MTL update rule(bottom in (b)).

and semantic segmentation. If these tasks are implemented as separate models, the system
will be complex and can involve redundant computations between each models. Multi-task
learning (MTL) [6, 26, 30], which shares a portion of the network between multiple tasks, isa possible solution to simplify the system and reduce the complexity.
In MTL, if the magnitude of task-wise gradients is unbalanced during training, the trainedmodel could be biased toward specific tasks. Therefore, several methods for balancing loss
function have been proposed [3, 12]. However, if only the balancing loss function is applied,each task may have adverse effects on the other in MTL. One possible cause of this is gradient conflict, in which the inner product between the gradients of tasks is negative. In thiscase, since the parameter updates of each task are oriented in different directions, conflicting
gradients sometimes lead to insufficient solution for each task.
Several studies [4, 16, 21, 29] have tackled this problem. For example, PCGrad [29]manipulated gradients such that the conflicting components were removed, and only theorthogonal components of each gradient were extracted and used for the update. Althoughsuch methods can remove gradient conflicts, a converged solution is no longer optimal for theoriginal objective due to gradient manipulation. Specifically, if the conflicting componentsof gradients have a large difference in magnitude, then these components contain significant
information. However, conventional methods simply discard this information by removing
these components.
In this paper, we propose a multi-task curriculum learning based on gradient similarity(MCLGS), which mitigates the negative impact of gradient conflicts between tasks. MCLGS
introduces a curriculum learning strategy [2] that removes hard samples in the early stage oftraining in multi-task learning. In single-task learning (STL), the difficulty of samples are
generally determined by how hard input is to classify. For MTL, we redefine the difficulty byamount of gradient conflicts. Thus, in MCLGS, samples which generates gradient conflicts,

IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING

3

are considered as hard to train in multi-task learning, and are downweighted in the earlystage of training. However, these samples are gradually included as training progresses.
Specifically, to validate our idea, we present a simple and somewhat heuristic function that
determines the weights for each gradient given the gradient similarity and the training step.
As shown in Figure 1, the function is designed such that the conflicting (aligned) gradientsare downweighted (more weighted) in the early stages, and any type of gradients is treatedequally at later stages; that is, the weights for each gradient approach a fixed value. By applying these strategies, MCLGS mitigates the negative impact of gradient conflicts without
gradient manipulation, and the update rule of MCLGS is consistent with that of naive MTLat the end. Thus, MCLGS retains the original objective toward the end of the training andhelps to converge on a better solution than conventional methods. We confirmed experimentally that MCLGS outperforms existing MTL methods, such as MGDA [21], PCGrad [29],GradDrop [4], and CAGrad [16], on NYUv2 [22] and BDD100K datasets [28]. Although existing methods do not improve the performance from the baseline on the BDD100K dataset,
MCLGS performs even better than the baseline.

2

Related WorkAs categorized in [26], existing approaches in MTL belong to either architectural methods
or optimization strategy methods. An example of architectural methods is a self-attention
mechanism adopted to obtain better features shared among tasks [17]. In MTL, a backbone,which works as a feature extractor, is generally shared among multiple tasks as shown in
Figure 1 (a). However in [8, 18, 19], task-wise backbones were implemented individuallyand connected via connection layers for sharing features. For a head, such as the classifieror regressor, a cascaded structure was proposed to share features of the early stages [27,31]. These methods manually introduced new connections between task-specific networks.
To automatically find such connections during training, neural architecture search has also
been utilized [10, 23]. Since MCLGS does not depend on a specific architecture, it can be
combined with these methods.
In contrast, several approaches focus on the optimization strategy of MTL. For example,
a loss balancing scheme was proposed based on homoscedastic uncertainty [12] or the normof the gradient [3]. Similar to MCLGS, some studies [9, 15, 20] introduced a learning
strategy inspired by the curriculum [2] that orders training data from easy ones to hard ones.
For example, [9] and [15] prioritized tasks during training depending on the difficulty ofthe task. [20] divided tasks into strongly and weakly correlated groups, and applied transfer
learning from the former to the latter. Since these methods are not motivated by the reductionof the gradient conflict, MCLGS can also be combined with them.
Several gradient manipulation methods [4, 16, 21, 29] have been proposed to remove
gradient conflicts. For example, PCGrad [29] removed the conflicting gradient componentsof two tasks by simply selecting one task and subtracting the conflicting component for theother task. This was repeated for random combinations of the tasks. Only the orthogonalcomponent for the other task is used for the parameter update. CAGrad [16] modified gradients to be a Pareto-optimal point around the original objective. However, if we manipulate
the gradients, the retained solution may no longer be optimal since the objective function
will be deviated from the original one. Meanwhile, MCLGS retains the original objective
toward the end of the training and helps to converge on a better solution than these methods.

4

IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING

Algorithm 1 MCLGS’s Update Rule
Require: Model parameter θ , # of current training step t
1: for i = 0, . . . , (N − 1) do
2:gi ← ∇[θ ⊤ ,θ ⊤ ]⊤ Li (θ̃ , θi )ig̃i ← ∇θ̃ Li (θ̃ , θi )4: end for
5: for i = 0, . . . , (N − 2) do
6:for j = (i + 1), . . . , (N − 1) do
7:wti, j ← f (g̃i , g̃ j ,t)
8:
end for
9: end for
N−1
1t
10: return update ∆θ = N(N−1)∑N−2i=0 ∑ j=i+1 wi, j (gi + g j )3:

3

Method

MTL aims to train a partially shared network to minimize the objectives of all tasks
simultaneously. Considering that we have N tasks, then the loss function for MTL is given
as follows:
N−1

L(θ̃ , θ0 , . . . , θN−1 ) = ∑ Li (θ̃ , θi ),

(1)

i=0

where L(θ̃ , θi ) denotes the loss function of the i-th task, and θ̃ and θi represent the sharedparameters and specific parameter of the i-th task, respectively (i.e., the so-called head, suchas the classifier and regressor). Hence, θ := (θ̃ , θ0 , . . . , θN−1 ) refers to all parameters in
the network. Here, gi := ∇θ Li (θ̃ , θi ) denotes a batch gradient of the i-th task, and η is thelearning rate. Thus, the update rule in MTL is as follows:θ′ = θ −η

1 N−1∑ gi .
N i=0(2)

MCLGS focuses on g̃i := ∇θ̃ Li (θ̃ , θi ), which is a batch gradient of the shared parameterθ̃ . While conventional methods [16, 21, 29] removed gradient conflicts by manipulating thegradients, MCLGS introduces a curriculum learning strategy based on directional similarity
into multi-task learning. Curriculum learning [2] is a training paradigm that orders training
data from easy to hard, like a human learning strategy. By introducing this strategy, thelearner can update parameters toward better local minima in the early stages and can reach abetter solution.
To introduce this strategy in an MTL setting, MCLGS considers samples that produce
more gradient conflicts as harder samples. In addition, we ignore these hard samples in theearly stages of training by downweighting their gradients. In MCLGS, the curriculum iscontrolled by the weighting function f , and the batch gradient is weighted by wti, j , which
is an output of the weighting function f . An entire process of MCLGS’s update rule isrepresented in Algorithm 1. First, task-wise gradients of shared parameters are extracted.
Second, a relative weight among tasks is calculated by the weighting function f based on
the number of gradient conflicts between the i-th and j-th tasks and the training step t. More

IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING

5details of the weighting function f will be described later. We define the relative weight wti, j
as follows:
wti, j := f (g̃i , g̃ j ,t) .
(3)Here, note that a curriculum is defined by f based on not only gradients but also the training
step t. Specifically, f is designed such that the output increases according to the similarity
in the beginning part of training. In the later training stages, the output will be independentof the similarity (i.e., the output approaches a fixed value). Finally, the batch gradient isweighted by wti, j , and the parameter θ is updated. The update rule of MCLGS is given as
follows:
N−2 N−11θ′ = θ −η(4)∑ ∑ wti, j (gi + g j ) .
N(N − 1) i=0
j=i+1

3.1

Weighting Function f

As formulated in [11], curriculum learning comprises scoring and pacing functions. The
scoring function defines how hard the fed sample is, while the pacing function denotes how
many hard samples are accepted in the current training step. These two functions also need
to be introduced in the weighting function f . Additionally, following the definition of curriculum learning, the weighting function f should be defined as a monotonically increasing
function. In this paper, we use the following function for the weighting function:f (g̃i , g̃ j ,t) = tanh (s (g̃i , g̃ j ) p (t)) + 1,

(5)

where s denotes the scoring function, and p represents the pacing function. Moreover, similar
to [24], we use a cosine similarity given in the following equation as the scoring function:s (g̃i , g̃ j ) =g̃i · g̃ j.
∥g̃i ∥∥g̃ j ∥(6)

The pacing function p is designed to approach 0 according to the increasing t. For example,
it could be a linear decay, which is given as follows:
p(t) = max (a0 − t∆a, 0) .

(7)

Conversely, it could be an exponential decay, given as follows:
p(t) = a0 rat .

(8)

An example of the weighting function f is shown in Figure 2. The angle of f will besmoother according to the increasing t, and finally, it will converge at 0, which signifies afixed weight (wti, j = 1). Additionally, a0 and ∆a for the linear decay and rat for the exponential decay are hyperparameters, and should be tuned to the target model architecture or
the target dataset appropriately. These hyperparameters could be one of the limitations of
MCLGS, but it is a common problem in many MTL methods[3, 4, 16]. Regarding how to
select the hyperparameters, we will experimentally show the parametric sensitivity studies
in Section 4.2.1. Additionally, since this paper focuses on introducing curriculum learning
based on gradient similarity in MTL, we selected the simple and somewhat heuristic weighting function f . Thus, the f in this paper might not be best solution and finding better f might
be future work.

6

IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING
2.00
1.75
1.50
weight1.25

0 epoch(t=0k)
20 epoch(t=46k)
25 epoch(t=69k)
27 epoch(t=92k)
30 epoch(t=115k)

1.00
0.75
0.50
0.25
0.00
−0.20 −0.15 −0.10 −0.05 0.00
0.05
cosine similarity

0.10

0.15

0.20

Figure 2: An example of weighting function f on the BDD100K dataset. The angle of fwill be smoother according to the increasing t, and finally, it will converge at 0, signifying afixed weight (wti, j = 1). Note that linear decay (Eq. 7) is used for the pacing function, wherea0 = 40, and ∆a = 3e − 4.

4

Evaluation

We evaluated MCLGS in a common MTL setting and the AD/ADAS MTL setting. For thecommon MTL setting, we chose the NYUv2 [22] dataset, which consists of three computer
vision tasks: semantic segmentation, depth estimation, and surface normal prediction. For
the AD/ADAS setting, we selected the BDD100K [28] dataset, which contains two computer
vision tasks: object detection and semantic segmentation.

4.1

Setup

4.1.1

The NYUv2 Dataset

We followed the evaluation setup in [16]. The single-task learning (STL) baseline modelis SegNet [1], as described in [16], and the MTL baseline model is SegNet with MTAN
[17]. Conventional methods and MCLGS are applied to the MTL baseline model. For theconventional methods, we evaluated MGDA [21], PCGrad [29], GradDrop [4], and CAGrad[16]. Additionally, the combination of MCLGS and CAGrad shows the compatibility of
MCLGS. For the training setup, we apply the SGD optimizer with a learning rate of 0.007,
a momentum of 0.9 and a weight decay of 0.0001 because the adaptive learning rate on theAdam optimizer could be incompatible with MCLGS. The results with the Adam optimizer,
which is used in the evaluation setup [16], is reported in the supplementary material. Since
MGDA [21] and CAGrad [16] changed the balance between task objectives, we applied
uncertainty weigh loss [12] for loss balancing on all methods to achieve a fair comparison.
We trained the model three times using each different random seed and calculated the average
accuracy. Similar to [16], we also used the average per-task performance drop ∆m. While∆m is directly calculated for all metrics in [16], we first calculated the average per-metricperformance drop in task i. Thus, ∆mi = K1 ΣKj=1 (−1)li, j (Mm,i, j − Mb,i, j )/Mb,i, j , where m andb represent the target method and the STL baseline, respectively; K denotes the number of
metrics on task i; and li = 1 if a higher value satisfies a criterion Mi, j for the metric j of task
i better; otherwise, li = 0. We calculated the average of mi for all the tasks to get ∆m. We
used the STL baseline with the Adam optimizer as a baseline for ∆m calculation.

IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING

4.1.2

7

The BDD100K Dataset

For the STL baseline model, we used FCOS-RT [25] for object detection. For semantic
segmentation, we combined ResNet50, the feature pyramid network (FPN) used in FCOSRT [25], and a segmentation head in [13]. For the MTL models, we applied an FCOS head
and a segmentation head in [13] to ResNet50 and an FPN used in FCOS-RT [25], whichwere shared among tasks. For the loss function, we followed [25] for object detection andused the cross entropy and dice loss weighted by 0.5 for semantic segmentation.
Although the BDD100K dataset comprises two tasks, each dataset is separated, and labels are not annotated on the same image (this setting is more closer to the actual operation
than the common MTL setting). Therefore, if each sample could have a ground truth of
only one task, then the loss function for the other sample would be missing. A missing loss
function of the task means that the learner studies each task alternatively, which may cause
catastrophic forgetting [14] during training. To avoid this, we pre-trained STL models foreach task and utilized their outpus to train MTL model as pseudo labels for each one. Moreover, we always fed the ground truths and pseudo labels to the MTL setting. Therefore, theloss function of this setting is formulated as follows:
Lmtl = wgt wod Lgt,od + wgt wss Lgt,ss + w pseudo wod L pseudo,od + w pseudo wss L pseudo,ss ,(9)where gt and pseudo represent the ground truth and the pseudo labels, respectively, andod and ss denote object detection and semantic segmentation, respectively. Additionally,w denotes the weights of loss functions for each label and task type. Here, we searchedthese weights without any MTL methods and used wgt = 0.8, w pseudo = 0.2, wod = 1.7 and
wss = 0.3. Note that Lgt,od and Lgt,ss could be missing, but L pseudo,od and L pseudo,od always
exist. To generate the pseudo label, we used thresholds for the teacher model output. The
threshold is 0.3 for object detection, 0.2 for non-maximum suppression, and 0.8 for semantic
segmentation.
Further details of the setup are as follows. We used the SGD optimizer with 0.9 momentum, 0.0001 weight decay, and we enabled Nestrov. The batch size was 16, and the total
epoch was 30. We used a multi-step learning rate schedule with 0.1 times learning decay at
the 16-th, 22-th, 28-th epoch. The initial learning rate was 0.01, and we used the learning
rate warmup with a 500 step. We reduced the gradient norm below 10. For the evaluation
metrics, we used COCO mAP@0.5:0.95 for object detection and mIoU for semantic segmentation. Similar to NYUv2, we trained the model three times with each different random
seed and calculated the average accuracy. We applied MCLGS and conventional methods,
such as MGDA [21], PCGrad [29], GradDrop [4], and CAGrad [16].

4.2

Results

4.2.1

Parametric Sensitivity Study

As mentioned in Section 3.1, the hyperparameters of the pacing function is one of the limitations in MCLGS. These hyperparameters could have a large impact regarding the performance. Thus, first, we performed a parametric sensitivity study for the pacing function. In
this paper, we used linear decay (Eq. 7) for the pacing function. Therefore, the pacing function comprises two hyperparameters: a0 and ∆a. Since ∆a should be correlated with a0 , weintroduced the following definition of ∆a:
a0
∆a =
,
(10)
rt ttotal

8IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING

rt
2
1
1/2
0
-0.50 ± 1.03
20 -0.04 ± 0.31 -1.51 ± 0.67 -1.45 ± 0.61
a0
40 -0.75 ± 0.59 -2.12 ± 0.48 -1.03 ± 0.18
80 -1.18 ± 0.46 -1.33 ± 0.31 -0.78 ± 0.89
Table 1: A parametric sensitivity study of the pacing function p for ∆m% (lower is better) on
the NYUv2 dataset. The performance is more sensitive regarding rt than a0 . Note that theformat of table values is (mean ± stderr).
rt
4
3
2
1
1/2
0
-3.27 ± 0.17
40 -3.88 ± 0.26 -3.94 ± 0.12 -3.79 ± 0.28 -3.91 ± 0.27 -3.36 ± 0.27
80 -3.96 ± 0.29 -3.79 ± 0.35 -3.86 ± 0.36 -3.76 ± 0.18 -3.58 ± 0.21
a0
120 -3.76 ± 0.36 -3.93 ± 0.24 -4.03 ± 0.27 -3.83 ± 0.30 -3.38 ± 0.19
160 -3.80 ± 0.20 -4.00 ± 0.23 -3.78 ± 0.19 -4.02 ± 0.29 -3.72 ± 0.21
200 -3.88 ± 0.22
NaN
NaN
-3.78 ± 0.19 -3.43 ± 0.18
Table 2: A parametric sensitivity study of the pacing function p for ∆m% (lower is better)on the BDD100K dataset. The performance is more sensitive regarding rt than a0 . NaN
represents the diverged cases due to the large weight of the curriculum. Note that the format
of table values is (mean ± stderr).
where ttotal represents the total training steps, and rt denotes the decreasing ratio, which isa hyperparameter in this definition. For example, rt = 1 means that when the training isfinished, the curriculum also converges at the fixed weight. Similarly, rt = 1/2 means thatthe curriculum converges in the middle of training.
Tables 1 and 2 show the sensitivity study results regarding a0 and rt on the NYUv2 and
BDD100K datasets, respectively. In both cases, the performance is more sensitive regarding
rt than a0 . To maximize performance, a0 and ∆a should be appropriate values. However,most cases with the curriculum outperform cases without the curriculum, shown as a0 = 0,
rt = 1. Additionally, a0 and rt should have larger values than those of the NYUv2 dataset,meaning that the pacing function should be slow.
4.2.2

Main Results

We present the results of the NYUv2 dataset in Table 3. MCLGS achieves the smallest average per-task performance drop ∆m of -2.12%. If just focusing on single-task performance,
PCGrad and MGDA [21] are the best for depth estimation and surface normal prediction,
respectively. However, these method can not improve the performance of the other tasks
well. This could be because they change the objectives by gradient manipulation and theirretained solution is biased toward specific tasks. In contrast, MCLGS is consistent with theoriginal objectives and achieves the better performance of all tasks. Furthermore, the uncertainty weigh loss [12] is suitable with MCLGS and improves the performance comparedto equal weighting. Additionally, MCLGS with CAGrad improves the performance evenbetter. We set a0 and ∆a to 60 and 5e − 4, respectively, based on the parametric sensitivity
study. Moreover, as shown in Tables 1 and 3, MCLGS with several hyperparameter settings

IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING#P.

Method3

STL Baseline
MTL Baseline
1.77(MTAN [17])1.77MGDA [21]1.77

PCGrad [29]1.77

GradDrop [4]1.77

CAGrad [16]1.77 MCLGS (ours)
1.77

CAGrad +MCLGS (ours)

Weightingequal
uncert.
equal
uncert.
equal
uncert.
equal
uncert.
equal
uncert.
equal
uncert.
equal
uncert.

Segmentation
Depth
Accuracy↑
Error↓
mIoU Pix Acc Abs Err Rel Err
39.33 64.55
0.5785 0.2339
40.88 66.14
0.5489 0.2290
38.95 64.76
0.5423 0.2185
20.52 53.16
0.6635 0.2532
36.42 63.54
0.5912 0.2286
40.73 66.24
0.5558 0.2275
39.05 65.10
0.5366 0.2163
40.56 66.13
0.5538 0.2251
39.18 64.87
0.5397 0.2201
39.39 65.27
0.5578 0.2270
37.51 64.05
0.5722 0.2339
40.98 66.43
0.5729 0.2358
40.20 65.63
0.5429 0.2174
41.37 66.47
0.5513 0.2230
39.72 65.65
0.5470 0.2222

9

Surface Normal∆m%↓Angle Distance↓
Within t ◦ ↑(mean ± stderr)
Mean Median 11.25 22.5
30
26.12 20.44 28.04 54.56 66.84
27.83 23.23 23.63 48.93 62.15
1.29 ± 0.60
26.55 21.67 25.69 52.12 65.20-0.50 ± 1.03
26.00 20.45 28.07 54.58 66.96
14.69 ± 0.17
25.51 19.95 28.89 55.70 68.06
0.71 ± 0.84
27.70 23.07 23.70 49.25 62.49
1.24 ± 0.66
26.36 21.35 26.38 52.81 65.70-1.40 ± 0.56
27.90 23.35 23.26 48.70 61.99
1.47 ± 0.06
26.42 21.52 26.04 52.42 65.44-0.81 ± 0.42
25.88 20.61 27.69 54.34 66.98-1.21 ± 0.82
25.93 20.57 27.77 54.40 66.99
0.78 ± 0.28
27.51 22.96 23.96 49.44 62.65
1.98 ± 0.43
26.03 21.11 26.68 53.32 66.26-2.12 ± 0.48
25.50 20.22 28.26 55.16 67.75-3.37 ± 0.72
25.51 20.15 28.47 55.33 67.82-2.74 ± 0.29

Table 3: Multi-task learning resuluts of the NYUv2 dataset: MCLGS with uncertainty weighloss [12] outperforms all the other methods. MCLGS with CAGrad improves the performance even better. For the loss weighting scheme, "equal" represents no loss balancing and"uncert" denotes the uncertainty weigh loss [12]. #P denotes the relative model size compared to the vanilla SegNet. The best average result for each method is marked in bold. The
best average result among all multi-task methods is annotated with boxes.
OD
SS
mAP@.50:.95 ↑mIoU ↑∆m%↓(mean ± stderr) (mean ± stderr) (mean ± stderr)2
STL baseline
25.37 ± 7.31e-4 49.66 ± 1.53e-3
1.15 MTL baseline 26.16 ± 1.48e-4 51.43 ± 4.53e-4-3.33 ± 0.17
1.15
MGDA [21]15.33 ± 6.24e-4 46.61 ± 1.37e-3
22.85 ± 0.40
1.15
PCGrad [29]26.14 ± 1.56e-4 51.37 ± 3.16e-4-3.25 ± 0.23
1.15
GradDrop [4]26.09 ± 4.78e-5 51.42 ± 5.66e-4-3.20 ± 0.21
1.15
CAGrad [16]25.37 ± 5.64e-5 50.70 ± 7.30e-4-1.05 ± 0.23
1.15 MCLGS (ours) 26.31 ± 1.99e-4 51.82 ± 8.13e-4-4.03 ± 0.27
Table 4: Multi-task learning results of the BDD100K dataset: MCLGS outperforms all theother methods. Following the evaluation format of [16], #P denotes the relative model size
compared to the STL baseline. The best average result among all multi-task methods ismarked in bold.
#P.

Methodachieves the better average per-task performance ∆m than existing methods.
We present the results of the BDD100K dataset in Table 4. MCLGS achieves the best
accuracy in both object detection and semantic segmentation even including standard errors.
Meanwhile, the accuracy of the conventional methods is below the naive MTL setting. Thismight be because of pseudo labels, which are less accurate than ground truth labels. Hence,
gradients of pseudo labels could conflict with those of ground truth labels. Here, conventional methods, such as PCGrad, remove the gradient conflicts whether the gradient comesfrom the ground truth or pseudo label. Thus, conventional methods may update parametersin the wrong direction. Meanwhile, since MCLGS just downweights a pair of conflicted gradients, it may clean up samples that contain inaccurate labels in this setting. We set a0 and∆a to 120 and 4.3e − 4, respectively, based on the parametric sensitivity study. Moreover,10

IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING

as shown in Tables 2 and 4, MCLGS with all hyperparameter settings achieves the better
average per-task performance ∆m than existing methods.

5Conclusion

In this paper, we proposed MCLGS, which mitigates the negative impact of gradient conflicts between tasks. MCLGS introduces a curriculum learning strategy [2] that utilizes only
easy samples in the early stages of training in multi-task learning. Conventional methods do
not update parameters in the corresponding direction for the original objective because they
manipulate gradients to remove the conflicts. Meanwhile, MCLGS just downweights samples that generate gradient conflicts in the early stage of training, and any type of gradient istreated equally at later stages, which is consistent with the naive MTL update rule. Therefore,
MCLGS retains the original objective toward the end of the training and helps to converge at
a better solution than conventional methods. As a result, we confirmed experimentally that
MCLGS is superior to the conventional methods and compatible with them, and it can reduce
the average per-task performance drop ∆m on the NYUv2 [22] and BDD100K datasets [28].

References[1] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on
pattern analysis and machine intelligence, 39(12):2481–2495, 2017.
[2] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum
learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009.
[3] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm:
Gradient normalization for adaptive loss balancing in deep multitask networks. In
International Conference on Machine Learning, pages 794–803. PMLR, 2018.
[4] Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models
with gradient sign dropout. Advances in Neural Information Processing Systems, 33:
2039–2050, 2020.
[5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler,
Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes
dataset for semantic urban scene understanding. In Proceedings of the IEEE conferenceon computer vision and pattern recognition, pages 3213–3223, 2016.
[6] Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv
preprint arXiv:2009.09796, 2020.
[7] Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser,
Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions on Intelligent Transportation Systems, 22(3):1341–1360,
2020.

IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING

11[8] Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille. Nddr-cnn: Layerwisefeature fusing in multi-task cnns by neural discriminative dimensionality reduction. In
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
pages 3205–3214, 2019.
[9] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic
task prioritization for multitask learning. In Proceedings of the European conferenceon computer vision (ECCV), pages 270–287, 2018.
[10] Pengsheng Guo, Chen-Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task
learning. In International Conference on Machine Learning, pages 3854–3863. PMLR,
2020.
[11] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training
deep networks. In International Conference on Machine Learning, pages 2535–2544.
PMLR, 2019.
[12] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty toweigh losses for scene geometry and semantics. In Proceedings of the IEEE conferenceon computer vision and pattern recognition, pages 7482–7491, 2018.
[13] Alexander Kirillov, Kaiming He, Ross Girshick, and Piotr Dollár. A unified architecture for instance and semantic segmentation. http://presentations.
cocodataset.org/COCO17-Stuff-FAIR.pdf, 2017.
[14] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka
Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
[15] Changsheng Li, Junchi Yan, Fan Wei, Weishan Dong, Qingshan Liu, and Hongyuan
Zha. Self-paced multi-task learning. In Thirty-First AAAI Conference on Artificial
Intelligence, 2017.
[16] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradientdescent for multi-task learning. Advances in Neural Information Processing Systems,
34, 2021.
[17] Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learningwith attention. In Proceedings of the IEEE/CVF conference on computer vision andpattern recognition, pages 1871–1880, 2019.
[18] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch
networks for multi-task learning. In Proceedings of the IEEE conference on computer
vision and pattern recognition, pages 3994–4003, 2016.
[19] Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latentmulti-task architecture learning. In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 33, pages 4822–4829, 2019.

12

IGARASHI ET AL.: MULTI-TASK CURRICULUM LEARNING[20] Nikolaos Sarafianos, Theodore Giannakopoulos, Christophoros Nikou, and Ioannis A
Kakadiaris. Curriculum learning for multi-task classification of visual attributes. In
Proceedings of the IEEE International Conference on Computer Vision Workshops,
pages 2608–2615, 2017.
[21] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization.
Advances in neural information processing systems, 31, 2018.
[22] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European conference on computer
vision, pages 746–760. Springer, 2012.
[23] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning
what to share for efficient deep multi-task learning. Advances in Neural Information
Processing Systems, 33:8728–8740, 2020.
[24] Mihai Suteu and Yike Guo. Regularizing deep multi-task networks using orthogonal
gradients. arXiv preprint arXiv:1912.06844, 2019.
[25] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: A simple and strong anchorfree object detector. IEEE Transactions on Pattern Analysis and Machine Intelligence,
2020.
[26] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans,
Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A
survey. IEEE transactions on pattern analysis and machine intelligence, 2021.
[27] Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided
prediction-and-distillation network for simultaneous depth estimation and scene parsing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 675–684, 2018.
[28] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu,
Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer
vision and pattern recognition, pages 2636–2645, 2020.
[29] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, andChelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824–5836, 2020.
[30] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on
Knowledge and Data Engineering, 2021.
[31] Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. Patternaffinitive propagation across depth, surface normal and semantic segmentation. In
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
pages 4106–4115, 2019.