#705

Multi-task Curriculum Learning
Based on Gradient Similarity

Hiroaki Igarashi1, Kenichi Yoneji1, Kohta Ishikawa2, Rei Kawakami3, Teppei Suzuki2, Shingo Yashima2, and Ikuro Sato23

1DENSO Corp., 2DENSO IT Lab., 3Tokyo Tech

Proposed method (MCLGS)

Background
Robotics applications such as autonomous driving require
multiple perceptional tasks.
(e.g. Object Detection and Semantic Segmentation)=Multi-task Learning × Curriculum Learning**It removes hard samples in the early stage of training and makes the solution better.
Early𝑔෤𝑖⇒ Multi-task learning (MTL)

Small

MTL Model (e.g. 2 task)

Training
step 𝑡Task Specific DNN: 𝜃1input

Task Specific DNN:𝜃2

Task1 output

𝑔෤𝑖

𝑡𝑤1,2
(𝑔෤1 + 𝑔෤2 )2𝑔෤𝑗𝑑𝐿𝑖
𝐿𝑖 : Loss of task i, 𝑔𝑖 = : Gradient of task i,𝑑𝜃

𝑁: #of task, 𝜂: learning rate𝑔෤𝑖
Toward naïve update(Treat them equally)𝑡
𝑤1,2≈1𝑔෤1 ∙ 𝑔෤2𝑠 𝑔෤1 , 𝑔෤2 =𝑔෤1 𝑔෤2𝑝 𝑡 = max 𝑎0 − 𝑡∆𝑎, 0

Related Works

𝑖=0 𝑗=𝑖+1

Experimental Results

20.05

1.5Wi,tj

0.1

00.5

-0.1050000 100000 150000
Training step t𝑔෤2

PCGrad manipulates gradients such that the conflicting
components are removed.
A Problem of PCGrade.g. when each task has a big gap in magnitude of gradients

Training step t𝑔෤1 ′Improvement from STL Baseline %↑Methods
ΔmΔseg Δdepth Δnormal(mean±stderr*)
STL Baseline
0.00
0.00
0.00
0.00±0.00
MTL Baseline
-0.32
6.39
-4.59
0.49±1.03
MGDA [21]
-4.48
0.03
2.33
-0.71±0.84
PCGrad [29]
0.07
7.37
-3.25
1.40±0.56
GradDrop [4]
0.06
6.30
-3.920.81±0.42
CAGrad [16]0.63
3.24-0.28
1.20±0.82
MCLGS (ours)
1.94
6.59-2.18
2.12±0.48
CAGrad [16] + MCLGS (ours) 4.08
4.68
1.34
3.37±0.72

The BDD100K Dataset [28]w/ PCGrad𝑔෤250000 100000 150000

The NYUv2 Dataset [22]𝑔෤1 ∙ 𝑔෤2′𝑔෤2 = 𝑔෤2 −𝑔෤12𝑔෤1𝑔෤10The weight depends on the similarity in the early stage of training,
but converges to a fixed value of 1 at the end of training.

Conflicting components𝑔෤1 + 𝑔෤2𝑔෤ =21-0.05
0w/o PCGrad* 𝑎0 ,∆𝑎: hyper parameters1𝑡′𝜃 ←𝜃−𝜂෍ ෍ 𝑤𝑖,𝑗 (𝑔𝑖 + 𝑔𝑗 )𝑁(𝑁 − 1)

Cosine Similarity

෨ since the parameter updates of each
In shared DNN (𝜃),task are oriented different directions, conflicting gradientssometimes lead to insufficient performance for each task.

𝑔෤1 ∙ 𝑔෤2′𝑔෤1 = 𝑔෤1 −𝑔෤22𝑔෤2+1

𝑁−2 𝑁−1

Gradient components can point in opposite directions between tasks.
𝑑𝐿2
𝑑𝐿1𝑔෤2 =𝑔෤1 =𝑑 𝜃෨𝑑 𝜃෨𝑔෤1𝑔෤𝑗

MCLGS’s update rule

A major challenge of MTL: gradient conflict𝑔෤1 ′𝑡𝑤1,2(𝑔෤1 + 𝑔෤2 )2

MCLGS doesn’t manipulate gradients but just downweights
samples that generate gradient conflicts in the early stage oftraining.

𝑖=0𝑔෤2 ′𝑔෤𝑗Large

More weighted𝑡(𝑤1,2> 1)𝑡Loss weight: 𝑤𝑖,𝑗 = tanh 𝑠 𝑔෤1 , 𝑔෤2 𝑝(𝑡)𝑁

PCGrad [29]𝑡𝑤1,2(𝑔෤1 + 𝑔෤2 )2

Amount of gradient conflicts

MTL shares a portion of the network between
multiple tasks, and reduce the complexity.

𝑔෤1𝑔෤𝑖𝑔෤𝑗Task2 output

1𝜃 ′ ← 𝜃 − 𝜂 ෍ 𝑔𝑖𝑁Less weighted𝑡(𝑤1,2< 1)

Later

Shared DNN: 𝜃෨Naïve update rule𝑡𝑤1,2(𝑔෤1 + 𝑔෤2 )2

𝑔෤1 ′ + 𝑔෤2 ′𝑔′
෤ =2𝑔෤2 ′𝑔෤2

Acquired gradient by PCGrad is much different fromthe original one.
A converged solution is no longer optimal for
original objective due to gradient manipulation.

methods
STL baseline
MTL baseline
MGDA [21]PCGrad [29]GradDrop [4]CAGrad [16]MCLGS (ours)

improvement from STL baseline %↑ΔmΔodΔseg(mean±stderr*)
0.00
0.00
0.00±0.00
3.10
3.56
3.33±0.17-39.57-6.14-22.85±0.40
3.06
3.44
3.25±0.23
2.85
3.55
3.20±0.21
0.00
2.10
1.05±0.23
3.71
4.34
4.03±0.27*The model is trained over 3 random seeds, and the average and the stderr are reported.