Training of Multiple and Mixed Tasks
With A Single Network
Using Feature Modulation
Mana Takeda1, Gibran Benitez1, and Keiji Yanai1
1The University of Electro-Communications, TokyoAbstract1

Abstract2

Abstract3

Introductionl Multi Task Learning– Run multiple tasks on a single network.
– In general, MTL models require task-speciﬁc parts, in addition
to the parts shared by all the tasks.
– As the number of tasks increases, the network becomes larger.
l We propose a single network with negligibly small task-speciﬁcparts.
Task 1
Task 2
Encoder

Task 3Decoder

General multi-task learning model4

Objective 1
1. Learning multiple heterogeneous image transfer tasks in a single
network.
Task specification signal

Style Transfer
Semantic
Segmentation

2. Learning mixed-task in the proposed network using synthesized
mixed-task training samples.
Task specification signal

Style Transfer＋Semantic
Segmentation5

Objective 2
1. Learning multiple heterogeneous image transfer tasks in a single
network.
Task specification signal

Style Transfer
Semantic
Segmentation

2. Learning mixed-task in the proposed network using synthesized
mixed-task training samples.
Task specification signal

Style Transfer＋SemanticSegmentation
6

Overview of the method – Task conditional vectorl Task conditional vector 𝒄 = [𝑐!, 𝑐", ⋯ , 𝑐# ]– 𝑐︓Task Strength (0 ≤ 𝑐 ≤ 1)
– 𝑛︓Number of tasks

Ex.）If the number of tasks 𝑛 is 3，𝐜 = [𝑐! , 𝑐" , 𝑐# ]– Learn only Task 1 ︓[1, 0, 0]– Mixed learning of Task 1 and Task 3 ︓ [1, 0, 1]

7

Overview of the method - FiLM-based network architecturel After normalizing the input features by Instance Normalization,affine transformation with FiLM parameters is applied.

Task conditionalvector 𝑐FC

FiLM generator
𝛾! , 𝛽! ⋯ 𝛾" , 𝛽"Instance Normalization + FiLM Layer
Instance Normalization{𝜇( , 𝜎( }𝑥!

𝑥( − 𝜇(𝑥( #)*+ =𝜎(Apply 𝛾( , 𝛽(𝑧( = 𝛾( 𝑥( #)*+ + 𝛽(𝑧!
8Experiment 1: Learning of Multiple Diﬀerent Tasks
l Multiple diﬀerent image translation tasks can be learned with asingle FiLM-based model.
Input(Task 0,3,4,5)

Input(Task 1)

Input

(Task 2)

Task 0

Task 1(reconstruction) (inpainting)

Task 2(denoising)

Task 3(SemanticSegmentation)

Task 4(StyleTransfer 1)

Task 5(StyleTransfer 2)

9Experiment 2: Learning of Mixed Tasks - Setting up the experimentl For verification of mixed-task learning, four tasks were used.
NumberTask name

Conditional vector

Task 0reconstruction[0, 0, 0]Task 1

denoising[1, 0, 0]Task 2

semantic segmentation[0, 1, 0]Task 3

Style Transfer 1[0, 0, 1]

Mix 1

denoising + Style Transfer 1[1, 0, 1]

Mix 2
Mix 3

denoising+ semantic segmentation
semantic segmentation+ Style Transfer 1[1, 1, 0][0, 1, 1]
10

Experiment 2: Learning of Mixed Tasks - How to create a target

Style Transfer1[0, 1, 0]denoisingStyle Transfer1[0, 1, 0]

input

target(denoising)Semantic
Segmentation
target(mix)

target(mix)
input(a) sequential mixing

target(mask)(b) mixing by region masking
11Experiment 2: Learning of Mixed Tasks - Resultsl The results are almost the same as ground truth (GT), whichmeans mixed- task learning succeeded.
MIX 1

A︓denoising
B︓Style Transfer 1

MIX 2

A︓denoising
B︓semantic segmentation

MIX 3

A︓semantic segmentation
B︓Style Transfer1

InputGTOutput

12

Experiment 3: Comparison to the Baselines
l This shows their outputs when learning of multiple heterogeneous
image translation tasks.
Style
Stylesemantic
reconstruction

inpainting

denoisingsegmentation

Transfer 1

Transfer 2

Ours

SGN
[ICCV2019]
SGN+bias[ICCV2019]
Extention
Piggyback1[ECCV2018]
Piggyback2[ECCV2018]13

Experiment 3: Comparison to the Baselines
l This compares the output of the proposed method and thebaseline for mixed-task learning.
MIX 3MIX 1
MIX 2
A︓denoising
B︓Style Transfer 1

A︓denoising
B︓semantic segmentation

A︓semantic segmentation
B︓Style Transfer1

Input
GT
Ours
SGN
SGN+bias14

Experiment 3: Comparison to the Baselines
Ours

SGN

SGN+bias Piggyback1 Piggyback2

reconstruction (PSNR↑)

8.514

3.786

3.777

7.964

7.908

inpainting (PSNR↑)

10.306

3.445

3.451

9.101

8.139

denoising (PSNR↑)

10.762

semantic segmentation (IoU↑)

0.5907

Ours achieved the best
3.665
3.656
7.590
6.942
evaluation
scorestasks 0.6014
0.4299 in almost0.4454 all the0.4030

Style Transfer 1(FID↓)

281.3

299.0

307.1

333.4

331.4

Style Transfer 2 (FID↓)

235.3

263.6

250.1

297.6

323.3

denoising+ Style Transfer 1 (FID↓)

304.3

349.1

343.4

-

-

denoising+semantic segmentation (PSNR↑)

6.716

2.442

2.470

-

-

semantic segmentation+ Style Transfer 1 (FID↓, IoU↑)

313.0
0.5457

339.0
348.6
small
model
size 0.5011
0.4555-Model size (num. of prams)

1,698,435

1,765,363

1,679,235

1,902,243

1,679,235

15

Conclusions
l we performed learning of multiple different image translation tasks
and their mixed tasks with the single FiLM-based network.
l Our method realized mixed task learning in addition to learning
multiple individual tasks.
l In future work, we plan to add more tasks such as various kinds ofimage do- main translation tasks and mix them.

16

17

Motivation to use FiLM
l The effectiveness of the method has been demonstrated in various
image transformation tasks.
→ It has high versatility and can be used to learn various imagetransformation tasks.
l FiLM allows the combination ofmultiple styles in Style Transfer.
→Possibility to combine
several diﬀerent imagetransformation tasks.

style1style2style3style4[3] Dumoulin et al.: A learned representation for artistic style, ICLR 2017.

18

Why the value of a task conditional vector can manipulate the strength of a transformationl A similar example of manipulating the strength of a transformation withconditional signals is Dynamic-Net.
l We consider the objective space of various image transformation tasks
to be a linearly coupled space of pre- and post-task execution.

Alon Shoshan, Roey Mechrez, and Lihi Zelnik-Manor. Dynamic-net: Tuning the objective without re-training. In Proc.ofIEEE International Conference on Computer Vision, 2019.
19

Mixed-task learning in the presence of many tasksl When there are many tasks, it is hard to learn all combinations ofthem.
l Therefore, our goal is to be able to generate task combinations
during inference by simply learning each task individually.

20