CL-2018

Continual Learning
for an Encoder-Decoder CNN Using “Piggyback”
Asato Matsumoto
Keiji Yanai
The University of Electro-Communications, Tokyo, Japan

Objective

Experiments

Continual learning of an Encoder-Decoder network • Apply “Piggyback” to 3 kinds of image-to-image tasks.
Baseline 1: independent models (scratch)for heterogeneous image-to-image tasks
Baseline 2: copying models and incremental fine-tuning (fine-tune)
Baseline 3: shared decoder and task-specific decoder (decoder)

Semantic
Segmentation
for segmentaion andstyle transfer

Fast
Style transfer
Backbone

Colorization

Single network
for colorizationRelated work
• Simultaneous training of multiple tasks(1) Single encoder & task-specific decoders
e.g. UberNet [Ikkinos CVPR2017] Segmentation
Decoder

VGG16(encoder)

Encoder-Decoder network : U-net
task-specificbinarylast layermask• 5 tasks to be learned continuously
Task1 : Semantic segmentation with MSCOCO
Task2 : Semantic segmentation with PASCAL VOC 2012
Task3 : Colorization for gray-scaled MSCOCO images
Task4 : Fast Style Transfer with “Gogh’s Starry Night”
Task4’: Fast Style Transfer with “Munk’s Scream”• Experimental Results“Piggyback” worked well for Encoder-Decoder net !

Detection
Decoder
Boundary
Decoder
Surface
Normal(2) Multiple inputs & multiple outputs
e.g. “One Model To Learn Them All” [Kaiser et al. arXiv 2017]

Only intermediate parts are shared.

• Continual learning : major approarches
for overcoming “catastrophic forgetting”

Task2: segmentation results

Task3: colorization results(1) Rehearsal [Hetherington et al. 1989]• Keep (sampled) old training samples and use them for new training as well(2) Distillation (Learning without Forgetting (LfW)) [Li and Hoiem 2016]• Reproduce training labels of old tasks with trained model and use them for new training(3) Elastic Weight Consolidation (EWC) [Kirkpatrick et al. 2016] (regularization)• Train weights for new tasks according to un-importantness of weights(4) Progressive Neural Networks [Ruse et al. 2016]• Fix trained weights for the previous tasks, and extend the net and train extended weights.
• “PackNet” [Mallya et al. CVPR2018] : a pruning version of progressive network(5) Weight selection: “Piggyback” [Mallya et al. ECCV2018]• Select the weights of the fixed base network with task-specific binary masks

Task4: Gogh’s style transfer results

Task4’: Munk’s style transfer results(4) (5) bring “no catastrophic forgetting” with small additional task-specific weights .

In this work, we adopt “Piggyback” (5) as a basic approach.

Weight selection approach: “Piggyback”

Input

Input

scratch fine-tune decoder piggyback

scratch fine-tune decoder piggyback• The ratio of Zero’ed out weights
Task 2

Task 4

Task 3(1) For the first task, train a network as a “backbone network.”(2) For the second task or more, fine-tune the network, and [Mallya et al. ECCV2018]
obtain the task-specific binary mask by thresholding.
(3) At evaluation time, calculate the task-specific weights by elementwise masking.
The tendency is totallydifferent from VGG16
with classification tasks.

ImageNet -> WikiArt(VGG16)[Mallya et al. 2016]

Binary mask similarity matrix(ratio of the same values)

binarymask
backbone
network“piggyback”• Feature works
No “catastrophic forgetting” happen
with small additional binary masks
and task-specific final layers.

• Reducing the size of binary masks• Combining pruning• Analyzing the trained masks → binary masks are “task features” ??