ICIP2022
The University of Electro-Communications Department of Informatics Tokyo, Japan Mana Takedea, Keiji Yanai ● Deep learning models forget previously learned tasks when given a new task (catastrophic forgetting) ●  Continual Learning addresses this problem by allowing users to continuously learn new tasks while retaining knowledge of previously learned tasks 1 ●  Recently, the Vision Transformer, which utilizes the Transformer architecture used in natural language processing for computer vision, has shown accuracy that exceeds that of CNN ●  Conventional Continual Leaning methods are generally designed to be applied to CNNs, so methods that can be applied to Vision Transformer are limited ●  Vision Transformer, which has a larger model size than CNN, requires a larger additional model size when applying Continual Learning methods → Need to suppress catastrophic forgetting with fewer parameters than conventional methods for application to CNN ●  Recently, the Vision Transformer, which utilizes the Transformer architecture used in natural language processing for computer vision, has shown accuracy that exceeds that of CNN ●  Conventional Continual Leaning methods are generally designed to be applied to CNNs, so methods that can be applied to Vision Transformer are limited ●  Vision Transformer, which has a larger model size than CNN, requires a larger additional model size when applying Continual Learning methods → Need to suppress catastrophic forgetting with fewer parameters than conventional methods for application to CNN Objective Method to suppress catastrophic forgetting with few parameters applicable to Vision Transformer
● Continual Learning is a method of continuously learning new tasks while retaining knowledge of tasks learned in the past – Class incremental: a new class is added – Task incremental: a new task is added ▲ Class incremental ▲ Task incremental

●  Rectification-based Knowledge Retention (RKR)
[1] Singh et al. Rectification-based Knowledge Retention for Continual Learning. CVPR 2021 – Apply task-specific modification parameters to the base parameters • Rectification Generator (RG) :  Parameters to modify weights •  Scaling Factor Generator (SFG) :  Parameters to modify intermediate outputs : Feature Map : Task Sharing Parameters : Task Specific parameter 5

●  Piggyback
[3] Arun et al. Piggyback: Adding multiple tasks to a single, fixed network by learning to mask. ECCV 2018 –  Apply the learned weight masks to the weights of the base model to transform the output – The weight mask is represented by a binary mask, so the number of additional parameters is small

● ViT
[2] Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021. –  Method directly applying the standard Transformer to a sequence of image patches
●  Swin Transformer
[3] Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. CVPR 2021. –  A method that solves the problems of ViT, such as limited resolution of object detection and a large number of input patches 7

● DyTox
[19] Arthur et al. Dytox: Transformers for continual learning with dynamic token expansion. CVPR 2022. –  Use task-specific tokens to generate task-specific embedding
●  Learning to Prompt for Continual Learning (L2P)
[20] Zifeng et al. Learning to prompt for continual learning. arXiv:2112.08654, 2021. – Methods for applying prompt learning in the field of natural language processing ● These methods are not comparable because they are class incremental methods 8 ●  In this work, we propose Mask-RKR as a method to perform task incremental Continual Learning ●  Mask-RKR is a method that applies Piggyback to the base RKR ●  Main features of Mask-RKR – Adaptation to task by RKR –  Parameter reduction by Piggyback
●  Mask-RKR adapts the network to each task by using RKR as the base. ●  RKR uses two generators, the Rectification Generator (RG) and the Scaling Factor Generator (SFG), to modify the weights and intermediate outputs of the network : Task Sharing Parameters : Task Specific parameter

RG Overview(1/2)
●  In RG, task- and layer-specific weight modification parameters are added to the weights of each task and layer that have already been pre-trained on the large data set : Task Sharing Parameters : Task Specific parameter
RG Overview(2/2)
●  Parameter reduction with low-rank approximation ●  Learn two matrices LM and RM of small size and use their product to generate parameters for weight modification

SFG Overview
●  In SFG, the intermediate output of each task and layer is multiplied by the intermediate output modification parameters specific to each task and layer : Task Sharing Parameters : Task Specific parameter
●  Piggyback transforms the output by applying a learned weight mask to the base weights ●  Mask-RKR further reduces the number of parameters by applying Piggyback to the RKR parameters : Task Sharing Parameters : Task Specific parameter
Parameter reduction in SFG )
●  Experiments were conducted in three Continual Learning settings to verify the performance of Mask-RKR ● Model – ResNet-18, ViT, Swin Transformer
●   Baseline
–  Single :  Learning each task with a unique model –  Multi Head :  Only the final output layer is replaced for each task –  RKR(K=2) :  A method to modify network weights and intermediate outputs for each task –  Piggyback : A method of transforming output by applying learned weight masks – Ours •  Ours(K=2) :  Mask-RKR of the proposed method •  Ours K+ :  Mask-RKR with the same number of parameters as "RKR" by adjusting the value of K
●  Experiments were conducted in three Continual Learning settings to verify the performance of Mask-RKR ● Model – ResNet-18, ViT, Swin Transformer
●   Baseline
–  Single :  Learning each task with a unique model –  Multi Head :  Only the final output layer is replaced for each task –  RKR(K=2) :  A method to modify network weights and intermediate outputs for each task –  Piggyback : A method of transforming output by applying learned weight masks – Ours •  Ours(K=2) :  Mask-RKR of the proposed method •  Ours K+ :  Mask-RKR with the same number of parameters as "RKR" by adjusting the value of K
●  Using CIFAR-100, which contains 100 classes of plants, animals, equipment, etc. – Divided into 10 tasks with 10 classes and studied in sequence (Task 1 → Task 2 → … → Task 10)
●  Using CIFAR-100, which contains 100 classes of plants, animals, equipment, etc. – Divided into 10 tasks with 10 classes and studied in sequence (Task 1 → Task 2 → … → Task 10)
●  Using CIFAR-100, which contains 100 classes of plants, animals, equipment, etc. – Divided into 10 tasks with 10 classes and studied in sequence (Task 1 → Task 2 → … → Task 10)
●  Using ImageNet-1k, a large dataset with 1000 classes –  Split into 10 tasks with 100 classes and train them in sequence (Task 1 → Task 2 → ... → Task 10)
● Use datasets from different domains – 5 tasks trained in sequence (D. Textures → GTSRB → SVHN → UCF101 → VGG-Flower)
●  The usefulness was verified by comparing RG and SFG w/ and w/o applying masks to each. – "RG w/ Mask": Apply mask to RG – "SFG w/ Mask": Apply mask to SFG ●  In this experiment, the model with Piggyback applied to RG and SFG with the lowest number of parameters is used
● Verified where masks are applied in RG (1) Not applied (2) Applied to weight modified parameters (3) Applied to low-rank approximated parameters (Mask-RKR) ●  To reduce the number of parameters, it is more effective to apply Piggyback to each of LM and RM
) ))
● Verified where masks are applied in RG (1) Not applied (2) Applied to weight modified parameters (3) Applied to low-rank approximated parameters (Mask-RKR) ●  To reduce the number of parameters, it is more effective to apply Piggyback to each of LM and RM
●  We proposed Mask-RKR, a continual learning method that can be applied to both CNN and Vision Transformer ●  Experimental results show that Mask-RKR can achieve higher accuracy than conventional methods while minimizing the increase in the number of parameters ●  In the future, we would like to improve Mask-RKR to make it flexible enough to handle continuous learning using datasets from different domains