Remembering Transformer for Continual Learning
Yuwei Sun, Ippei Fujisawa, Arthur Juliani, Jun Sakuma, Ryota Kanai

Catastrophic Forgetting§

Low-Rank Adaptation:
Mixture-of-Adapters:§RT Module

Novelty detection%"
RT Moduless

Reconstruction
loss ℓ&$%Reconstruction
loss$"
Inputs !

Inputs !

Old task replay

Cross-entropy loss

Outputs

Outputs

Herding("#$)

ReplaymemoryHerding

EE

Reconstructionloss

L2 loss

Outputs

×" Transformer blocks…

×"

Attention layer
outputs+%"#$!
#"#$sr$"#$!
""#$+Pretrainedweights!!

New task
Replay memory

Ø Soft probability outputof the old adapter

Attention layer
outputs%%!
""(")

We explore a scenario where the number of adapters is constrained and thenpropose the adapter fusion to identify and aggregate resembling adapters.

New task learning

Parameter sharing of "! and #!
Novelty detection!
#"r& = arg min ℓ%!
"% ∗Ø Permutation task: the input pixels of an image are shuffled
Replay with a random permutation for each task.

memoryTask/Permutation 1

Knowledge Distillation

Frozen weights'×

Pretrained
weights!!

§…

…

+…

Herdingr!!

Old task…

Attention layer
outputs∗!!

Replay memory

Step 2: Distillation

Autoencoder

Reconstruction
weightssr
similarity
betweenthecurrent
task
andloss

New task…

!"!

Inputs !

!
#% ∗%"#$ loss represents theØ The reconstruction
Pretrained

the old knowledge$"#$ encoded in theseautoencoders.

…

Old task…

Replaymemory!
#% ∗r!
"% ∗Ø The old adapter isremoved, and its
task samples are
distributed to the
newly learned
adapter E+1.

Task/Permutation 2…

selection" Step 1: Old task
""E!
#"#$!
""#$("#$)

Outputs

Task/Split 2…

Attention layer outputs

Reconstruction
loss ℓ&$%L2 loss×" Transformer
Ø During inference, the autoencoders,each blocks ×"
New task learning
Attention layer
Attention layer
encoding different task knowledge,
outputs
outputsOld
task
replaypredict routing weights to allocatea++sample to the most relevant adapter.

%%
sOutputs

Outputs

Task/Split 1…

Key idea: leverage the mixture-of-adapters that are sparsely activatedwith a novelty detection mechanism in a pretrained Transformer.

$"#$

Frozen weights∗

Cross-entropy loss…

Remembering Transformer

Adapter

Generative Routing
Novelty detection

Ø Hippocampus developed a novelty
detection mechanism to facilitate
consolidation by switching among[Sun 2023]
neural modules for various tasks.

!

Step 2: Distillation'Ø Hippocampus rapidly encodes task
data and consolidates the taskknowledge into the Neocortex byforming new neural connections.

!

Step 1: Old task selection& = arg min ℓ%Complementary Learning Systems Theory+

Ø Split task: a dataset is split into several subsets of equal
numbers of classes, with each subset as a task.

…

§Continual Learning Tasks…

Ø Catastrophic Forgetting (CF): new task knowledge interferes withold knowledge, causing previously learned tasks forgotten.
Ø Existing fine-tuning and regularization methods necessitate task
identity information and cannot eliminate task interference, whilesoft parameter sharing encounters an increasing parameter size.

Mixture-of-Adapters in ViT

Model Efficacy Under a Constrained Capacity

Remembering Transformer demonstrated SOTA continual learning
task accuracy and parameter efficiency through the mixture-ofadapters and generative routing in ViT.

arXiv:2404.07518
yuwei_sun@araya.org