DEIM-2023 Tutorial

Lifelong and Continual Learning
Bing Liu,1 Tatsuya Konishi,2 Gyuhak Kim,1 and Zixuan Ke1(The order of the presenters is based on the order of their presentations)
1

University of Illinois Chicago (UIC), USA
2
KDDI Research, Japan (currently visiting UIC)

Introductionn Classical machine learning: Isolated single-task learningn

A Key WeaknessnnHumans learn continually and accumulate knowledge over time.
nn

No continual learning, no knowledge accumulation or transfer
We never stop learning

ChatGPT can only answer questions based on its training data going
up until 2021. It needs to be continually updated or adapted.

DEIM2023 Tutorial, March 6, 2023

Chen and Liu. Lifelong machine learning. Morgan & Claypool. 2016, 2018
Liu. Learning on the Job: Online Lifelong and Continual Learning. AAAI-2020

2

Early definition of lifelong learning(Thrun 1996, Silver et al 2013; Ruvolo and Eaton, 2013; Chen and Liu, 2014, Chen and Liu, 2016)n The learner has performed learning on a sequence of tasks

from 1 to N.
n When faced with the new or (N+1)th task, it uses the relevant

knowledge learned from the earlier tasks stored in theknowledge base (KB) to help learn the (N+1)th task.
q

After learning (N+1)th task, KB is updated with the learned resultsfrom the (N+1)th task.

n Goal: knowledge transfer
Thrun. Is learning the n-th thing any easier than learning the first? NIPS-1996.
DEIM2023 Tutorial, March 6, 2023

3

New definition of lifelong/continual learning(Chen and Liu, 2014, 2018)n Learn a sequence of tasks, T1, T2, …, TN, … incrementally.

Each task t has a training dataset.

n Goal: Learning all tasks in a single neural network

1. without catastrophic forgetting: Learning of new task TN+1 should not
result in degradation of accuracy for the previous N tasks.
2. with knowledge transfer: leveraging the knowledge learned from theprevious tasks to learn the new task TN+1 better.
nAssumption: Once a task is learned, its data is no longer
accessible, at least a majority of it, and both the task TN+1.
Chen and Liu. Lifelong machine learning. Morgan & Claypool. 2018

DEIM2023 Tutorial, March 6, 2023

4

Lifelong/continual learning(Thrun 1996, Silver et al 2013; Ruvolo and Eaton, 2013; Chen and Liu, 2014, 2018)

Chen and Liu. Lifelong machine learning. Morgan & Claypool. 2018
DEIM2023 Tutorial, March 6, 2023

5

Different continual learning settingsn Task incremental learning (TIL)

train a “separate” model for each task
q task-id is provided in testingqn Class incremental learning (CIL)

produce a single model from all tasks
q classify all classes in testingqn Domain incremental learning (DIL)

All tasks have the same set of classes
q Task-id is not provided in testingq

DEIM2023 Tutorial, March 6, 2023

6

Topics
Early Research on Lifelong Learning (Bing Liu)n Task Incremental Learning (Tatsuya Konishi)n Class Incremental Learning (Gyuhak Kim)
n Domain Incremental Learning & Continual Learning in NLP (Zixuan Ke)
n Open-World Continual Learning (Bing Liu)n Summary (Bing Liu)n

DEIM2023 Tutorial, March 6, 2023

7

Early Research on Lifelong Learning
Bing Liu
University of Illinois Chicago
https://www.cs.uic.edu/~liub/Early work of lifelong learning (LL)
(Thrun, 1996)
n Concept learning tasks: The functions are learned over the

lifetime of the learner, f1, f2, f3, … Î F.
n Each task: learn the function f: I ® {0, 1}. f(x)=1 means x
belongs to a particular concept (or class).
q

For example, fdog(x)=1 means x is a dog.

n For nth task, we have its training data X and alsoq

the training data Xk of k =1 , 2, …, n-1 tasks.

n Most early LL is about task incremental learning (TIL)n

In learning each new task, it may still use all previous task data.
Thrun. Is learning the n-th thing any easier than learning the first? NIPS-1996.

DEIM2023 Tutorial, March 6, 2023

9

Approaches in (Thrun, 1996)
n The paper proposed a few approaches based on two learningalgorithms,
Memory-based, e.g., kNN
q Neural networks,qn Goal: when we learn fdog(x), we can leverage functions or

knowledge learned from previous tasks, such as fcat(x), fbird(x),ftree(x), etc.
q

Data for fcat(X), fbird(X), ftree(X)… are called support sets.
Thrun. Is learning the n-th thing any easier than learning the first? NIPS-1996.

DEIM2023 Tutorial, March 6, 2023

10

KNN based “lifelong learning”
n Use the support sets of {f1, f2, …, fk, …, fn-1} to learn a new

representation, or function
g: I ® I’

which maps input vectors to a new space. The new space is the
Adjusting g to minimize E forces the distance
input space for the final kNN.
between pairs of examples of the same class
to be small, and the distance between
q Adjust g to minimize the energy function.
examples of different classes to be largeqq

g is a neural network, trained with Back-Prop. kNN is then applied forthe nth (new) task. g basically learns a feature extractor.

DEIM2023 Tutorial, March 6, 2023

11

ELLA: Efficient Lifelong Learning Algorithm(Ruvolo & Eaton, 2013)
n ELLA is based on GO-MTL (Kumar et al., 2012)q

GO-MTL is a batch multitask learning methodn ELLA is an online multitask learning method

ELLA is more efficient and can handle a large number of tasks
q Becomes a lifelong learning methodqnn

The model for a new task can be added efficiently.
The model for each past task can be updated rapidly.

Ruvolo and Eaton. ELLA: An efficient lifelong learning algorithm. ICML-2013.
DEIM2023 Tutorial, March 6, 2023

12

Notations
n N tasks in totaln k (< N) latent basis model components

n Each basis task is represented by a l (a vector of size d)n For all latent tasks, L = (l1, l2, …, lk)n

L is learned from N individual tasks.
qE.g., weights/parameters of logistic regression or linear regression

DEIM2023 Tutorial, March 6, 2023

13

The Approachn st is a linear weight vector and is assumed to be sparse.

n Stacking st for all tasks, we get S. S captures the task

grouping structure.

DEIM2023 Tutorial, March 6, 2023

14

Initial Objective Function of ELLAnObjective Function

DEIM2023 Tutorial, March 6, 2023

15

Approximate Equation (1)n Eliminate the dependence on all of the past training data

through inner summationq

By using the second-order Taylor expansion around q = q (t) whereqq (t) is an optimal predictor learned on only the training data for task t.

DEIM2023 Tutorial, March 6, 2023

16

Taylor Expansion
n One variable functionn Multivariate function

DEIM2023 Tutorial, March 6, 2023

17

Removing inner summation
Plugging the second-order Taylor expansion into Eq. (1) yields(see (Chen and Liu, 2018) for the derivation. The original paper didn’t give the derivation)

Chen and Liu. Lifelong machine learning. Morgan & Claypool. 2018
DEIM2023 Tutorial, March 6, 2023

18

Simplify optimization with an assumptionnAfter s (t) is computed given the training data for task t, it will not
be updated when training on other tasks. Only L changes.

DEIM2023 Tutorial, March 6, 2023

19

ELLA Accuracy Resultn ELLA vs. GO-MTL

ELLA was based on the multitask learning method in GO-MTL

DEIM2023 Tutorial, March 6, 2023

20

LTM: Lifelong Topic Modelingn Topic modeling (Blei et al 2003) finds topics from a collection of

documents.
nn

A document is a distribution over topics
A topic is a distribution over words, e.g., {price, cost, cheap, expensive, …}

n Problem: Learn a sequence of aspect extraction tasks insentiment analysis with review data D1, D2, …,DN.
nLearned topics from each Di is Si. S = Ui Si is the topic base so far.

n Goal: Given a new task with its review data Dt, learn from Dt

with the help of S.
Chen and Liu. Topic Modeling using Topics from Many Domains, Lifelong Learning and Big Data. ICML-2014.
DEIM2023 Tutorial, March 6, 2023

21

Sentiment Analysis (SA) Context
n “The size is great, but pictures are poor.”q

Aspects/topics (product features): size, picturen Why lifelong learning can help SA? A fair amount of aspect

overlapping across reviews of different products or domains
Every product review domain has the aspect price,q Most electronic products share the aspect battery
q Many also share the aspect of screen.
qn This sharing of concepts / knowledge across domains is truein general, not just for SA.
DEIM2023 Tutorial, March 6, 2023

22

What knowledge?
n Should be in the same aspect/topic=> Must-Links
e.g., {picture, photo}n Should not be in the same aspect/topic=> Cannot-Links
e.g., {battery, picture}
DEIM2023 Tutorial, March 6, 2023

23

Lifelong Topic Modeling (LTM)n Must-links are mined dynamically.

Chen and Liu. Topic Modeling using Topics from Many Domains, Lifelong Learning and Big Data. ICML-2014.
DEIM2023 Tutorial, March 6, 2023

24

LTM Modeln Step 1: Run topic modeling (e.g., LDA) on each task/domain Di to

produce a set of topics Si called Topic Basen Step 2: Mine prior knowledge (must-links) and use knowledge to
guide modeling.

DEIM2023 Tutorial, March 6, 2023

25

Knowledge Mining Functionn Topic matching: find similar topics from the topic base for each

topic in the new domainn Pattern mining: find frequent itemsets from the matched topics

DEIM2023 Tutorial, March 6, 2023

26

An ExamplenGiven a newly discovered topic:{price, book, cost, seller, money}q We find matching topics from topic base S
n Domain 1: {price, color, cost, life, picture}n Domain 2: {cost, screen, price, expensive, voice}n Domain 3: {price, money, customer, expensive}n

If we require words to appear in at least two domains, we get two
must-links (knowledge):{price, cost} and {price, expensive}.
q Each set is likely to belong to the same aspect/topic.
q

DEIM2023 Tutorial, March 6, 2023

27

Model Inference: Gibbs Samplingn

How to use the must-links knowledge?
qe.g., {price, cost} & {price, expensive}

Graphical model: same as LDA (Latent Dirichlet allocation)n But the model inference is very differentnqnGeneralized Pólya Urn Model (GPU)

Idea: When assigning a topic t to a word w, also assign a fractionof t to words in must-links sharing with w.

DEIM2023 Tutorial, March 6, 2023

28

Simple Pólya Urn model (SPU)

Generalized Pólya Urn model (GPU)…

Experiment Results

DEIM2023 Tutorial, March 6, 2023

31

Early work vs. more recent work
n Early research of lifelong/continual learning focused on thetask incremental learning (TIL) setting.
q

A separate model is built for each task.
nq

No catastrophic forgetting

Knowledge transfer across tasks is the main goal.

n Recent deep learning based continual learning

Build all task models within a single neural networkq Deal with both catastrophic forgetting and knowledge transfer
q Investigate all three settings (TIL, CIL and DIL) of continual learningq

DEIM2023 Tutorial, March 6, 2023

32

Task Incremental Learning

Tatsuya KonishiKDDI Research, Inc. / Visiting scholar at UIC

Goals of this talkn

To be helpful for those who're interested in continual learning (CL) and want to apply
CL techniques to your own applications/data.
q

TIL is the simplest scenario of CIL, but it handles many real world aplications.
nq

You can step in the world of CL through learning of TIL.
nn

Humans can give instructions to models in the form of task ID.
e.g., classification over fine-grained labels of animals (CV), sentiment analysis/text summarization (NLP).

Next two talks by Gyuhak Kim and Zixuan Ke will explain CIL and DIL, respectively, which are morechallenging scenarios!

This talk gives …
qq

The standard TIL setup.
Simple introduction of representative CL methods which are helpful for you all to know thetrend of ideas on CL.

DEIM2023 Tutorial, March 6, 2023

2

Agenda1.

What is TIL?

2.

Categorization in CL& Deep dive into representative methods

3.

Future directions

DEIM2023 Tutorial, March 6, 2023

3

What is TIL?
n

TIL is the simplest scenario in CL.
qq

TIL learns a sequence of tasks, 1,2, … , %. Each task & ∈ ( = {1,2, … , %} has a set of labels, ,! , and its
corresponding data, - ! .
The goal is to build a model .: - ×( → , to identify the class labels 2! ∈ ,! for a test instance 3! from task &.
Both in training and testing, a model requires task ID to identity a correct classification head.

Classification head for task 1…

!!

Feature extractor(shared)Classification head for task !
n#"!

Objective: Plasticity-stability dilemmaqq(Stability) Preventing forgetting and (Plasticity) Knowledge transfer
Some TIL approaches have achieved no fogetting, but balancing stability-plasticity still remains an interesting
topic.

DEIM2023 Tutorial, March 6, 2023

4

How to evaluate TIL systems?
nCommon datasets
Split CIFAR100 (into " tasks, e.g., " = 10)

Metrics" is the total number of tasks.
&!,# is the test accuracy on task ' after learning task (.
q

Task 1:

Label 1 to 10

Task 2:

Label 11 to 20…

Continual learningqnTask 10:

Label 90 to 100

Average accuracy&

1) &!,&"!$%q

Forgetting ratio (negative values mean forgetting)&'%1) (&!,! − &!,& )"−1!$%

After learning all tasks
Just after learning task "DEIM2023 Tutorial, March 6, 2023

5

Agenda1.

What is TIL?

2.

Categorization in CL& Deep dive into representative methods

3.

Future directions

DEIM2023 Tutorial, March 6, 2023

6

Categories of CL methodsnRegularizationq

ConceptnqnqPenalize changes to important parametersnqFunction regularizationqq

Let each task monopolize a own sub-network inside the whole network

ApproachesnModel decompositionnModular network

LwF

q

Weight regularizationq

ConceptnApproachesnArchitecture

EWC
UCL

q
q
n

q

Replay
q

n

q

q
q

Concept
Store some previous samples to be learned together with the learning of a
new current task

Approachesn

GEM

Generative replay
q

DGR

q

Conceptnq

Control the loss such that forgetting can be alleviatedApproachesnGradient projection
q

n

CUBER

Loss landscapeq

DEIM2023 Tutorial, March 6, 2023

HAT
SupSupCAT
PRM

Optimization

Experience replayqnnProgressive Networks
PathNetParameter isolation
q

n

CCLL

Stable-SGD

7

Regularization>Function regularization | LwFn

LwF (Learning without Forgetting) is one of the earliest work, which shows that preserving
outputs between old and new models is an effective strategy to mitigate forgetting.
q

LwF uses the Knowledge Distillation loss to ensure the output probabilities are close to the recorded
output probabilities of the previous tasks' network.

Not to change output of the previous model during learning of the new task.
LwF, https://arxiv.org/abs/1606.09282, Learning without Forgetting. Li and Hoiem, In Proc. of ECCV, 2016.

DEIM2023 Tutorial, March 6, 2023

8

Regularization>Weight regularization | EWCnn

EWC (Elastic Weight Consolidation) is an early work in CL.
EWC selectively slows down learning on the weights important for previous tasks.
qqq

The posterior distribution, - . /( , is regarded as the importance for task A.
Since the true posterior probability is intractable, the Laplace approximation is applied toapproximate it as a Gaussian distribution with a diagonal precision given by the diagonal of theFisher information matrix, 0.
The Fisher information matrix works as a regularization term to suppress parameter changes.

Regularization

EWC ensures task A is remembered
Results on permuted MNIST tasks.
while training on task B.
EWC, http://arxiv.org/abs/1801.01423, Overcoming catastrophic forgetting in neural networks. Kirkpatrick et al., PNAS, 2017.

DEIM2023 Tutorial, March 6, 2023

9

Regularization>Weight regularization | UCLn

UCL (Uncertainty-regularized Continual Learning) proposes node-wise uncertainty based on
Bayesian learning framework, with two additional regularization terms.
Weights with small uncertainty should be treated as important.
à Weights with more uncertainty are going to be updated more easily.

nDrawbacksq

Weights indentified as important in previous tasks should be kept asimportant for future tasks, which is helpful to prevent forgetting.

Though UCL can effectively balance plasticity and stability, it still suffers from forgetting of old tasks.

UCL, https://arxiv.org/abs/1905.11614, Uncertainty-based Continual Learning with Adaptive Regularization. Ahn et al., In Proc. of NeurIPS, 2019.

DEIM2023 Tutorial, March 6, 2023

10

Categories of CL methodsnRegularizationq

ConceptnqnqPenalize changes to important parametersnqFunction regularizationqqnModel decompositionnModular network

q

EWC
UCL

q
q

Replay methods are most popular in the CIL setup,
which are also discussed in the next talk by Gyuhak Kim.
n

n

q

n

q

q
q

Concept
Store some previous samples to be learned together with the learning of a
new current task

Approachesn

GEM

Generative replay
q

DGR

q

HAT
SupSupCAT
PRM

Conceptnq

Control the loss such that forgetting can be alleviatedApproachesnGradient projection
q

n

CUBER

Loss landscapeq

DEIM2023 Tutorial, March 6, 2023

Progressive Networks
PathNetOptimization

Experience replay
q

n

n

CCLL

Parameter isolationq

Replayq

Let each task monopolize a own sub-network inside the whole network

Approaches

LwF

Weight regularizationq

ConceptnApproachesnArchitecture

Stable-SGD

11

Replay>Experience replay | GEMn

GEM (Gradient Episodic Memory) is one of the most popular replay-based methods. It storesa few samples for previous tasks that are used together in learning of a new task.
qn

GEM simply collects last 4 examples from each task, then optimizes the model such that the loss for previous
tasks with the examples decreases (i.e., getting improved), by exploiting gradients over examples.

This paper is also known for proposing some new evaluation metrics in addition to accuracy:qq%
Backward transfer: &'% ∑&'%!$% &&,! − &!,!
%∑&) &!'%,! − 32!
Forward transfer:&'%

How benefitial?
Previous tasks

Current new task

How benefitial?
Previous tasks

Current new task

where &!,# is the test accuracy of the model on task ' after observing the last sample from task (,and 32! is the test accuracy for task ( at random initialization.
GEM, https://arxiv.org/abs/1706.08840, Gradient Episodic Memory for Continual Learning. Lopez-Paz and Ranzato, In Proc. of NeurIPS, 2017.

DEIM2023 Tutorial, March 6, 2023

12

Replay>Generative replay | DGRnnn

Simply replaying all previous data alleviate the forgetting, but it requires a large memory.
DGR (Deep Generative Replay) is the first work that employs an additional generative model
to replay generated data. Learning each task is accomplished with replaying generated data
sampled from the old generative model.
In training a new task, two independent procedures are involved:qq"generator", a deep generative model, learns to reconstruct the cumulateive input space.
"solver", a task solving model, is trained to couple the inputs and targets.

DGR, https://arxiv.org/abs/1705.08690, Continual Learning with Deep Generative Replay, Shin et al., In Proc. of NeurIPS, 2017.

DEIM2023 Tutorial, March 6, 2023

13

Categories of CL methods
These methods are probably most effective for TIL,
as they can achieve learning without forgetting.
nRegularizationq

ConceptnqnqPenalize changes to important parametersnqFunction regularizationqq

Let each task monopolize a own sub-network inside the whole network

ApproachesnModel decompositionnModular network

LwF

q

Weight regularizationq

ConceptnApproachesnArchitecture

EWC
UCL

q
q
n

q

Replay
q

n

q

q
q

Concept
Store some previous samples to be learned together with the learning of a
new current task

Approachesn

GEM

Generative replay
q

DGR

q

Conceptnq

Control the loss such that forgetting can be alleviatedApproachesnGradient projection
q

n

CUBER

Loss landscapeq

DEIM2023 Tutorial, March 6, 2023

HAT
SupSupCAT
PRM

Optimization

Experience replayqnnProgressive Networks
PathNetParameter isolation
q

n

CCLL

Stable-SGD

14

Architecture>Model decomposition | CCLLn

CCLL (Calibrating CNNs for Lifelong Learning) effectively re-uses the features trained on theinitial task, and efficiently (re)calibrates them, using a very small number of calibration
parameters.

n

CCLL is trained as follows:qqn

For the first task (4 = 1), all modules are trained.
For all the subsequent tasks (4 > 1),
the base modules, colored in gray, are frozen.
Only calibration modules, colored in green,
are trained for each task.

frozenfrozenfrozenDrawbacksqq

The performance highly depends on the first task.
Backward transfer is impossible.

trained

CCLL, https://dl.acm.org/doi/10.5555/3495724.3497031, Calibrating CNNs for Lifelong Learning. Singh et al., In Proc. of NeurIPS, 2020.

DEIM2023 Tutorial, March 6, 2023

15

Architecture>Modular network | Progressive Networksn

PGN (Progressive networks) tries both toqqn

prevent forgetting by instantiating a new network (a column) for each task
enable transfer via lateral connections to features of previous learned columns

Task 1

Task 2

Task 3Since modules used for previous tasks are not updated,
PGN does not suffer from forgetting.
Layer 2nDrawbacksqqLayer 1A new column may be added, which may induce another issue,e.g., the growth in the number of parameters with the number of tasks.
Backward transfer is impossible.

Progressive Networks, https://arxiv.org/abs/1606.04671, Progressive Neural Networks. Rusu et al., 2016.

DEIM2023 Tutorial, March 6, 2023

16

Architecture>Modular network | PathNetnn

PathNet has # layers with each layer consisting of $ modules. Each module is a neuralnetwork, e.g., convolution or linear.
Then, learning decides which modules are assigned to each task based on a tounament
selection genetic algorithm.

nSince the model remembers which modules
are used for a task and uses them in testing,no forgetting happens.

n

Drawbacksqq

Later tasks will have limited numer of parameters to use.
Backward transfer is impossible.

PathNet, https://arxiv.org/abs/1701.08734, PathNet: Evolution Channels Gradient Descent in Super Neural Networks. Fernando et al., 2017.

DEIM2023 Tutorial, March 6, 2023

17

Architecture>Parameter isolation | HAT (1/2)
nn

HAT (Hard Attention to the Task) is the first work that dynamically allocates each task a subnetwork by using masksof neurons and modifies parameters' gradients to prevent forgetting.
In the forward pass of learning task 5:qqq

HAT optimizes not only model parameters but also task embeddings, !!" , for each layer ".
Through a sigmoid function, a mask, #!" = %('!!" ), is computed. As ' is a large number, #!" is expected to behave as a binary
function mask, taking 0 or 1.
Based on the mask, neurons of the layer are either activated (by multiplying 1) or deactivated (by multiplying 0), being passed to
the next layer.
That is, #!" can be regarded as importance of neurons for that task.

ℎ′" has non-zero values only when its
corresponding mask values of 8"# are 1.
A pseudo binary mask
Task embedding is
also trained.
HAT, https://arxiv.org/abs/1801.01423, Overcoming Catastrophic Forgetting with Hard Attention to the Task, Serrà et al., In Proc. of ICML, 2018.

DEIM2023 Tutorial, March 6, 2023

18

Architecture>Parameter isolation | HAT (2/2)
nIn the backward pass of learning task 5:
The importance of neurons, #!" , has been accumulated in CL from task 1 to ) − 1.

q<latexit sha1_base64="jN+dKFom1K0PLbLrolWIoutBar4=">AAACJXicbVDLSgMxFM3UV62vqks3wSJU0DJTirpQKLpxWcE+oFNLJs20oZkHyR2xDP0ZN/6KGxcWEVz5K2amXdTqgcDJOeeS3OOEgiswzS8js7S8srqWXc9tbG5t7+R39xoqiCRldRqIQLYcopjgPqsDB8FaoWTEcwRrOsObxG8+Mql44N/DKGQdj/R97nJKQEvd/CXpiofYFgzDqTXGV9j2yBPWdxeKOPUS/QTPxcpjbEveH8BxN18wS2YK/JdYM1JAM9S6+YndC2jkMR+oIEq1LTOETkwkcCrYOGdHioWEDkmftTX1icdUJ063HOMjrfSwG0h9fMCpOj8RE0+pkefopEdgoBa9RPzPa0fgXnRi7ocRMJ9OH3IjgSHASWW4xyWjIEaaECq5/iumAyIJBV1sTpdgLa78lzTKJeusVLmrFKrXszqy6AAdoiKy0DmqoltUQ3VE0TN6Re9oYrwYb8aH8TmNZozZzD76BeP7BzOso0A=</latexit>

An accumulated mask:<latexit sha1_base64="bkhojhR+z9wXBs4B03OkrL3Sxis=">AAACS3icbVBNb9NAEF2nBEL4qClHLiMiRJCSyEYR9FKpggvHViJJpdhE68062Xa9tnbHSJHl/9dLL9z4E730AEIcunF8CAkjrfbNmzczuy/KpDDoeT+dxsGD5sNHrcftJ0+fPT90XxyNTZprxkcslam+iKjhUig+QoGSX2Sa0ySSfBJdfV7XJ9+5NiJVX3GV8TChCyViwShaauZGi1khe+KyfAsnEEge4xR86EOQCFWlXaCVovxW2Byw75e9iur7vcstEgItFkt8V98hVIPBTp65HW/gVQH7wK9Bh9RxNnN/BPOU5QlXyCQ1Zup7GYYF1SiY5GU7yA3PKLuiCz61UNGEm7CovCjhjWXmEKfaHoVQsdsdBU2MWSWRVSYUl2a3tib/V5vmGB+HhVBZjlyxzaI4l4AprI2FudCcoVxZQJkW9q3AllRThtb+tjXB3/3yPhi/H/gfBsPzYef0U21Hi7wir0mX+OQjOSVfyBkZEUauyS35RX47N86d88f5u5E2nLrnJfknGs17DVWwjw==</latexit>

0
gl,ij
=

2
atl 1 , atl⌘If a neuron has importance 1 for any of previous task 1
to 5 − 1, 8"$#%& has importance 1 for the neuron.

h

1

min

⇣

1 t 1at, al 1,j
l,i

⌘i

gl,ij

Parameters connecting important neuron : of layer ; − 1 (with 1 for
$#%&
$#%&
8"%&,() and neuron i of layer ; (with 1 of 8",)) is not updated.

Resultsqn⇣

The gradients of parameters in layer ", ,!,$% , are modified such that parameters feeding to the important neurons are not going to
be changed. The modified ones, ,′!,$% , are acutally used in the optimization.

q

n

1
at
= max
l

In the forward pass (in testing), binary masks from task embeddings are used to select neurons by which the model can recall
the sub-network dedicated to the task without forgetting.

Drawbacksq
Later tasks will have limited number of "alive" parameters.
q
Backward transfer is impossible.

DEIM2023 Tutorial, March 6, 2023

19

Architecture>Parameter isolation | SupSup (1/2)
nSupSup (Supermasks in Superposition) applies the lottery ticket hypothesis to CL.
qnWhat is the lottery ticket hypothesis (which was proposed in ICLR2019)?
n
The number of possible subnetworks is combinatiorial in the number of parameters.
n
It has been observed that the number of combinations is large enough that even within randomly weighted
neural networks.
nThere exist supermasks that create corresponding subnetworks for tasks to achieve good performance.

For each task 4, the model trains only a binary mask 6*
and does not train parameters in the network W itself.
qqAlthough the binary mask <# needs to be stored per task,
since it's binary and sparse, it requires only a small storage.
Then, output for task 5, .(3, >⨀<# ), can always retrieve
the correct subnetwork, without forgetting.

SupSup, https://arxiv.org/abs/2006.14769, Supermasks in Superposition, Wortsman et al., In Proc. of NeurIPS, 2020.

DEIM2023 Tutorial, March 6, 2023

20

Architecture>Parameter isolation | SupSup (2/2)<latexit sha1_base64="N8pMrD7xVuOx1V9es5WLkD9sNxw=">AAACDHicbVDLSgMxFM3UV62vqks3wSJUkDIjRd0IRTduhAr2AZ2xZNJMG5qZDMkdcSj9ADf+ihsXirj1A9z5N6aPhbYeCBzOOZebe/xYcA22/W1lFhaXlleyq7m19Y3Nrfz2Tl3LRFFWo1JI1fSJZoJHrAYcBGvGipHQF6zh9y9HfuOeKc1ldAtpzLyQdCMecErASO18IcXnOMCuYAEU8cMRbmBXdiTg6zvAruLdHhyalF2yx8DzxJmSApqi2s5/uR1Jk5BFQAXRuuXYMXgDooBTwYY5N9EsJrRPuqxlaERCpr3B+JghPjBKBwdSmRcBHqu/JwYk1DoNfZMMCfT0rDcS//NaCQRn3oBHcQIsopNFQSIwSDxqBne4YhREagihipu/YtojilAw/eVMCc7syfOkflxyTkrlm3KhcjGtI4v20D4qIgedogq6QlVUQxQ9omf0it6sJ+vFerc+JtGMNZ3ZRX9gff4AIF6ZLA==</latexit>nSupermask training with Edge-Popup
q
q

q

n

y = f x, W

Mt

This technique was proposed by another work.
The Edge-Popup alg. learns a score matrix 7, which has the same size as of 8 or 6, and computes
the mask via 6 = ℎ(7), where the function ℎ sets the top :% of entries in 7 to 1 and the rest to 0.
Simply, it keeps some (top :%) edges important to performance while pruning the remaining ones.

Drawbacksq

Neither forward nor backward transfer is possible, because each task has a completely separate
subnetwork.

DEIM2023 Tutorial, March 6, 2023

21

Architecture>Parameter isolation | CATn

CAT (Continual learning with forgetting Avoidance and knowledge Transfer) is the first workthat enables transfer among similar tasks by extending HAT.
qn

Evaluated with not only dissimilar tasks (e.g., CIFAR100 with 10 tasks),
but also mixed sequences that consist both dissimilar and similar tasks.

Only similar tasks are transferred

To detect similar tasks, two models are compared.
Transfer model actually tries to transfer each of previous task to a new task.
qReference model does NOT try to transfer any previous task.
If the transfer model outperforms the reference model, the task is regarded as similar.
qnThose detected similar tasks are then transferred via the attention
to the new task, which leads to better knowledge transfer.
Since dissimilar tasks are supposed to non-similar,
the HAT-like masking still prevents forgetting for them.

Same as HAT

CAT, https://arxiv.org/abs/2112.10017, Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks. Ke et al., In Proc. of NeurIPS, 2020.

DEIM2023 Tutorial, March 6, 2023

22

Architecture>Parameter isolation | PRMnnn

PRM (Partially Relaxed Masks) exploits task similarity in a different way than CAT, using HAT's
scheme as well.
PRM finds that task similarity can be computed via clustering of task embeddings that control thebinary masks.
When training, basically following HAT's algorithm,
PRM accumulates only masks of dissimilar tasks,
which promotes knowledge transfer among similar tasks.
<latexit sha1_base64="jN+dKFom1K0PLbLrolWIoutBar4=">AAACJXicbVDLSgMxFM3UV62vqks3wSJU0DJTirpQKLpxWcE+oFNLJs20oZkHyR2xDP0ZN/6KGxcWEVz5K2amXdTqgcDJOeeS3OOEgiswzS8js7S8srqWXc9tbG5t7+R39xoqiCRldRqIQLYcopjgPqsDB8FaoWTEcwRrOsObxG8+Mql44N/DKGQdj/R97nJKQEvd/CXpiofYFgzDqTXGV9j2yBPWdxeKOPUS/QTPxcpjbEveH8BxN18wS2YK/JdYM1JAM9S6+YndC2jkMR+oIEq1LTOETkwkcCrYOGdHioWEDkmftTX1icdUJ063HOMjrfSwG0h9fMCpOj8RE0+pkefopEdgoBa9RPzPa0fgXnRi7ocRMJ9OH3IjgSHASWW4xyWjIEaaECq5/iumAyIJBV1sTpdgLa78lzTKJeusVLmrFKrXszqy6AAdoiKy0DmqoltUQ3VE0TN6Re9oYrwYb8aH8TmNZozZzD76BeP7BzOso0A=</latexit>

HAT:
PRM:

1
at
= max
l
1
at
= max
l
<latexit sha1_base64="ybH9+5ijtdL0xYGUAtX4npboM1Q=">AAACOXicbVBNSyNBEO1Rd9Wou1GPXgqDoLCGmUXUiyDqwWMEo0Imhp5OT9LY0zN014hhzN/y4r/wJnjxoIhX/4DdSQ5+FTS8fq8eVfWiTAqDvn/vjY1P/Po9OTVdmpmd+/O3PL9wYtJcM15nqUz1WUQNl0LxOgqU/CzTnCaR5KfRxb7TTy+5NiJVx9jLeDOhHSViwShaqlWu0ZY8L0LJAdeDPuxAmNArsP8YV8FpAq5BQCiUU7DLqCwO+s7j+v85aeiFUItOF9da5Ypf9QcF30EwAhUyqlqrfBe2U5YnXCGT1JhG4GfYLKhGwSTvl8Lc8IyyC9rhDQsVTbhpFoPL+7BimTbEqbZPIQzYj46CJsb0ksh2uu3NV82RP2mNHOPtZiFUliNXbDgoziVgCi5GaAvNGcqeBZRpYXcF1qWaMrRhl2wIwdeTv4OT/9Vgs7pxtFHZ3RvFMUWWyDJZJQHZIrvkkNRInTByQx7IE3n2br1H78V7HbaOeSPPIvlU3ts7CECqqg==</latexit>⇣2
atl 1 , atl⌘ail |i 2 Dlt 1 , i  t

1

Clustering of task embeddings tells task similarity.

where A"#%& is the set of dissimilar tasks for task 5 − 1 in layer ;.
<latexit sha1_base64="bkhojhR+z9wXBs4B03OkrL3Sxis=">AAACS3icbVBNb9NAEF2nBEL4qClHLiMiRJCSyEYR9FKpggvHViJJpdhE68062Xa9tnbHSJHl/9dLL9z4E730AEIcunF8CAkjrfbNmzczuy/KpDDoeT+dxsGD5sNHrcftJ0+fPT90XxyNTZprxkcslam+iKjhUig+QoGSX2Sa0ySSfBJdfV7XJ9+5NiJVX3GV8TChCyViwShaauZGi1khe+KyfAsnEEge4xR86EOQCFWlXaCVovxW2Byw75e9iur7vcstEgItFkt8V98hVIPBTp65HW/gVQH7wK9Bh9RxNnN/BPOU5QlXyCQ1Zup7GYYF1SiY5GU7yA3PKLuiCz61UNGEm7CovCjhjWXmEKfaHoVQsdsdBU2MWSWRVSYUl2a3tib/V5vmGB+HhVBZjlyxzaI4l4AprI2FudCcoVxZQJkW9q3AllRThtb+tjXB3/3yPhi/H/gfBsPzYef0U21Hi7wir0mX+OQjOSVfyBkZEUauyS35RX47N86d88f5u5E2nLrnJfknGs17DVWwjw==</latexit>

à (HAT & PRM) Parameter update:

0
gl,ij
=

h

1

min

⇣

1 t 1at, al 1,j
l,i

⌘i

gl,ij

PRM, https://dl.acm.org/doi/abs/10.1007/978-3-031-05933-9_29, Partially Relaxed Masks for Knowledge Transfer Without Forgetting in Continual Learning, Konishi et al., In Proc. of PAKDD, 2022.

DEIM2023 Tutorial, March 6, 2023

23

Categories of CL methodsnRegularizationq

ConceptnqnqPenalize changes to important parametersnqFunction regularizationqq

Let each task monopolize a own sub-network inside the whole network

ApproachesnModel decompositionnModular network

LwF

q

Weight regularizationq

ConceptnApproachesnArchitecture

EWC
UCL

q
q
n

q

Replay
q

n

q

q
q

Concept
Store some previous samples to be learned together with the learning of a
new current task

Approachesn

GEM

Generative replay
q

DGR

q

Conceptnq

Control the loss such that forgetting can be alleviatedApproachesnGradient projection
q

n

CUBER

Loss landscapeq

DEIM2023 Tutorial, March 6, 2023

HAT
SupSupCAT
PRM

Optimization

Experience replayqnnProgressive Networks
PathNetParameter isolation
q

n

CCLL

Stable-SGD

24

Optimization>Gradient projection | CUBERnCUBER (ContinUal learning method with Backward knowlEdge tRansfer) theoretically analyzes theconditions under which updating the learned model of old tasks could lead to backward transfer.

nSpecifically, each layer characterizes the task correlation. It detects a regime based on theirorthogonality of parameter gradients.
qqq

Regime 1) There is little knowledge transfer or interference
Regime 2) Forward transfer is possible
Regime 3) Backward transfer is possible.

In all regimes, the model for a new task is updated based onorthogonal projection for the knowledge protection.
e.g., For regime 1,∇ℒ" 0 ← ∇ℒ" 0 − Proj& ! (∇ℒ" (0)),

Proj& ! defines the projection on the input subspace 6% of task 7,
i.e., Proj& ! 8 = 89 % (9 % )′ where 9 % is the bases for 6% .

CUBER, https://arxiv.org/abs/2211.00789, Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer, Lin et al., In Proc. of NeurIPS, 2022.

DEIM2023 Tutorial, March 6, 2023

25

Optimization>Loss landscape | Stable-SGDnRather than dedicating an algorithm, Stable-SGD enables SGD
to find a flat local minimum by adapting the factors in training regime,
such as dropout, learning rate, and batch size.

nIt hypothesizes that the wider the convergent points are, the less forgetting happens.
The following are suggested.
qqq

Dropout reguarization that widens the local minimum (better)
Large initial learning rate with exponential decay schedule
Small batch size

Stable-SGD, https://arxiv.org/abs/2006.06958, Understanding the Role of Training Regimes in Continual Learning. Mirzadeh et al., In Proc. of NeurIPS, 2020.

DEIM2023 Tutorial, March 6, 2023

26

Agenda1.

What is TIL?

2.

Categorization in CL& Deep dive into representative methods

3.

Future directions

DEIM2023 Tutorial, March 6, 2023

27

Review of current situation on TIL
n

In TIL, most effective approaches are "architecture"-based ones, such as PathNet, HAT, and
SupSup in terms of preventing forgetting.
q

As they enable each task to monopolize its own sub-network, they achieve no-forgetting learning.

q

At the same time, however, this training paradigm limits knowledge transfer among tasks.

qRecent studies (e.g., CAT and PRM) have tackled TIL problem in which not only preventing
forgetting but also promoting transfer are pursued.

DEIM2023 Tutorial, March 6, 2023

28

What is next for TIL?
n

In TIL, the forgetting problem itself has already been resolved by HAT or SupSup.
But, those approaches' paradigms may also induce another problem.
For example,
q Limited knowledge transfer caused by less shared parameters
q Reduced learnining capacity caused by using only allocated parameters
Limited research has been done to balance these.

nConnecting TIL with other ML research areas (e.g., self-supervised learning, meta-learning,use of pre-trained models, or federated learning) can be another way to go.

DEIM2023 Tutorial, March 6, 2023

29

Takeawayn

Not limited to TIL, continual learning methods are grouped into several types.
In this talk, 4 categories were introduced.
qqqq

Regularization – suppresses the change of important parameters.
Replay – store some examples of previous tasks to be used in learning the new current task.
Architecture – let each task monopolize a sub-network.
Optimization – controls the loss such that forgetting can be alleviated.

n

In particular, architecture methods are typically most effective for TIL, which can achievelearning with no forgetting. Representative methods are HAT and SupSup.

n

However, those approaches have induced other problems, e.g., limited knowledge transfer
or reduced learning capacity for new tasks. Developing TIL methods that can address such
issues while preventing forgetting still remains an interesting topic for TIL.

DEIM2023 Tutorial, March 6, 2023

30

Class Incremental Learning

Gyuhak Kim
University of Illinois Chicago
https://k-gyuhak.github.io

Overview
Motivation, application, definition■ Challenges■ Problem setups - online, blurry task, offline■ Proposed methods■ Evaluation protocols and datasets■ A theoretical guidance on solving CIL■ CIL using pre-trained models■ Future directions■

DEIM2023 Tutorial, March 6, 2023

2

Class-Incremental Learning (CIL)
Different learning problems in continual learning
(CL):

■

❑

Task incremental learning (TIL)❑

Class incremental learning (CIL)❑

Domain incremental learning (DIL)

❑

etc.

DEIM2023 Tutorial, March 6, 2023

3

Motivation
Task 1

Task 2

Task 5…

Learning new tasks without forgetting the previous knowledge■❑

Limited resource

DEIM2023 Tutorial, March 6, 2023

4

Application
Continual object recognition■❑

https://www.youtube.com/watch?v=Bs3tSjwbHa4

Lorenzo Pellegrini, Gabriele Graffieti, Vincenzo Lomonaco, and Davide Maltoni. Latent replayfor real-time continual learning. 2019

DEIM2023 Tutorial, March 6, 2023

5

Definition

Task 1

Task 2

Task 5…

■Given test sample x, what is its class?
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, Bing Liu. A Theoretical Study on Solving Continual Learning. In NeurIPS, 2022.

DEIM2023 Tutorial, March 6, 2023

6

Types of CIL Problems - Online
The system can observe the training samples only once■❑

e.g., single epoch

Task 1

Task 2

Task 5…

❑

Task transition vs. stream

Tyler L. Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. In CVPR Workshops, June 2020

DEIM2023 Tutorial, March 6, 2023

7

Types of CIL Problems - Online (blurry task)
There is no clear task transition■❑e.g., stream of data. Single sample per batch

…
Time
■

The CIL definition is still valid
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In CVPR, 2021.

DEIM2023 Tutorial, March 6, 2023

8

Types of CIL Problems - Offline
The system can observe the training samples multiple times■❑❑

e.g., multiple epochs during training v.s. single epoch
Enough time to train the system and collect data
Task 1

Task 2

Task 5…

■

Some variants (e.g., blurry task) makes less sense
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, Bing Liu. A Theoretical Study on Solving Continual Learning. In NeurIPS, 2022.

DEIM2023 Tutorial, March 6, 2023

9

Proposed Methods - 4 Categories■

Regularization methods■

Replay-based methods■

Pseudo-replay methods■

Parameter-isolation methods

DEIM2023 Tutorial, March 6, 2023

10

Proposed Methods - Regularization Methods

Used for CIL (but any CIL can be used for TIL)
■ Exemplar-free■ For task k, minimize■

DEIM2023 Tutorial, March 6, 2023

11

Proposed Methods - Regularization Methods■

Find important parameters in performing the previous tasks■

Preserve the input-output mapping (e.g., knowledge distillation)❑

Examples of the loss function L are KL-divergence and MSE

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, and Others. Overcoming catastrophic forgetting in neural networks. In PNAS, 2017.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pp. 3987–3995, 2017.
Zhizhong Li and Derek Hoiem. Learning Without Forgetting. In ECCV, pages 614–629. Springer, 2016.
DEIM2023 Tutorial, March 6, 2023

12

Proposed Methods - Regularization Methods
Regularize the features■ Denote the feature extractor by h. Then regularize the features as■

Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In CVPR, 2021.

DEIM2023 Tutorial, March 6, 2023

13

Proposed Methods - Replay-Based Methods
Used for CIL (but any CIL can be used for TIL)
■ Memory buffer■ For task k, minimize■■

The regularization R can be one of the regularization methods discussed in
the previous slide

DEIM2023 Tutorial, March 6, 2023

14

Proposed Methods - Replay-Based Methods
Since the memory buffer is very small, it’s important how
to leverage it more effectively■

Which samples to replay?
❑ Which samples to save?
❑

DEIM2023 Tutorial, March 6, 2023

15

Proposed Methods - Replay-Based Methods
Which samples to replay?

■❑

For a batch of training samples, select samples that give the maximum interference■

Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, and Others. Online continual learning with maximal interfered retrieval. In NeurIPS. 2019.

DEIM2023 Tutorial, March 6, 2023

16

Proposed Methods - Replay-Based Methods
Which samples to save?

■Reservoir sampling [Jeffery et al., 1985]. Select random samples from stream❑ Randomly select m samples per class [Chaudhry et al., 2019]❑ Nearest-mean classifier. Select the m training samples that best approximate the mean of training
samples [Rebuffi et al., 2017]❑

Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 1985
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, and Others. Continual learning with tiny episodic memories. 2019.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In CVPR, 2017.

DEIM2023 Tutorial, March 6, 2023

17

Proposed Methods - Replay-Based Methods
Due to sample imbalance between memory buffer and current training data,
the system is biased towards recent tasks■❑

Need to correct the bias

Bias toward later tasks■❑

Samples of earlier tasks are predicted as classes in later tasks

Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, 2019.

DEIM2023 Tutorial, March 6, 2023

18

Proposed Methods - Replay-Based Methods■

The weights w and biases b for old and new classes in the last layer:■

Correct it by cosine normalization and knowledge distillation

Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In CVPR, 2019.

DEIM2023 Tutorial, March 6, 2023

19

Proposed Methods - Replay-Based Methods
Correct the bias in output values■❑

Find scaling

and shiftingparameters for the output of each task k to correct the bias❑

Find the parameters via optimization using the held-out samples in the memory buffer

Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, 2019.

DEIM2023 Tutorial, March 6, 2023

20

Proposed Methods - Replay-Based Methods
Find robust parameters to
perturbation (i.e., flat minima)■❑

One can achieve a better generalization via
flat local minima as flat minima is morerobust to shift (or perturbation)■

Regularize on sub-optimal output(perturbation) [Buzzega et al., 2020]

z’ is the logit of replay sample x’
obtained via parameter sub-optimal
parameter■

Liyuan Wang, Xingxing Zhang, Hang Su, Jun Zhu. A Comprehensive Survey of Continual Learning: Theory, Method and Application. Preprint, 2023
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In NeurIPS, 2020.

DEIM2023 Tutorial, March 6, 2023

21

Proposed Methods - Pseudo-Replay Methods

Use a classifier and a generator■ The generator replaces the memory buffer■

where
DEIM2023 Tutorial, March 6, 2023

22

Proposed Methods - Pseudo-Replay Methods
How to prevent forgetting in the generator?

■❑

DGR [Shin et al., 2017]❑

DGM [Ostapenko et al., 2019]. Protect via parameter-isolation method (e.g., HAT)

Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NeurIPS, pp. 2994–3003, 2017.
Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In CVPR, 2019.

DEIM2023 Tutorial, March 6, 2023

23

Proposed Methods - Parameter-Isolation Methods

Originally used for TIL, but can be used for CIL as well
■ Use task-specific parameters■ For task k, minimize■

DEIM2023 Tutorial, March 6, 2023

24

Proposed Methods - Parameter-Isolation Methods■

Hard attention to the task■

CIL prediction using TIL models. Simply concatenate the outputs

TIL

CIL
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML, 2018.
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, Bing Liu. A Theoretical Study on Solving Continual Learning. In NeurIPS, 2022.

DEIM2023 Tutorial, March 6, 2023

25

Evaluation
Evaluation metrics■❑

Average accuracy❑

Average incremental accuracy❑

Forgetting rate (i.e., backward-transfer)

DEIM2023 Tutorial, March 6, 2023

26

Evaluation
Popular datasets

■
❑

MNIST

❑

CIFAR-10❑

CIFAR-100❑

Tiny-ImageNet

DEIM2023 Tutorial, March 6, 2023

CIFAR-10

ImageNet

27

A Theoretical Guidance on Solving CIL
CIL is difficult because of inter-task class separation■❑■

The system needs to establish decision boundaries between the classes of the new task andthe classes of the previous tasks without previous task data

At inference, if the system knows which task the test instance belongs
to, CIL problem is reduced to TIL

Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, Bing Liu. A Theoretical Study on Solving Continual Learning. In NeurIPS, 2022.

DEIM2023 Tutorial, March 6, 2023

28

A Theoretical Guidance on Solving CIL
Out-of-Distribution (OOD) detection■

Assign a class if an instance belongs to one of the classes used in the training data (IND)❑ Reject if an instance does not belong to any of the n IND training classes❑■

For task 1, OOD classes are
Task 1OOD…

■

For task 2, OOD classes are
Task 1

Task 2OOD…

DEIM2023 Tutorial, March 6, 2023

29

A Theoretical Guidance on Solving CIL■

CIL problem can be decomposed into two subproblems: within-task prediction(WP) and task-id prediction (TP)WP (i.e., TIL)TP■The loss of CIL is bounded by that of WP (TIL) and OOD (TP)■A good performance of WP (TIL) or TP (or OOD) are necessary and sufficient fora good CIL
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, Bing Liu. A Theoretical Study on Solving Continual Learning. In NeurIPS, 2022.

DEIM2023 Tutorial, March 6, 2023

30

Evaluation■

Average accuracy

Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, Bing Liu. A Theoretical Study on Solving Continual Learning. In NeurIPS, 2022.

DEIM2023 Tutorial, March 6, 2023

31

CIL Using Pre-Trained Models■

Strong pre-trained models have been proposed, but most CIL methods are notleveraging the strong pre-trained models (except the ones in NLP)■

Using pre-trained models, the CIL performances are much better■

Need different methods when using pre-trained models■

Make sure that CL datasets are different from the pre-training dataset to ensure
there is no information leakage from pre-training to CL training■

Fine-tune the entire network, adapters, or prompts

DEIM2023 Tutorial, March 6, 2023

32

CIL Using Pre-Trained Models
Make sure that CL datasets are different from the pre-training
dataset to ensure there is no information leakage from pre-training
to CL training■

We use CIFAR-10, CIFAR-100, and Tiny-ImageNet for CL experiment❑ We removed 389 classes from the original 1000 classes in ImageNet that are
similar/identical to the classes in CIFAR-10, CIFAR-100, or Tiny-ImageNet.
❑ We pre-train a network with the remaining subset of 611 classes of ImageNet❑

Failing to remove such classes seriously affects the results■❑

The accuracy of L2P [Wang et al., 2022] method on C100-10T drops from 84 to 62
Gyuhak Kim, Zixuan Ke, Bing Liu. A Multi-Head Model for Continual Learning via Out-of-Distribution Replay. In CoLLAs, 2022.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, and Others. Learning to prompt for continual learning. In CVPR, 2022

DEIM2023 Tutorial, March 6, 2023

33

Future Directions■

A lot of CIL techniques are proposed, but they are mostly
empirical. Need more theoretical understanding. We made anattempt in (Kim et al., NeurIPS-2022)■

Need more rigorous study on using pre-trained models in CL■

Can we do backward transfer for parameter-isolation
methods?

DEIM2023 Tutorial, March 6, 2023

34

Domain Incremental Learning andContinual Learning in NLP
Zixuan Ke
University of Illinois Chicago
https://vincent950129.github.io/Plann A quick summary of TIL and CIL
n Another setting: Domain-incremental Learningn What we have learned so far
n Continual learning of NLP Tasksn Conclusion and Future work

DEIM2023 Tutorial, March 6, 2023

n2

Roadmap
Continual
Learning

Settings
TIL

CIL

Desiderata

NLP (LM)

Challenges

Tasks

Approaches

DIL*LM: Language Model
DEIM2023 Tutorial, March 6, 2023

3

Settings
Notations:n We have known that

n

CIL

TIL
n

Assumption: task-IDs areavailable in both training andtestingn

Goals:

DEIM2023 Tutorial, March 6, 2023

nAssumption: task-IDs areavailable only in training. In
testing, a test instance fromany class may be presentednGoals:nn

T is the last task
learned and
I is the label space oftask tFor any given tasks, t
and t’,4Plan
n A quick summary of TIL and CIL
n Another setting: Domain-incremental Learningn What we have learned so far
n Continual learning of NLP Tasksn Conclusion and Future work

DEIM2023 Tutorial, March 6, 2023

n5

Domain-incremental Learningn Assumption
qClass labels are assumed to be the same for all tasks

n E.g.,A sequence of sentiment classification tasks (all tasks have thesame labels {positive, negative, neutral})q Generation tasks in NLP (all tasks depend on the language model
head which has the same number of vocabulary tokens)q

DEIM2023 Tutorial, March 6, 2023

6

Domain-incremental Learning
n Goalq

Learn a functionn Yes, it is a special case of TIL
q

A DIL problem can always be solved with a TIL methodn However, you may see some existing DIL systems do not use

task-ID in testing,qThis is because the tasks are very similar (task-ID does not matter)or very dissimilar (task-ID is easily predicted using the test data)

DEIM2023 Tutorial, March 6, 2023

7

Domain-incremental Learningn Now we know all the 3 settings, why are they important?

Together with TIL and CIL, they constitute the three fundamental
settings of continual learning.
q When attempting to leverage continual learning, the initial step is toestablish the appropriate setting.
q Different settings are for different applications and lead to different
specific challenges and approachesq

DEIM2023 Tutorial, March 6, 2023

8

Roadmap
Continual
Learning

Settings
TIL

CIL

Desiderata
DIL

NLP (LM)

Challenges

Tasks

Approaches

Before we go to NLP, let’ssummarize what we have known
about continual learning so far*LM: Language Model
DEIM2023 Tutorial, March 6, 2023

9

Continual learning
n General Desiderata (TIL, DIL, CIL)q

Not suffer from catastrophic forgetting (CF)nqAchieve positive forward knowledge transfer (forward KT)nqi.e., perform reasonably well on what has been learnedi.e., old knowledge helps new task

Achieve positive backward knowledge transfer (backward KT)
ni.e., relevant new task helps old tasks

DEIM2023 Tutorial, March 6, 2023

10

Roadmap
Continual
Learning

Settings
TIL

CIL

DIL

NLP (LM)

Desiderata
Prevent
Forgetting

Challenges

Tasks

Approaches

Knowledge
Transfer

ForwardBackward*LM: Language Model
DEIM2023 Tutorial, March 6, 2023

11

Continual learning
n ChallengesqStability-plasticity (TIL, CIL, DIL)nqTransfer-interference (mostly in TIL and DIL)nnqPreserving the learned knowledge v.s. accurately learning new experiences

Knowledge transfer vs. knowledge interference(increase parameter-sharing vs. reduce parameter-sharing)

Task separation (mostly in CIL, and DIL w/o ID)nEstablish decision boundaries between the current task and previous tasksq

In learning the current task, the learner does not see previous or future task data
*Task separation a.k.a. inter-task class separation in CIL

DEIM2023 Tutorial, March 6, 2023

12

Roadmap
Continual
Learning

Settings
TIL

CIL

DIL

NLP (LM)

Desiderata

Prevent
Forgetting

Knowledge
Transfer

Forward

Challenges
Stabilityplasticity

Transferinference

Tasks

Approaches

Task
separation

Backward*LM: Language Model
DEIM2023 Tutorial, March 6, 2023

13

Plan
n A quick summary of TIL and CIL
n Another setting: Domain-incremental Learningn What we have learned so far
n Continual learning of NLP Tasksn Conclusion and Future work

DEIM2023 Tutorial, March 6, 2023

n14

Continual Learning in NLP
nThe world is dynamic, and time is irreversible, new emerging{classes/domains/tasks/topics/events} keep coming

DEIM2023 Tutorial, March 6, 2023

15

Continual Learning in NLP
nThe world is dynamic, and time is irreversible, new emerging{classes/domains/tasks} keep comingnWe do not want to re-train for many reasonsn

We do not want to train separate models for many reasonsn

We want the model accumulates knowledge in its lifetime without
forgetting what it has learned

DEIM2023 Tutorial, March 6, 2023

16

Continual Learning in NLP
n “Task”q

In terms of NLP, what is a task?

n Formulationq

How are some NLP tasks learned continually?

n ApproachesqWhat are some popular approaches in CL for NLP?

DEIM2023 Tutorial, March 6, 2023

17

Continual Learning in NLP
Tasks
Pretraining

End-task
learningn

Domainadaptive Pretrainingnnn

Most popular
Each task is an end-task
which can be any NLP problem (e.g.,
sentiment classification, NER, summarization)

Each task uses a domain
corpus to adapt an LM to adomain/topic*Domain-adaptive pre-training (DAPT) a.k.a. post-training or pre-finetuning
DEIM2023 Tutorial, March 6, 2023

18

Formulation of
NLP Problems
Large #NLP problems belong to
DIL
n Many problems are converted
to generation tasks in NLP via
promptingContinual learning for NLP has
been growing rapidly, this table
keeps expanding.

n

n

DAPT

#Papers in Top NLP Conferences
2022

DAPT

2021
2020
2019
0

10

20
EMNLP

30
ACL

DEIM2023 Tutorial, March 6, 2023

NAACL

40

50

*DAPT: Domain-adaptive pre-training
19Roadmap
Continual
Learning

Settings
TIL

CIL

DIL

NLP (LM)

Desiderata

Prevent
Forgetting

Knowledge
Transfer

Forward

Challenges
Stabilityplasticity

Backward

Transferinference

Approaches

Tasks
Task
separation

GeneralPretraining

Pretraining

End-task
learning

Domainadaptive
Pre-training*LM: Language Model
DEIM2023 Tutorial, March 6, 2023

20

Approaches for Forgetting Prevention
n We have known the three popular families in continual learning toprevent forgettingn Add penalty when training

new taskn Memorize a subset of old

Regularizationbased

samples and train together with the
new task

Replay-basedn Allocate specific parameters to

Parameterisolation

specific tasks
DEIM2023 Tutorial, March 6, 2023

21

Approaches for Forgetting Prevention
n A simple example in NLP: Adapter-CL

DIL

End-task
learning

Parameter-isolation*adapter: Adding a small
number of parameters to thepre-trained LM. In fine-tuning,
only the adapters aretrainable.

µ! refers to the adapter for task !
Adapter-CL uses separate adaptersfor different tasks, which is a naïve
way to prevent forgetting.
It can do well in forgetting
prevention, but cannot achieve theother desideratum (knowledgetransfer)
Madotto et al., Continual learning in task-oriented dialogue systems, EMNLP 2020
DEIM2023 Tutorial, March 6, 2023

22

Approaches for Knowledge Transfer
n Approaches for preventing forgetting mainly try to reduce parameter-

sharingn This is not enough for knowledge transfer, which needs to allow someused parameters to be updated or shared.
n Compute task similarity and share theparameters for similar tasksn Learn the transferrable knowledge using

replay data (e.g., meta-learning)
n Soft-masking the backward pass, but keep

the forward pass untouched
DEIM2023 Tutorial, March 6, 2023

Similaritybased[1]Replaybased[2]*Same name,but with different
goals

Soft-maskbased[1]: Ke et al., Achieving forgetting prevention and knowledge transfer in continual learning, NeuIPS 2021[2]: d’Autume et al., Episodic Memory in Lifelong Language Learning, NeurIPS 201923

Continual Domain-adaptive Pre-training
DIL
nSetting

Domainadaptive Pretraining

Soft-maskbased

Continual domain-adaptive pre-training of a sequence of domains
without accessing the data that was used in original pre-training or
previously learned domains
q End-task doesn’t know its domain belonging
q

n

Goals
CF prevention
q KT (backward and forward)qnApproachq

DAS
Ke et al., Continual learning of language models, ICLR 2023

DEIM2023 Tutorial, March 6, 2023

24

MLM Head
Add & Layer Norm(A) Continual Domain-adaptive pre-training
×L

+
FFN
Add & Layer Norm

Given a pre-trained LM, continually domainadaptive pre-training a sequence of domains

Restaurant Phone Camera+
Attention
Hidden States

Classification Head
Add & Layer Norm(B) Individual Fine-tuning
×L

End-tasks

After continual learning, the domain-adaptive pretraining performance is evaluated by end-tasks+
FFN

ASC-Restaurant

Add & Layer Norm

ASC-Phone+

ASC-Camera

Attention
Hidden States

Each end-task corresponding to one domain andhas its own training and testing set. It is trained
individually and will not affect the domain-adaptive
pre-training

ASC: Aspect Sentiment Classification

25

Continual Domain-adaptive Pretraining
Pre-trained LM

1) Detect importance of units forgeneral and domain knowledge
2) Soft-mask the important units
when training new tasks/domains!=1

No
training

Key ideas:

ImportanceComputation

3) This can prevent forgetting andallow knowledge transfer
Key challenges:

Network
Training

Soft-masking
Backward

1) How to detect importance for thetwo types of knowledge
2) How to soft-mask

Ke et al., Continual learning of language models, ICLR 2023
DEIM2023 Tutorial, March 6, 2023

26

Importance Computation'!"!

…
Transformer
Layer !
Forward×L

To compute theimportance…

…

Element-wise
multiplication

×L(#
Transformer
Layer !
Forward!$ is the virtual parameters.
Each virtual parameter "$,& in !$corresponding to an attention
head or neurons (units)
It is initialized as all 1’s, andhas its gradient but will neverchange.
Why? We only use its gradient
to compute importancen27

Importance Computation#"#"

For domain knowledge,#$%&' = #"#"

#$%&'
…

…
…

×L

Transformer
Layer "
Forward(*)

To compute theimportance×L$(
Transformer
Layer "(*)*#$%&' (,/ , ./ )
/
).! =
*.!
(*)

%( =

,
∑- |)/.! |-ForwardUse absolute gradient to indicate importance[1]DEIM2023 Tutorial, March 6, 2023

Michel et al. Are sixteen heads really better than one? NeurIPS, 2019.

28

Importance Computation
(,)

(,)

0
0#"

Pre-trained LM,0#$%&' = KL(0#"(,/ ), 0#"(,/ ) ),
0#"Pre-trained LM

×LTransformer
Layer "For general knowledge, we leverage therandom dropout in standard Transformer
×L

Transformer
Layer "

(,)

,/

(,)

,/

KL: How different are the two representations?
1312/ 1412: Transformer with different dropouts(,),/ : We only use the first domain data because we wantto keep the pre-trained general knowledge
With the new #$%&' , we can use the absolute gradient toindicate the importance (same as in domain knowledge)
DEIM2023 Tutorial, March 6, 2023

Random dropout introduces random
noise. Given the same input, thedifference between the representations
with different random noise indicates therobustness.
The units that are important to therobustness is likely to be important to thegeneral/pre-trained knowledge because
its change will cause the pre-trained LMchange a great deal

Ke et al., Continual learning of language models, ICLR 2023

29

Soft-masking
First, we normalize the importance sothat they are comparable
×L
Normalize

EMax

(%)
{"# })*(%'($#(⩽$%&)

1 − ,!

%#$
Transformer
Layer "

Backward

(5)

%(

(5)

= Tanh(Norm(%( ))Second, we accumulate the importance(⩽*7,)%((*7,)= EMax({%(

(*70)

, %(

Third, we soft-mask the gradient (only in backward pass)(⩽*7,))′( = 1 − %(⊗ )(Ke et al., Continual learning of language models, ICLR 2023
DEIM2023 Tutorial, March 6, 2023))

30

Results

Overall end-task performance (final performance)

No DAPT
DAPT
DAPT
DAPT

DAPT
NCL DAPTSoTA DAPTDAS

w/o DAPT < DAPT < DAS
n +forgetting rate in NCL: it does suffer from forgettingn Regularization-based methods (KD, EWC) and replay-based method (DER++) are all worse: focus on CF prevention is not enoughn Parameter-isolation method (HAT) preforms much worse: the full LM is needed for domain-adaptive pre-trainingn Methods that tries to perform both KT and CF (DEMIX, BCL, CLASSIC): all weaker than DAS
nKe et al., Continual learning of language models, ICLR 2023
DEIM2023 Tutorial, March 6, 2023*Naïve continual learning (NCL):
31continual learning without any specific techniqueApproaches for Task Separationn Besides forgetting and knowledge transfer, another challenge in

CIL and DIL w/o ID is the task separationn

Use heuristic methods likeentropy/perplexity to detecttask ID at test time

Postprediction[1,2]nUse replay data to establish theboundary

Replay-based

DEIM2023 Tutorial, March 6, 2023

[1]: Gururangan et al., DEMix Layers: Disentangling Domains for Modular Language Modeling, NAACL 2022
32[2]: Li et al., Overcoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation, NAACL
2022

Approaches for Task Separationn A simple example in NLP: Adapter-CL

DIL

End-task
learning

Post-predictionµ! refers to the adapter for task !
Adapter-CL uses separate adaptersfor different tasks, but how to know
which one to use in testing?
Perplexity

Madotto et al., Continual learning in task-oriented dialogue systems, EMNLP 2020
DEIM2023 Tutorial, March 6, 2023

33

Approaches for Task Separation
DIL

LFPT5

n
n

End-task
learning

Replay-based

Pre-trained LM (T5) serves as both the problem solver and generator
When a new task comes, T5 first generates the old task data and then trains the pseudo data and newtask altogether. Since some previous data is available, the decision boundary is easier to establish
Qin et al. A unified framework for lifelong few-shot language learning based on prompt tunning of T5. ICLR, 2022.

DEIM2023 Tutorial, March 6, 2023

34

Roadmap
Continual
Learning

Settings

Desiderata

NLP (LM)

Challenges

Tasks

Forgetting
Prevention
Regularization-basedReplaybased

Parameterisolation

Approaches
Knowledge
Transfer
Similaritybased

Task
separation
Replaybased

Softmask
based

Replaybased

Postprediction*LM: Language Model
DEIM2023 Tutorial, March 6, 2023

35

Plan
n A quick review of we have talked
n Another setting: Domain-incremental Learningn Continual learning of NLP Tasksn Section summary and future work

DEIM2023 Tutorial, March 6, 2023

36

Continual
Learning
Settings

n
n
n

TIL
CIL
DIL

Challenges

Desiderata

n

n

n

Prevent
Forgetting
ForwardKnowledge
Transfer
Backward
Knowledge
Transfer

DEIM2023 Tutorial, March 6, 2023

NLP (LM)nnnStabilityplasticity
TransferinferenceTask
separation

Approaches

Tasks
PretrainingnnGeneral Pretraining
Domainadaptive Pretraining

End-task
learningnnnForgetting
Prevention

Regularization-basedReplay-based
Parameterisolation

n

n

n

Knowledge
Transfer
Similaritybased
Replaybased
Soft-mask
based

Task
separationnn

Replaybased
Postprediction37n Knowledge transfer is the major issue in TIL and DILq

A task sequence can consist of a combination of similar anddissimilar tasks[1]n Forgetting and task separation/discrimination are major

issues for CIL and DIL w/o IDqTask separation has been proved to be related to OOD detection,
but the accuracy is still far from the upper bound[2]DEIM2023 Tutorial, March 6, 2023

[1]: Ke et al., Continual learning of a mixed sequence of similar and dissimilar tasks, NeurIPS 2020[2]: Kim et al., A Theoretical Study on Solving Continual Learning, NeurIPS 2022

38

n Continual learning of language models (LM) (e.g., domain-

adaptive pre-training) is still in its infancyq

How to better preserve the general knowledge? DAS is an initialattempt, but it is still limited.

n Scalabilityq

How the network capacity issue affects the performance and how to
alleviate the issue effectively are also unclear.

n Temporal continual learningq

How to keep the LM up-to-date? Everything changes with time.

DEIM2023 Tutorial, March 6, 2023

39

Thank you
n We have benchmarked many SoTA

baselinesq

For continual end-task learningnq

https://github.com/ZixuanKe/PyContinual

For continual domain-adaptive pre-trainingnhttps://github.com/UIC-Liu-Lab/ContinualLM

n More details

Survey: Continual Learning of Natural
Language Processing Tasks: A Survey(Preprint)
q https://vincent950129.github.io/qDEIM2023 Tutorial, March 6, 2023

40

Open-World Continual Learning
Bing Liu
University of Illinois Chicago
https://www.cs.uic.edu/~liub/Closed-world and open-world(Fei et al, 2016; Shu et al., 2017)n Traditional machine learning:nnTraining data: Dtrain with class labels !train = {"1, "2, ..., "!}.
Test data:
Dtest, !test Í {"1, "2, ..., "!}n Closed-world:nClasses appeared in testing must have been seen in training, nothing new.

n Open-world:n!test Í !train!test - !train ¹ f

There are unseen classes in the test data, out-of-distribution (OOD).

n Open-world continual learning (autonomous learning)nn(1) detect novel/new objects (OOD detection) and(2) incrementally learn the new objects (continual learning).
Fei, and Liu. Breaking the Closed World Assumption in Text Classification. NAACL-HLT 2016

ATAL, Jan 7, 2023

37

Open-world CL - a greeting bot in a hotel(Chen and Liu, 2018; Liu et al., 2021)
n See an existing guest.
nNovelty detection = out-of-distribution(OOD) detection

Bot: “Hello John, how are you today?”n See a new guest - recognize he/she is new (OOD and create a new task)nnn

Bot: “Welcome to our hotel! What is your name, sir?”
Guest: “David”
Bot learns to recognize David automatically(get class label)take pictures of Davidq learn to recognize David(get training data)(continual learning)q(got class label: David)

n See David next time.
n

Bot: “Hello David, how are you today?”(use the new knowledge)

Liu, Robertson, Grigsby, and Mazumder. Self-Initiated Open World Learning for Autonomous AI Agents. AAAI Spring Symposium, 2022
Chen and Liu. Lifelong machine learning. Morgan & Claypool. 2015, 2018.
38
DEIM2023 Tutorial, March 6, 2023

Example (cont.)n

In a real hotel, the situation is much more complex.
q

How does the bot know that the novel object is a new guest?
n

Is the object a person, an animal, or a piece of furniture?
qqneeds to use existing knowledge to characterize the novel object!

Different characterizations require different responses (adaptation or
accommodation strategies)? E.g.,nnn

If it looks like an animal, report to a hotel staff.
If it looks like a policeman, do nothing
If it looks like a hotel guest (with luggage), ask for his/her name: “Welcome to ourhotel! What is your name, sir?” and learn to recognize him/her

Liu, Robertson, Grigsby, and Mazumder. Self-Initiated Open World Learning for Autonomous AI Agents. AAAI Spring Symposium, 2022 Chen and Liu.
Lifelong machine learning. Morgan & Claypool. 2015, 2018.
DEIM2023 Tutorial, March 6, 2023

39

Novelty characterization and adaptationn

Characterization: a description of the novel object based on theagent's existing knowledge.
nCharacterization at different levels of details => more or less precise responses.
qqnAdapting to novelty: a pair (Characterization, Response)n

Response: According to characterization, formulate a specific course of actions to
respond to the novelty, e.g.,
q
q

n

n

Often done based on similarity:§ E.g., it looks like an animal (general), or it looks like a dog (more specific).
Attributes/properties: e.g., a moving object, speed and direction of moving.

If novel object looks like an animal (characterization), report to hotel staff (response).
If cannot characterize, take default action (e.g., do nothing)Enable continual learning

Risk assessment: each decision carries risks
Liu, Robertson, Grigsby, and Mazumder. Self-Initiated Open World Learning for Autonomous AI Agents. AAAI Spring Symposium, 2022.

DEIM2023 Tutorial, March 6, 2023

40

SOLA: Self-initiated Open-world continual Learning & Adaptation(Chen & Liu, 2018, Liu, 2020 Liu et al., 2022)

Blue lines: Existing
continual learning

Orange lines:
Learning after model
deployment(Open-world continual
learning)

Liu, Robertson, Grigsby, and Mazumder. Self-Initiated Open World Learning for Autonomous AI Agents. AAAI Spring Symposium, 2022
and Mazumder. Lifelong and Continual Learning Dialogue Systems: Learning during Conversation. AAAI-2021
DEIM2023 Tutorial, MarchLiu
6, 2023

41

SOLA example: natural language interface (NLI)nPerformance task: user asks the system (CML, like Siri and Alexa) toperform a task in NL, the system does it via API actionsnApproach: natural language to natural language matching (NL2NL)n

CML builds NLIs for API-driven applications semi-automatically.

n

To build a new NLI (or add a new skill to an existing NLI),nApplication developer writes a set Si of seed commands (SCs) in NL to represent each
API action i.
qn

SCs in Si are just like paraphrased NL commands from users to invoke i, but the objects to beacted upon in each SC are replaced with variables, the arguments of i.

When the system does not understand a command (novelty), it adapts and learns new(paraphrased) SCs from users interactively and continually.

Mazumder, Liu, Wang, and Esmaeilpour. An Application-Independent Approach to Building Task-Oriented Chatbots with Interactive Continual Learning.
NeurIPS-2020 Workshop on Human in the Loop Dialogue Systems (HLDS-2020)
DEIM2023 Tutorial, March 6, 2023

42

An example – Smart homen SmartHome:

API action:
SwitchOnLight(X1)nnSwitching on a light at agiven place X1

Let an SC be “put on light in X1" for this API,nnwhere X1 is a variable representing the argument of the API.

User command: “power on the light in the bedroom”n

It can be matched or grounded to this SC, where the grounded API arguments are {X1= ‘bedroom’}.

DEIM2023 Tutorial, March 6, 2023

43

Novelty detection, characterization, adaptation
Novelty detection: when CML cannot ground a user command, e.g.,
it cannot understand “turn off the light in the kitchen”
n Novelty characterization: which part of the command it understands andnwhich part it has difficulty based on similarity. E.g., it cannot ground “turn off”
nAdaptation (or accommodation):nn

Response: ask the user by providing some options (to collect ground-truth data)
Bot: Sorry, I didn’t get you. Do you mean to:option-1. switch off the light in the kitchen,
option-2. switch on the light in the kitchen, orContinual learning: learn a new SC. No issue with related commands in future.

DEIM2023 Tutorial, March 6, 2023

44

Risk considerationn CML manages risk in two waysq

Do not ask user too many questions in order not to annoy the user.
nqLearning can be used to assess each user’s tolerance.

When characterization is not confident, take the default action, i.e.,n

Ask the user to say his/her command in another way
qrather than providing a list of random options for user to choose from§ which can be annoying or make the user lose confidence in the system!

DEIM2023 Tutorial, March 6, 2023

45

Summary
Bing Liu
University of Illinois Chicago
https://www.cs.uic.edu/~liub/Summaryn Classical ML: isolated, closed-world, single-task learningn This tutorial covered the following

Lifelong/continual learning definitions
q Early research of lifelong learningq Task incremental learning (TIL)
q Class incremental learning (CIL)q Domain incremental learning (DIL)
q Continual learning in NLP
q Continual learning in the open worldq

DEIM2023 Tutorial, March 6, 2023

47

Summary: challengesnCatastrophic forgettingnnKnowledge transfernnnStill a major challenge, especially for class incremental learning (CIL)
Correctness and applicability of knowledge
What to transfer and how to transfer forward and backward

Open world continual learning is a bigger challengenqNovelty, characterization, adaptation, and continual learning

Goal: AI autonomy - the next generation AI.
nAutonomous learning agents will be built in restricted environmentsqInteracting with people, other agents, and the environment and learning by itself

DEIM2023 Tutorial, March 6, 2023

48

Thank You!
Q&A