Class-incremental Learning in CVPR’22

Gengwei Zhang
University of Technology Sydney

BackgroundContinual Learning / Incremental Learning / Life-long Learning• Training sets are provided sequentially: 𝑋1 , 𝑋2 , … , 𝑋𝑡 , …,each training set is typically called a task
• Model can only access to one training set 𝑋 at step 𝑡(a few memory samples from previous tasks are acceptable),while being evaluated on test set of all tasks.
-> requires model to maintain performance on old tasks when learning on new tasks

Background
Class-incremental Learning• Each training set corresponds to a label set 𝑌𝑡 ,𝑌𝑡 are incrementally increasing• Typically, 𝑌𝑖 ∩ 𝑌𝑗 = ∅ for any 𝑖 ≠ 𝑗• Test set contains 𝑌 = 𝑌1 ∪ 𝑌2 ∪ ⋯ 𝑌𝑡 ∪ ⋯Background

Three typical scenario of class-incremental learning
1. Training from scratch-> DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion
2. Training with a large initial phase-> Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning
3. Training with no sample memory available-> Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental LearningDytoxDytoxDyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

DytoxDynamic task token expansion• For each task, learn a task token 𝜃𝑡 ,the task token is first concatenate with the feature tokens• Obtain task-specific feature via task-attention (cross attention):• After obtain task-specific embeddings 𝐸 = {𝑒1 , 𝑒2 , … , 𝑒𝑡 },classification score for each task is obtained by

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

DytoxOthers:• Objectives• Classification scores are obtained separately-> requires classification scores to be well calibrated-> BCE rather than CE-> post-hoc fine-tuning

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

DytoxResults

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

DytoxAblations

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

CwD

CwD

Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental LearningCwD
Quantitative Analysis
First obtain feature covariance matrix of class 𝑐

Then obtain the eigen value 𝜆, and calculate

𝛼 represents the importance of the top-k feature dimensions
If 𝛼𝑘𝑐 is close to 1 even when k is small, then the top eigenvalues dominate.

Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning

CwD
Class-wise Decorrelation (CwD) objective
Make eigen value distribute uniformly by minimizing:

It can be approve that

Therefore, we have

Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning

CwD
EffectMimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning

CwD
Results

Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental LearningCwD
Ablations

Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning

DSR
Overview

Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning

DSR
Details

Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning

DSR
Results

Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning

DSR
AblationsSelf-Sustaining Representation Expansion for Non-Exemplar Class-Incremental LearningConclusion

How to handle the compatibility between different tasks?

-> using task-specific design (DyTox, DSR)-> increase the compatibility of previous representation (CwD)-> selective optimization (DSR)