New Insights for the Stability-Plasticity Dilemma in Online Continual Learning
Authors: Dahuin Jung
Dongjin Lee
Sunwon Hong
Hyemi Jang
Ho Bae
Sungroh Yoon
Venue: ICLR 2023 poster (2023)
Keywords: Continual Learning
Online Continual Learning
Catastrophic Forgetting

Abstract: The aim of continual learning is to learn new tasks continuously (i.e., plasticity) without forgetting previously learned knowledge from old tasks (i.e., stability). In the scenario of online continual learning, wherein data comes strictly in a streaming manner, the plasticity of online continual learning is more vulnerable than offline continual learning because the training signal that can be obtained from a single data point is limited. To overcome the stability-plasticity dilemma in online continual learning, we propose an online continual learning framework named multi-scale feature adaptation network (MuFAN) that utilizes a richer context encoding extracted from different levels of a pre-trained network. Additionally, we introduce a novel structure-wise distillation loss and replace the commonly used batch normalization layer with a newly proposed stability-plasticity normalization module to train MuFAN that simultaneously maintains high plasticity and stability. MuFAN outperforms other state-of-the-art continual learning methods on the SVHN, CIFAR100, miniImageNet, and CORe50 datasets. Extensive experiments and ablation studies validate the significance and scalability of each proposed component: 1) multi-scale feature maps from a pre-trained encoder, 2) the structure-wise distillation loss, and 3) the stability-plasticity normalization module in MuFAN. Code is publicly available at https://github.com/whitesnowdrop/MuFAN.

Published as a conference paper at ICLR 2023
NEW INSIGHTS FOR THE STABILITY-PLASTCCITY DILEMMA IN ONLINE CONTINUAL LEARNING
Dahuin Jung', Dongjin Lee', Sunwon Hong', Hyemi Jang'. Ho Bae Sungroh Yoon!.4,
'Department of Electrical and Computer Engineering, Seoul National University, *DeepBrain AI Inc., DDepartment of Cyber Security, Ewha Womans University, and *Interdisciplinary Program in Artificial Intelligence, Scoul National University, Scoul 08826, Korea anng 6250 ansuuuuuc.. an..acc.k wkda195126 snu. ww..a...........................
1dj9510 00 snu.ac....... kr hobaelewha www.waa.a.k.a....
harryédeepbrainai. aryoonesnu. ac.kr
ABSTRACT
The aim of continual learning is to learn new tasks continuously (ie., plasticity) without forgetting previously learned knowledge from old tasks (f.e. stability). the scenario of online continual learning. wherein data comes strictly in a streaming manner, the plasticity of online continual learning is more vulnerable than offline continual learning because the training signal that be obtained from single data point limited. To overcome the stability-plasticii dilemma in online contin- ual learning, propose an online continual learning framework named multi-scale feature adaptation network (MuFAN) that utilizes a richer context encoding ex- tracted from different levels of pre-trained network. Additionally, we introduce a novel structure-win wise distillation loss and replace the commonly used batch nor- malization layer with a newly proposed sability-plasticee normalization module to train MuFAN that simultaneously maintains high plasticity and stability. Mu- FAN outperforms other state-of-the- continual learning methods on the SVHN, CIFAR1O0 100, minilmageNet, and CORe50 datasets. Extensive experiments and ablation studies validate the significance and scalability of each proposed compo- nent: I) multi-scale feature maps from pre-trained encoder, 2) the structure-wis distillation loss, and 3) the
normalization module MuFAN. Code is publicly available at htthtpss//gttuub.coom
INTRODUCTION
Humans excel in learning new skills without forgetting what they have previously learned over their lifetimes. Meanwhile, in continual learning (CL) (Chen Liu 2018), wherein a stream tasks observed, a deep learning model forgets prior knowledge when learning new task if samples from old tasks are unavailable. This problem known as catastrophic forgetting (MeCloskey & Cohen 1989). In recent years, promising research has been conducted address this problem (Parisi ct al. 2019). However, excessive retention of old knowledge impedes the balance between preventing forgetting (i.e., stability) and acquiring new concepts (i.e.. plasticity), which is referred as the Hlailiity-plasiice dilemma (Abraham & Robins 2005). In this study, we cover the difference in the mubilltt-plasttiie dilemma encountered by online CL and offline CL. and propose novel approach that addresses the Mability-plasticiy dilemma in online CL.
Most offline CL methods aim at less constraining plasticity in the process of preventing forgetting instead of improving it because obtaining high plasticity through iterative training is relatively easy. However, as shown in Figure [ the learning accuracy (showing plasticity) of online CL is way lower than that of offline CL, with gap of 10-20% on all three CL benchmarks. That is, for online CL. wherein data comes in a streaming manner (single epoch), an approach that aims at suppressing excessive forgetting in the process of enhancing plasticity is required. For it, we propose a multi- feature adaptation network (MuFAN), which consists three components to obtain high stability and plasticity simultaneously: 1) multi-scale feature maps exploited from shallow to deeper layers of pre-trained model, 2) novel structure-wise distillation loss across tasks, and 3) a
"Corresponding Authors

---

Published as a conference paper at ICLR 2023
60%
Learning Accuracy (Plasticity
Forgetting Measure (FM) (Stability
Offine
19%
CON
19%
19%
69%
05
Splt CIFAR18
Spli MIIVN
CORc56
Spli CIPAR100
Spit: minW
COR
LA
Learning
Learning
Comparison results of ER-Ring on three CL benchmarks in offline and online CL in bar (left and middle) and scatter (right) plots. For offline CL. plasticity is relatively high, whereas stability low. In contrast, for online CL, stability is relatively high, whereas plasticity low. It shows the difference in trend between offline and online CL in terms of the stability-plasticity dilemma. Further analysis of the difference in the trend is provided in Appendix A
Figure 1:
novel stability-plasticity normalization module considering both the retention of old knowledge and fast adaptation in a parallel way.
First, using pre-trained representations has become common in computer vision (Patashnik et al. 2021 Kolesnikov et al. 2020 Ranftl et al. 2020) and natural language processing (Radford al. Peters et al. (2018). Meanwhile, the use of pre-trained representations in CL still naive, for example. using an ImageNett-pretranne ResNet as backbone (Yin et al.) 2021] Wu al.] 2021 Park et al. 2021 Hayes et al. 2020), which limits the structure or size of the pre-trained model that can be used. As shown Figure 2 (a), instead of using pre-trained model as a backbone, we propose an approach that uses the pre- pre-trained model as an encoder to obtain a richer multi-context feature map. Rather than leveraging a raw RGB image, we accelerate classifier training by leveraging an aggregated feature map from the meaningful spaces of the pre-trained encoder. We also verify the scalability of the aggregated multi-scale feature map by integrating it into existing online CL methods
Second, we present a novel structure-wist wise distillation loss to suppress catastrophic forgetting. Most distillation losses CL are point-wise (Chaudhry et al. 2019 Buzzega et al., 2020), and point-wise distillation is indeed effective in alleviating forgetting. Another effective way to preserve knowledge in classification task is through a relationship between tasks, especially in a highly non-stationary online continual setting. As described in Figure (b). we propose novel structure- wise distillation loss that can generate an extra training signal to alleviate forgetting using the relationship between tasks in given replay buffer.
Finally, the role of normalization in CL has been investigated in recent studies (Pham et al., 2021b Cha al. 2022 Zhou al., 2022). In the field of online CL, switchable normalization (SN) (Luo et al. 2018) and continual normalization (CN) (Pham et al. 20216), which use both minibatch and spatial dimensions calculate running statistics, have led to improvement in the final performance. However, we observed that these approaches do not fully benefit from either batch normalization (BN) [Iffe & Szegedy 2015) or spatial normalization layers. To address this problem, we propose a new stability-pllaticity normalization (SPN) module that sets one normalization operation efficient for plasticity and another normalization operation efficient for stability a parallel manner.
Through comprehensive experiments, we validated the superiority of MuFAN over other ot the-art CL methods on the SVHN (Netzer et al. 201T), CIFAR 100 (Krizhevsyy et al. 2009), mini- ImageNet (Vinyals et al., 2016). and CORe50 (Lomonaco & Maltoni] (017] datasets. On CORe50, MuFAN significantly outperformed the other state-of-the-arr methods. Furthermore, we conducted diverse ablation studies to demonstrate the significance and scalability of the three components.
2 RELATED WORK
2.1 CONTINUAL LEARNING
To date, in the field of CL, the utility of a pre-trained model relatively limited to obtaining the last feature map projected in meaningful space by using a pre-trained model feature extractor. CL methods that use pre-trained model can be categorized into two groups based on the scenario.
The first utilizes an ImageNet-
ResNet backbone for CL fine- tnnegrrined or video-based datasets to increase the base accuracy (Yin et al. 2021 (W et al.) [021] Park et al., [021]] [Hyes et al. 2020). In general, these methods update entire classifier continuously during training on
2

---

Published as a conference paper at ICLR 2023
(a) Overview
Memory
eympou NdB
C
apit
Data stream (online)
Pre-trained encodere
Top-down module
Classifler m
stability-plasticit normalization
(b) Cross-task Structure wise Distillation
Task
Task 2
Task
Task 2
Task
Task
Angular distance
Decision boundary
Task
Task
Task 3
Figure 2: (a) Overview MuFAN. MuFAN consists of pre-trained (fixed) encoder e and classifier m. MuFAN obtains an aggregated multi-scale feature map using top-down module and replaces every BN in m with the proposed SPN. (b) Simplified visualization of how the proposed structure- wise potential function builds structural relation between tasks during training on a stream of tasks.
stream of tasks rather than freezing feature extractor. The methods in the second group mostly feature extractor after fine-tuning in a base training session (Hayes) Kanan 20200 Maracani et al. 2021 Ganea al., 2021 Zhang al. 2021 Cheraghian al. 2021 Wu et al. 2022). They focused on finding a novel CL method that can efficiently train
freeze an
classification layer. Deep SLDA (Hayes & Kanan 2020 trained classification layer using linear discriminant analysis, and CEC (Zhang et al. 2002 trained classification layer to transfer context information for adaptation with a graph model. Lately, a continual semantic segmentation method, called RECALL (Maracani et al. 2021), which uses a pre-trained model as an encoder was proposed. However, MuFAN the first online CL. method that utilizes richer multi-scale feature map from different layers of the pre-trained encoder to obtain strong training signal from single data point.
2.2 NORMALIZATION LAYER
BN [loff Szegedy 2013) has been used as standard improve and stabilize the training of neural networks by normalizing intermediate activations. To perform normalization, numerous alternatives BN, which effectively aid the learning of neural networks in various tasks, have been proposed. A normalization layer can be categorized into three approaches based on the method of calculating the running mean and variance along convolutional features for normalization: 1) BN and its variants, batch renormalization (BRN) (Ioffe 2017) and representative batch normalization (RBN) (Gao et al., 2021), calculate the running statistics along the minibatch dimension, 2) instance normalization (IN) (Ulyanov et al. 2016), group normalization (GN) (Wu & He 2018), and layer normalization (LN) [Ba et al. 2016) calculate the running statistics along the spatial (channel or layer) dimension, and 3) SN (Luo et al. 2018) and CN (Pham et al.) 2021b) are hybrid methods that utilize both the running statistics along the minibatch and spatial dimensions in either a mixture or sequential manner. detailed explanation of each normalization layer be found in Appendix B Unlike SN and CN, our proposed SPN splits convolutional feature map along the channel dimension and applies different normalization operation to each feature map in parallel manner. Recently, several studies have suggested using multiple normalization layers in parallel manner (Pan et al. 2018) Xie et al.) 2020: Wang al. 2021 Zada al. 2002). However, mostly, they utilize multiple normalization layers in way of selecting one from them according to a purpose. The multiple normalization module has been explored improve the generalization of the model the field of adversarial training (Xie)
et al. 2020) and insufficient data (Zada et al. 2022).
METHOD
In this section, we first explain why MuFAN makes use of a pre-trained network as an encoder and an aggregated multi-scale feature map rather than single feature map for online CI. We then introduce
3

---

Published as a conference paper at ICLR 2023
novel structure- wise distillation loss that regularizes the shift in structural relations between tasks. Finally, we propose the SPN module that can be easily integrated to CL models to maintain high stability and plasticity simultaneously.
Notation. MuFAN consists of two networks: a pre-trained encoder e and a classifier m, and an experience replay buffer M, as illustrated in Figure 2 The classifier m consists of a feature extractor and a classification layer. Suppose that there are different tasks with respect to data streams (DI, Dr). where D; {Ia, 3 for i E [1,..... Т} (labeled sample), We denote a tuple of N data samples from jth task by X3 (æ), "1 1 a.)). ). For SPN, the feature map of a minibatch from m is denoted
"",,,,"")
IR Bxxxxw xH where B is the minibatch size, C is the number of channels, W is the width, and H is the height.
3.1 MULTI-SCALE FEATURES FROM A PRE-TRAINED ENCODER
Recent studies in computer vision and natural language processing have shown that utilizing pre- trained models in various ways improves the model's capacity and expands a range of available tasks (Patashnik et al. 2021 Kolesnikov et al.) 2020 Ranfil et al., 2020 Peters et al. 2018 Radford et al.) However, in CL. the utility of a pre-trained model is still limited to using a pre-trained network as a backbone. For CL, there is no such approach that utilizes a pre-trained network to obtain an aggregated multi-scale feature map constructed by augmenting contextual information extracted from different levels.
To consider why aggregated multi-scale feature map from a pre-trained model improves classi- fication, we first need to know the characteristics of the feature map from the pre-trained encoder, excluding multi-scale. For raw image, pixel value of 127 is not semantically close to a pixel value of 126 or 128. When leveraging a raw RGB image, a classifier in online CL should learn not only discriminative features but also semantic embedding within single epoch training. However, by leveraging the feature map from the semantic space of the pre- pre-trained encoder, we can accelerate classifier training. Then, we should consider why we need utilize multi-scale feature maps rather than a single feature map such as first or last feature map. Multi-scale feature maps have been demonstrated to be beneficial in object segmentation (Lin et al. (019 Guo at. 2020 He et al. 2019) and also recently in generative models (Sauer et al.) 2021 (022). The lewel features extracted by shallow layers encode more pattern-wise and general information, whereas the high-level features extracted by deeper layers contain more contextual information that focuses on prominent features. That is, the aggregated multi-scale feature map from shallow to deeper layers of the pre-trained encoder can compensate for the shortcomings of each layer. In an online setting, the classifier needs to learn as much information as possible within the restricted training iteration. For that, MuFAN takes advantage of the multi-scale feature map that provides condensed information obtained by diverse layers.
The concept of augmenting the context across different levels of a deep neural network consists of two approaches: bottom-up and top-down. As illustrated in Figure (a) obtain convolutional feature maps from four layers of the pre-trained model at different resolutions. The bottom-up module propagates more pattern- mattern-wise wise and general features into deeper layers using max-pooling, whereas the top-down module propagates more semantic features to shallow layers using bilinear upsampling. In this process, the bottom-up module propagates only specific features and discards the remaining ones. Between these two modules, MuFAN uses the top-down module to fully exploit the benefits of multi-scale hierarchical features. By taking the aggregated multi- scale feature map using the top-down module as input, the classifier m can carefully integrate the information required for classification among the rich features. We tested the possibility of wide range of pre-trained models as an encoder, such as a large supe-supe model or an object detection model. For mixing strategies between multi-scale feature maps, we follow differentiable random projections, yielding a U-Net-like architecture, proposed by Sauer et al. 2002 as described in more detail in Appendix C
(a) and Figure B in Appendix C we
3.2 CRSSS-TASK STRUCTURE-WIST DISTILLATION LOSS
Most distillation losses in CL have been wisnt for each data point, they regularized the change by distilling individual outputs. More precisely, the conventional pise distillation losses for CL essentially reduce forgetting by using an original prediction output soft label.
4

---

Published as a conference paper at ICLR 2023
The conventional distillation loss based on the replay buffer M Phham et al. 2020 is expressed as follows:
DDCCTN Σ (1-CTN(m3) mx))) e M,
(1)
where ID-CTN is a Kullback-Leibler divergence, t represents a current task, represents task ID, and m; represents the final classifier at the end of task j. The point-wise loss is commonly used in many
CL methods to alleviate forgetting (Buzzega et al.
2020 Pham et al.
2020 2021
Based on knowledge distillation (Chen et al. 2018 Park al. 2019 Fang al. 2020 Xu et al. 2021), in addition to the point-wiss wine information, we propose another regularization loss that can create extra training signal irrelevant label information. Unlike conventional approaches, present a structure-win wise distillation loss for CL. We compute a structure-wise potential v for tuple of data samples across tasks from M and distill relations through the potential. Our distillation loss CD-CSD regularizes forgetting by forcing an angular distance to be maintained between tasks. The distillation loss is defined as follows:
DDCSD = ΣΣ
2,"), Vine 1(0m (,) i=
(2)
where e(x) (and Z e(22) is the multi-scale feature map from with the top-down module, I is a cross-entropy loss that penalizes the difference between the previous and current models, m
and m,- At the end of each task, N samples per task randomly selected from M. se represents structure-wise potential function that measures the relational energy of the cross-task tuple.
The distillation loss CD-CSD can force the current model m, to maintain the same relationship between cross-task samples. We suggest potential function that measures the angular distance between samples of a cross-task. Our structure-wis potential function 3 returns an N N-dimensional vector as
Pm (1)-1, a(< m( m²3)) >,-., m(z5-1), >),
(3)
where a represents softmax function and <, > represents cosine similarity (CS). As illustrated in Figure (b), our loss function seeks to maintain the angular distance between the samples of consecutive tasks. In this process, we penalize the angular differences between tasks 1 and and tasks j and j + 1, where task j is then forced to maintain the same relationship with both j - and j 1. We empirically confirmed that maintaining the angular distance with the previous and next tasks, j - 1 and j + respectively, is more advantageous in terms of stability than that with a single task. Our structure- wise distillation loss can be used with conventional point-wise distillation losses to further improve performance. The overall objective of the proposed method expressed as follows:
L= Leu + LEx + ADCCICooCIN + ADsCSDLD.CSD,
(4)
- Ei== Mly', mm(h')) E D, is a cross-entropy loss for the current task t and LER = Si-ILER(E mm(("")) E .M is cross-entropy loss for the past tasks. Also, the inputs of LER and LDCCIN are replaced with h in our objective. AD-CTN and Ap-CSD are balancing factors for each loss. By maintaining the angular similarity of task embedding spaces between consecutive tasks with LDCCSD forgetting can be effectively reduced. While several studies (Hou al.] 2019 Wu al., 2021) apply adaptive weight for effective distillation as the task increases, our LDoCSD naturally grows as the task increases without additional adaptation, rendering the use of an adaptive weight unnecessary.
where =C==
3.3 MAABILITY-PLASTICII NORMALIZATION MODULE
We propose a novel online CL normalization module that splits the feature map of the classifier m into halves along the channel dimension and applies a different normalization operation to each half feature map. A few recent works have investigated the role of normalization in CL and proposed a new normalization module for CL (Pham al.) 202 lb) Cha et al. 2022 Zhou et al. 2022). However, to the best of our knowledge, this is the first study that sets normalization layers with different strengths in parallel way to obtain high stability and plasticity simultaneously. By setting normalization operation efficient for stability and another normalization operation efficient for plasticity in parallel manner, observed significant improvement in the final performance.
5

---

Published as a conference paper at ICLR 2023
Table I: Comparison results on four CL benchmarks. The same backbone and 50 memory slots per task used by all methods (MF: the aggregated multi-scale feature map from the pre-trained encoder as input). Bold fonts represent the best performance in each evaluation metric.
Split SVHN
Split CIFARI00
Method
ACC
FM
LA
ACC f
FM L
LA 1
GEM
82.30
12.16:+4.81
1,406+1.46
57.89+-0.98
8.62+0028
13.01=
ER-Ring
91.68+1.11
5.26+1.10
95.48+0.68
61.32+0.86
5.16+0.50
63.20+0.84
MIR
91.22-+0.43
6.18-0.54
96.16+0.14
64.97+00.4
7.78+1.47
7..03+1.90
CTN
92.14-+1.84
3.08=11334
94.42:+2.69
67.04:12.86
4.25++3.00
69.21
DualNet
93.88:10.51
3.04=:044
16.18:19988
72.6110.76
3.62:10.3
74.65:10.40
ER-Ring w/ MF
92.30=0031
5.76++1.39
96.8++2.06
69.33+1161
8.78+1126
77.41+0.65
CTN w/ MF
93.53+1122
3.90++1111
95.97+1.37
72.26+0.87
5,30+0.68
76.27+0064
DualNet w/ MF
94.06=00.55
96.58+2202
74,66+00.58
5.01
76.71+0.29
MuFAN
94.76=00.68
2.90=0.60
0.36
75.86+00.55
4.24+:026
0.44
Split miniMMN
CORe50
Method
ACC1 f
FM 4
LA f
ACC1
FM 4
L.A f
GEM
56.90:10.91
5.32:10.86
60.12:10.98
41.50:10.84
5.78
44,24 1.58
ER ER-Ring
54.22:10.82
10.50=0.63
63.92.1092
45,11
8.82:10.52
50.73=1.81
MIR
54.36+11.20
7.28+0072
60.25+0.99
45.60+1.67
5.24+1.72
48.00-0.55
CTN
66.70++1.98
4.30+1.94
68.02-0042
54.40+1.337
5.18+1.61
55.40 1.47
DualNet
72.40:10.54
4.04++0.1
74.16=10.47
1..64
4.43++0.22
58.86=0.66
ER-Ring w/ MF
63.00:12.87
13.44=2.82
7..40=10822
50.5611.88
15.30.13.34
63.76=0071
CTN w/ MF
70.30-10.61
6.52.10.22
74.7410.73
55.70
9.67-1.55
61.77=2.20
DualNet w/ MF
73.34:10.89
4.06:10.6
74.82+1.42
59.40
1.5611162
62.72=0.65
MuFAN
75.40:10.44
4.40+0.30
76.87=11.66
67.30+1.57
4.38++0.2
67.74-1.85
We split the minibatch the feature maps a E R BXCxWxX into halves along the channel dimension:
a1 E R Bx & x and a2 E I Bx & xW
The first feature map aj is fed into BN which can
learn and preserve global context follows:
YBN
a1
JARN
+ Bux,
(5)
+
where MaN BHW 2d=1
W -m Ek=1 a and og BHW 2b=1 Ew Σ hh=1 (a: pBN)
W
are the mean and variance calculated along the minibatch dimension. YAN and BAN are the affine transformation parameters of BN. The second feature map ag undergoes the combination of IN and LN which utilize instance-specifi running statistics along the spatial dimension as follows:
EKE(IN,AN) WAIAK
à = -INLLN ERE(INAN w(o
+ BIN,LN,
(6)
where MIN HW W mm ΣH a2 and ON HW Σw Σ (az MIN) are the mean and variance calculated along the channel dimension, and PLN = CHW ΣE MW == Σ a2 and 2 are the mean and variance calculated along the
LiN CHw ΣE ΣW Ei= layer dimension. w. and we are balancing weights between IN and LN which are optimized by backpropagation. Using softmax, make [EE W/ and ΣkE(IN.N)) w. 7NN,IN and BIN.LN are the affine transformation parameters of the combination of IN and LN. Then, the concatenation of å and å along the channel dimension is fed into the next convolutional kernels. As shown in Huang et al. (2020) and Luo al. (2018), depending on the characteristies of data or task, the spatial normalization layer that exhibits stable performance is different. Thus, we utilize the combination of IN and LN for robustness various datasets.
BN commonly performs better than spatial normalization layers in image classification tasks because it is beneficial in stabilizing training and improving generalization. However, due to the nature of CL, the basic assumption of BN that the running statistics between training and inference are consistent cannot be held. This inconsistency causes negative bias in moments and provokes catastrophic forgetting. In contrast, spatial normalization layers, such IN, GN, and LN, compute the running statistics along the channel or layer dimension. Therefore, these normalization layers are effective in reducing the negative bias in inference because they do not require the minibatch statisties obtained

---

Published as a conference paper at ICLR 2023
Table Comparison results in task-free setting whe- re the notion of tasks are unavailable (S: the criteri- on of a new potential task in Appendix F)
Table 3: Effectiveness of multi-scale feature maps (ST: standard, BU: bottom-up, and TD: top-down).
Online Task free CL
ACC1
SVIIN
CIFARIOD
CIFARIGO
miniLMN
CORe50
ER-Ring (Fixed, ST)
63.6-:.55
61,9::022
ER
20.5+0.9
11.0::0.55
1..
Ring (Fixed, BU)
65.811.3
64.0.044
DER++
20.74:2.7
13.74:122
31.71:4.7
ER-Ring (ST)
83.74-4.2
62.8::09
DualNet
25.51.0.7
20.9.1.66
35.6:10.6
2)
ER Ring (BU)
78.312.33
55.01009
MuFAN (5
0..6
34.74:211
47.2423.6
MaFAN (S = 10)
38.2+008
33.3::1.5
48.5=
ER-Ring (TD)
92.3-10033
69.3++116
during training. Unlike SN and CN which merge normalization for stability and normalization for plasticity in a mixture or sequential way, SPN applies independent normalization on the divided feature maps. We suggest this structure allow the convolutional kernels to independently utilize differently normalized feature maps during training. Specifically, the channel-wise separation boosts both the stable learning property of BN and the pegeettinn--reeentiin ability of IN and LN combinations. We confirmed that the proposed SPN outperforms SN and CN through experiments.
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
We tested the effectiveness of MuFAN on online task-incremental and online task-free CL settings.
We compare MuFAN with the following state-of-the-ar baselines: GEM (Lopez-Paz & Ranzato 2017a). ER-Ring (Chaudhry et al. 2019), DER++ (Buzzega et al. 2020), MIR (Aljundi et al. 2019). CTN (Pham et al. 2020), and Nut (Pham al.]
2021a). We reported the results of five runs for all the experiments. We used three standard metrics: the average accuracy (ACC 1) (Lopez-Paz et al.,
2017), forgetting measure (FM L) (Chaudhry et al. 2018a), and learning accuracy (LA 1) (Riemer al. 2018), to measure accuracy, stability (FM) and plasticity (LA). Further details the evaluation metrics, baselines, training, data augmentation, and hyperparameters can be found in Appendix D and E We observed MuFan's 's insensitivity to hyperparameter tuning. Although
used a single hyperparameter configuration that gave the best average performance over all four benchmarks, MuFAN still outperformed the baselines (Appendix H has more results on hyperparameter tuning).
Datasets. We consider four CL benchmarks in our experiments. For Split SVHN (Tang & Matteson
2020), we split the SVHN dataset (Netzer al. 2011) into 5 tasks, each task containing 2 different
classes sampled without replacement from a total of 10 classes. Similarly, for Split CIFAR 100 (Lopez)
Paz & Ranzato 2017a), we split the CIFAR 100 dataset (Krizhevsky et al. 2009) into 20 tasks, each
task containing 5 different classes. The Split minilmageNet (Split minilMN) (Chaudhry et al.) (2018b)
constructed by splitting the minilmageNet dataset (Vinyals al. 2016) into 20 disjoint tasks. Lastly. the CORe50 benchmark is constructed from the original CORe50 dataset (Lomonaco &
Maltoni 2017). with a sequence of 10 tasks.
Architecture. As the pre-trained encoder e, we used an AmageNet-preeraingg EfficientNet-lite0 (Tan Le. 2019) on Split SVHN, Split CIFAR 100, and CORe50 and a COCO peee--prte SSDlite (Howard et al. 2019), which uses MobileNetV3 as backbone, on Split minilMN, We implemented the SSDlite model trained on the MS COCO object detection dataset (Lin et al. 2014) from scratch. For the classifier m, we used a randomly initialized ResNet18 (He et al. 2016). Details the libraries used can be found in Appendix
4.2 RESULTS ON ONLINE CONTINUAL. LEARNING BENCHMARKS
Pask-incremental setting. Table I shows the performance comparison on four CL benchmarks. Overall, GEM showed a lower ACC than the rehearsal based methods, ER-Ring and MIR. DualNet showed a significantly improved performance, particularly, on Split CIFAR 100 and Split minilMN. However, MuFAN exhibited superiority over the other stateoo-thheart methods on all four benchmarks. Specifically, on CORe50, MuFAN exhibited 17% and 15% relative improvements in ACC and LA over DualNet, respectively. Despite the trade-off between FM and LA, MuFAN obtained the highest LA and comparable lowest FM, resulting in showing the highest ACC on all four benchmarks. In other words, MuFAN succeeded addressing the stability-plastice dilemma in online CL. Furthermore, to test the scalability of the aggregated multi-scale feature map from the pre-trained model,
7

---

Published as a conference paper at ICLR 2023
Table 4: Importance of a distance metric used in CD-CSD.
Table 5: Ablation study on a base task in the structure-wise distillation loss.
ER-Ring
Splic CIFARIOO
Split minilMN
CORe50
ACC
FM
ACC
FM
ACC T
FM 4
LA f
1.2 Norm
0.85
4.7=1.2
60.3:11.7
4.7.1:10
Lpeso
67.30==1.57
67.74+ 0,38
CS
6.5
4.6=0.3
10.9
4.5:20.6 4.9:00.5
Enasn
66.54:11.22
5.98:1
18.25:11.66
ADC
64.3=0.9
4.9=0.8
60.3:t0.7
6.899.11.89
5.42.12.01
6.22
Table 6: Comparison results of different normalization layers. The same backbone and 50 memory slots per task are used by all methods.
Num
BN
RBN
IN
GN
LN
SN
CN
SPN (Ours)
DER+
ACC
FM
LA
ACC
FM
1A
ACC
FM
LA
ACC
FM
L.A
ACC
PM
LA
ACC
FM
LA
ACC
FM
1.
ACC
FM
LA
miniMMN
55.0
62.2
7.4
97
58.5
59.6
53.6
66.8
64.8
CORE50
45.6
72
196
3.9
55.7
42.0
33
40.
46.3
47.2
55.9
54.1
55.0
integrated it into the baselines and reported the results in Table I MuFAN still outperformed the baselines using the multi-scale feature map. We observed a significant increase in LA in ER-Ring and CTN. However, relatively, the multi-scale feature map was less effective for DualNet. This is because that DualNet already leverages general representations using self self-supervised learning. Consequently, we demonstrated that our proposed concept of leveraging the aggregated multi-scale feature map is scalable with existing online CL. methods. To check the effectiveness of the proposed components more clearly, we first tested each component with the baseline CL method in several experiments.
Task-free setting. To test the effectiveness of MuFAN in an online task-free CL setting where the notion of tasks and task boundaries are unavailable during training and inference, we revised the original Eq. 2 to Eq. iinn in Appendix with two different Ss (the criterion of new potential task). We confirmed that MuFAN is still superior to the baselines in a task- free setting. Appendix (oppenddx/Fcoverrss Foooeers more details.
FF] Table 2 presents the comparison results on three benchmarks
Multi-scale feature maps from pre-trained encoder. In Ta- ble 33 to demonstrate the significance of the top-down module, ablated over three scenarios with standard, bottom-up, and top-down modules. A standard module represents case that only the last convolutional feature map from the pre-trained network is utilized. For the standard and bottom-up modules, due to the small height and width size, they are applicable only when the pre-trained network is used as feature extractor. The first and second scenarios are that the pre-trained model is used as a feature extractor. The difference between the two is whether the feature extractor is fixed during training. The third is that the pre-trained model is used as an encoder. As shown in the results of Table using the top-down module with the pre-trained encoder exhibits superiority over all cases of using the standard or bottom-up module. In addition, we tested which and how many layers must be used to make the most informative multi-scale feature map and reported the results in Table Rather than only the first layer or last three layers, the four layers including a shallow layer showed impressive results. We also performed ablation studies by varying pre-trained models, including self-supervised model, as the encoder and checked that MuFAN mostly shows better performance than DualNet with any pre-trained model (Appendix G
Table 7: Ablation study on the layer and the number of layers that must be used.
1CC
CIFARIOO
CORc50
First layer
73.89-11.4
62.01::3.04
Layers
72.22
61.07.10.99
4 Laayrss
05.86:10.35
67.30 1.57
Model complexity. We measured and reported the complexity of MuFAN and the baselines in Table g Because MuFAN uses EfficiintNet-litee0, which is lightweight version of EfficientNet, as the pre-trained encoder, only minimal parameters are added to the ResNet18 8 backbone. Compared with the baselines, and especially with the recent state-of of-tte----hh-rrr art method, DualNet, the increase in the number of parameters is not significant. Also, the total number of hyperparameters of MuFAN is smaller than those of DualNet and CTN. Further analysis of model complexity in terms of FLOP and memory is provided in Appendix H
Table 8: Model complexity.
# Params
ER-Ring CTN DualNet
11.202.162
11,261,362
13,272,690
MuFAN
14,565.234
Cross-task structure-wise distillation (Cn-csp). First, to evaluate whether the intuition of our proposed structure-wise distillation loss indeed holds, we visualized the latent space of MuFAN using T-SNE (Van der Maaten & Hinton, 2008) as shown in Appendix[[ (Figure 4). We observed that the relation between cross-task data points is well maintained throughout the stream of tasks. In addition to CS, we tested other distance metrics, Euclidean distance (2-norm distance, L2 norm) and angular distance using arccos (ADA) (Cer et al. 2018), to check whether there is better way
8

---

Published as a conference paper at ICLR 2023
Table 9: Ablation study on the proposed three components and data augmentation (DA).
Split miniMMN
CORc50
ACC1
FM
LAT
ACC
FM 4
LA
Lix: Lisen ADECTTCD-CTI
61.56=0.77
3.95=0.67
61.44#131
48.71=0911
7.1240.81
52.55=0.98
MF
DA Liscso
SPN
())
72. 02.08:0:42
5.72+0.42
18+008
56.00=1 663
11 01.52=0.88
65. 2..30.2.14 14
62.34=0.91
4.70
60,57+0.88
01.2220..88
5.29+0049
52.14=0,78
(3)
63.71=0.59
3,47+0441
02.48+0.32
$1.88=1.11 1.11
4.60++0.37
52,63=11005
65.62=0.58
5,29::0.56
65.15+0.99
16.03=181
4.39:1191
17.31=1.62
(5)
V
73.22-0..0
5.10110.6
15.5010.69
60.36:1..7
7.7611105
65.9011.48
(6)
73.36
3.6010.36
15.46
60.66.10.76
5.02.1.0.82
1.15
(7)
75.52-0.88
5.90++0.38
77.80+0.79
61.57+2.16
8,52++2.72
68.62+1.50
(8)
74.14- 0.81
3.34
74.42100.66
1..98
3.92.10.88
63.44:1.64
(9)
75. 0.992
5.65+0.80
77.66+0.48
65 19=
6.88++0.22
68.31+1822
(10)
15.11=1.16
4.4420.89
66.00=1181 16.61+146
5.46+0088
68.73=1.50
(11)
V
7..40=0.44
4.4010.30
76.87:1.66
67.30=1.57
4.38100.2
67.7421.85
to penalize the distance between samples across tasks. As shown in Table 4 although the result using CS is slightly superior, in general, the proposed structure- wise distillation loss showed stable performance whether using a distance-based metric (Eq. 13 in Appendix D or any other angle-based metric (Eq. 15 in Appendix J.. Besides, we tested the effectiveness of suppressing forgetting when the first task (Eq. [6 in Appendix K or the most recently learned task (Eq. 17 in Appendix K is used instead of the consecutive tasks to build the relation and reported the results in T Among the three losses, LD-CSD brought most impressive improvement in FM. As result, we observed that using consecutive tasks (Eq. 2) is most effective in ACC and FM. Lastly, we assessed the number of samples per task N that must be used to efficiently utilize LDCCS-- We confirmed that since N is equal to the number of classes in each task, it is possible to build the reliable relation across tasks for distillation. Appendix K covers more details of equations and results.
stability-llasticity normalization. As shown in Table 6 our SPN exhibited superiority over both single and multiple normalization layers, especially on 50R SPN showed a 13% relative im- provement in ACC over CN. More precisely, RBN, variant of BN, had the best performance among the single normalization layers. RBN is advantageous in the nature of CL because it additionally leverages the statistics of -pecifc features for feature calibration. Although it tends to vary according to the dataset, the spatial normalization layers such as IN, GN, and LN show low LA but also low FM. Among IN, GN, and LN, we only utilize IN and LN, not GN, for SPN to avoid an additional hyperparameter (the number of groups). We also tested the performance of SPN and the other normalization layers on ER-Ring and obtained consistent results (Table [9 in Appendix [ As an ablation study, we conducted the experiments of using only IN or LN for the stability normalization operation and summarized the results with analysis in Appendix L
Ablation study of each component. We assessed the importance of each component proposed in this study by using various combinations of the components. As shown in Table 9 we verified that the aggregated multi-scale feature map and SPN consistently enhance plasticity (LA 1), and the structure-we wise distillation loss enhances stability (FM 4) under diverse combinations. Most surprisingly, the LA results of entry (1) in Table p verify that multi-scale semantic feature maps generate a very strong training signal that can help to quickly acquire new information on the fly. The results of entries (1) and (6) in Table validated that catastrophic forgetting can be largely suppressed by a relational distillation signal from our structure- wise loss while maintaining plasticity.
CONCLUSION
For online CL, wherein data comes strictly in a streaming manner, achieving high plasticity is challenging; thus, it difficult to expect a sigmificant final performance by only improving stability. In this study, we propose a novel online CL framework named MuFAN, which enhances both stability and plasticity while being less impediment to one another. This is the first CL method that leverages multi-scale feature maps, which are constructed by projecting raw images into meaningful spaces, as the input for the classifier in order to improve plasticity. In addition, we propose a novel structure- wise distillation loss that alleviates forgetting using relation between samples across tasks. Finally, we present SPN that can be easily integrated into online CL methods to maintain high plasticity and stability simultancously. Carefully designed ablation studies demonstrated the significance and scalability of the proposed three components in MuFAN. In future work, we would like to delve into the utility of pre-trained models in few-shot CL setting.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2022R1A3D 1077720], Institute of Information & Communications Technology Planning Evaluation (LITP) grant funded by the Korea government (MSIT) [2021- 0-01343, AI Graduate School Program (SNU); 2022-0-00959; 2022-00155966, Al Convergence Innovation Human Resources Development (EWU)) Samsung Electronics (10221213-04111.001) and LG Innotek.
REFERENCES
Wickliffe C Abraham and Anthony Robins. Memory retention-the synaptic stability versus plasticity dilemma. Trends in neurosciences, 28(2):73-78. 2005.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in neural information processing systems, 32, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1rX.v:16070064000 2016.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33: 1059900-19300 2020.
Daniel Cer, Yinfei Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardoo-espeddss Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv: 11155, 2018.
Sungmin Cha, Soonwon Hong, Moontae Lee, and Taesup Moon. Task-balanced batch normalization for exemplar-based class-incremental learning. arXiv preprint arXiv:2201.122 12559, 2022.
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings the European Conference on Computer Vision (ECCV), pp. 532-547, 2018a.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. International Conference on Learning Representutions, 2018b.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprini arXiv: 10486, 2019.
Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Darkrank: Accelerating deep metric learning via cross sample similarities transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1-207, 2018
Ali Cheraghian, Shafin Rahman, Pengfei Fang, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi. Semantic-aware knowledge distillation for few-shot class-incremental learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2021.
pp. 2534 2543,
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 255. leee, 2009.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. In International Conference on Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2023
Dan Andrei Ganea, Bas Boom, and Ronald Poppe. Incremental few-shot instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1185-1194, June 2021.
Shang- Hua Gao, Qi Han, Duo Li, Ming-Ming Cheng, and Pai Peng. Representative batch normaliza- tion with feature calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8669-8679, 2021.
Chaoxu Guo, Bin Fan, Qian Zhang. Shiming Xiang, and Chunhong Pan. Augfpn: Improving multi-scale feature learning for object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Putlern Recognition, pp. 12595-12604, 2020.
Tyler L. Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR) Workshops, June 2020.
Tyler L. Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan, Remind your neural network to prevent catastrophic forgetting. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 466-483, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58598-3.
Junjun He, Zhongying Deng, and Yu Qiao. Dynamic multi-scale filters for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3562-3572, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2019.
Hexiang Hu, Ozan Sener, Fei Sha, and Vladlen Koltun. Drinking from a firehose: Continual learning with web-scale natural language. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Normalization techniques in training dnns: Methodology, analysis and application. arXiv preprint arXiv: /2836, 2020.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. Advances in neural information processing systems, 30, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. Advances in neural information processing systems, 31, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Grabska-Barn Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference computer vision, pp. 491-507. Springer, 2020.

---

Published as a conference paper at ICLR 2023
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung- nog Ha, and Byoung-Tak Zhang. Overcom- ing catastrophic forgetting by incremental moment matching. Advances in neural information processing systems, 30, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
Di Lin, Dingguo Shen, Siting Shen, Yuanfeng Ji, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Zigzagnet: Fusing top-down and bottom-up context for object segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 7490-7499, 2019.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In Conference Robot Learning, pp. 17-26. PMLR, 2017.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017a
David Lopez-Par Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017b.
David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Discovering causal signals in images. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9979-6987,55 2017.
Ping Luo, Jiamin Ren, Zhanglin Peng. Ruimao Zhang, and Jingyu Li. Differentiable learning-to- normalize via switchable normalization. arXiv preprint arXiv: 10779, 2018.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773. 2018.
Andrea Maracani, Umberto Michieli, Marco Toldo, and Pietro Zanuttigh. Recall: Replay-basee continual learning in semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7026-7035, October 2021.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 464-479, 2018
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 11::54771, 2019.
Jaeyoo Park, Minsoo Kang, and Bohyung Han. Class-incremental learning for action recognition in videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13698-13707, October 2021.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
12

---

Published as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buse E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019, URL http///tppppappeeseeeriiisssaass
ibbrrr.
pdf
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- 2021.
ence on Computer Vision, pp.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 2227 2237, New Orleans. Louisiana. June 2018. Association for Computational Linguistics. doi: 10. 11188665/11/18/ 1202. httpt
//aclantho Logy. org/N18 -222
Quang Pham, Chenghao Liu, Doyen Sahoo, and HOI Steven. Contextual transformation networks for
online continual learning. In International Conference Learning Representations, 2020.
Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34, 2021a.
Quang Pham, Chenghao Liu, and HOI Steven. Continual normalization: Rethinking batch normal- ization for online continual learning. In International Conference on Learning Representations, 2021b.
Robi Polikar, Lalita Upda, Satish S Upda, and Vasant Honavar. Lear++: An incremental learning algorithm for supervised neural networks. IEEE transactions on systems, man, and cybernetics, part C (applications and reviews), 3144)497-500, 2001.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training.
René Ranfil, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pallern analysis and machine intelligence, 2020.
Sylvestre-AIvy Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampent. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pallern Recognition, pp. 2001-2010, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2018
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 04671, 2016.
Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Neural Information Processing Systems, 34, 2021.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. arXiv preprint arXiv:2202.00277 2022.
13

---

Published as a conference paper at ICLR 2023
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp- 4548 4557. PMLR, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-61 14. PMLR, 2019.
Binh Tang and David S Matteson. Graph-based continual learning. International Conference on Learning Representations, 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint axXiv:160700002) 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. Augmax: Adversarial composition of random augmentations for robust training. Advances in Neural Information Processing Systems, 34, 2021.
Guile Wu, Shaogang Gong, and Pan Li. Striking balance between stability and plasticity for class- incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1124-1133, October 2021.
Tz- Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vasconcelos, Rahul Bhotika, and Stefano Soatto. Class-incremental learning with strong pre-trained models. arXiv preprint wwXivv2204.036::2 2022.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV). pp. 3-19, 2018
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 819-828, 2020.
Haohang Xu, Jiemin Fang, Xiaopeng Zhang, Lingxi Xie, Xinggang Wang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Bag of instances aggregation boosts self- sulf-super distillation. In Interna- tional Conference on Learning Representations, 2021.
Haiyan Yin, peng yang, and Ping Li. Mitigating forgetting in online continual learning with neuron calibration. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 10260-10277 10272. Cur- ran Associates, Inc., 2021. URL https: /ttthtttsss//pooeeeeeee http:///pooeei/cceeeeecoee neur i ps wttpccppppp/c///.//// fi 0lle/54ee2 9 0 58 a 22w.2332 839f5-55-0 Paperrappr...... .
Shiran Zada, Itay Benou, and Michal Irani. Pure noise to the rescue of insufficient data: Improving imbalanced classification by training on random noise images. In International Conference on Machine Learning, pp. 25817-25833. PMLR, 2022.
Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incremental learning with continually evolved classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14455-1264,, June 2021.
Minghao Zhou, Quanziang Wang, Jun Shu, Qian Zhao, and Deyu Meng. Diagnosing batch normal- ization in class incremental learning, arXiv preprint 2202::2202.00225. 2022.
14

---

Published as a conference paper at ICLR 2023
A ONLINE CL AND OFFLINE CL
In the conventional scenario of online learning, they assume data that comes strictly in a streaming manner. That is, the model immediately adapts to data that moment, and the streaming instance or small batch is immediately discarded (Polikar et al. (00T). Therefore, storing all of the streaming data in the online validation buffer during a short moment suggested in the papers (Hu et al. 2022 Aljundi et al. 2019) is not common in the conventional online learning scenario. However, whether equaling the number of iterations and the number of epochs brings similar performance results. As shown in Table TO even if the number of iterations and the number of epochs match one another, we still observed lower LA and lower FM in online CL. In other words, the characteristics of online CL are maintained unless offline storage is assumed.
tested
Table 10: Comparison on equaling the number epochs (10 epochs) and the number of iterations (10 iterations).
Split CIFARIO0
Split minilMN
CORc50
Method
ACC
FM 4
LA f
ACC
FM
LA
ACC
FM 4
LA +
Epochs
60.38
17.67
77.16
59.14
20.02
78.10
53.68
11.76
65.32 59.05
10 Iterations
58.93
13.40
71.62
58.70
9.87
67.13
51.70
9.30
B NORMALIZATION LAYERS
Batch normalization (BN) (lofe & Szegedy 2015) has been demonstrated beneficial in a wide range of deep learning applications. Inspired by BN, varying normalization layers have been proposed by exploring different normalization dimensions. According to that how the running statisties are calculated along which convolutional feature dimensions, categorize normalization layers into three approaches: 1) BN and its variants, batch renormalization (BRN) (Ioffe (017)) and representative batch Normalization (RBN) (Gao et al. (22TT. 2) instance normalization (IN) (Ulyanov et al. [2166) layer normalization (LN) (Ba et 2016). and group normalization (GN) (Wu & He 2018)) and 3) switchable normalization (SN) (Luo et al. 2018) and continual normalization (CN) (Pham et
2021b).
B. BN AND ITS VARIANTS, BRN AND RBN
When convolutional feature map is fed into BN (lffe & Szegedy 2013) layer, the mean and variance of a current minibatch are calculated to perform normalization. A BN layer then goes through an affine transformation process of scaling and shifting the normalized features with two learnable parameters.
BN can promote preserving the global context of task during training. However, BN tends to fail to show full performance when a minibatch is small or non-independent and identical distribution (non-i.i.d.). To address this problem, BRN (Toffe 2017) proposed simple extension of BN, which implements a re paraaetriizationn trick before the affine transformation.
Recently, RBN (Gao et al. 202I) proposed another simple extension of BN that goes through an additional centering and scaling calibration process by leveraging the statistics of instance- specific features to alleviate the inconsistency of the running moments between training and inference. Therefore, RBN helps construct stable feature distribution between channels with minimal additional cost.
B.2 IN, LN, AND GN
The works in the second approach normalize a convolutional feature map along the spatial (channel or layer) dimension, which is advantageous to the discrepancy between training and inference.
IN (Ulynnov et al. 2016) was first proposed for an image style transfer network. The mean and variance calculated for each channel of single input feature. Similar IN, LN (Ba et al. 2016).
15

---

Published as a conference paper at ICLR 2023
which is widely used in recurrent neural networks, normalizes a convolutional feature map along the layer dimension instead of the minibatch dimension. In other words, LN calculates the mean and variance along all channels for input feature. Lastly, GN (Wu & He 2018 is closely related to IN and LN, but instead normalizing for each individual channel or along all channels; GN normalizes input feature by dividing the channels into G groups. Therefore, when G = GN behaves the same as LN, and when G equals the number of channels, GN behaves the same as IN.
B.3 SN AND CN
Third, SN and CN normalize convolutional feature map along both the minibatch and spatial dimensions to take advantage both approaches.
In SN, operations in BN, IN, and LN are used to compute the means and variances of three types estimated along the minibatch, channel, and layer dimensions. By learning their blending weights, SN combines the statistics of these three normalization layers for final normalization.
Moreover, recently, CN (Pham et al. 202Tb) was specially designed address the non-i.idd nature of CL. To mitigate the side effects arising from the model normalizing previous tasks' data using the moments of a current task during inference, they first normalize an input feature along the spatial dimension (GN) without the affine transformation. Then, the spatially normalized feature map is fed into a normalization operation along the minibatch dimension (BN). Mathematically, CN is defined
follows:
aGN GN), aCN BBN ((GG) + B,
where GN1 o and 1N1,0 represent a group normalization layer and batch normalization layer without an affine transform, respectively. Empirically, CN exhibited improvements in the final performance. However, still, it hard to argue that the benefit of BN was fully exploited in CN due to the characteristics of calculation in order. We, thus, propose the SPN module that performs different normalization operations parallel way.
C MIXING STRATEGIES
Upsampling vanen
Pre-trained encoder
Hierarchical
module
Max-pooling
Standard
ii) Bottom-up
iii) Top-down
Figure 3: Hierarchical module. For the standard and bottom-up modules, the final feature map is fed into the classification layer. For the top-down module, the final feature map is fed into the entire classifier.
Features different scales extracted by a deep neural network have different characteristics and the utility of multi-scale features has been exploited in several fields. Starting with a U-Net (Ronneberger al. 2015), methods for augmenting multi-scall features have been proposed (Lin et al. 2019 Guo ct al. 2020; He et al./ 2019 Sauer al. 2021I. As described in Figure 3] broadly, these methods can be divided into two major categories: top-down and bottom-up approaches. The top-down method propagates the features from deeper layers to shallow layers, whereas the bottom-up approach propagates the features from shallow layers to deeper layers. In the case of MuFAN, it follows the proposed method by Sauer et al. (2021), which is based on fixed, differentiable random projections.
Sauer et al. (201 proposed an augmenting method that consists of two different strategics to efficiently mix features from different layers: 1) Cross-Channel Mixing (CCM), and 2) Cross-Scale
16

---

Published as a conference paper at ICLR 2023
Mixing (CSM). The purpose of implementing CCM is to mix across channels, which encourages information preservation of each feature by permutation (Kingma & Dhariwal 2018). CCM utilizes a randomly initialized 1 X convolution with equal number of output and input channels. Second, CSM mixes features from two different scales. For the top-down module, to match the resolution and the number of channels of shallow layers, the feature from deeper layers enlarges resolution by bilinear upsampling and reduces the number of channels by random 3 x 3 convolution. For the bottom-up module, oppositely. the feature from shallow layers shrinks resolution by max-pooling and increases the number of channels by random 3 X 3 convolution. Then, the resized feature is added to another feature from different scale element- as illustrated in Figure For detailed intuition and analysis about the mixing strategies, please refer to Sauer et al. (202T).
By considering the resolution of final multi-scale feature map, the output from the bottom-up module is fed into classification layer without an average pooling.
Table 11: Ablation study on the mixing strategies (CCM: Cross-Chama Mixing, and CSM: Cross- Scale Mixing).
ACC
CIFARI00
CORc50
CSM
75.09.10.50
64.78.11.31
CCM + CSM
75.86:10.35
6..30.11.57
For MuFAN, we conducted an ablation study on the mixing strategies (CCM and CSM) proposed by Sauer et al. 2021). To test whether CCM is beneficial for MuFAN, checked the performance difference according to using only CSM and both. As shown in Table II we observed that using both mixing strategies is effective for MuFAN.
D EVALUATION METRICS
Three common ongoing learning metrics- accuracy (ACC) (Lopez-Paz al. 2017 forget- ting measure (FM) (Chaudhry et al. 2018a). and learning accuracy (LA) (Riemer et al., 2018 are used assess performance.
For notation, the accuracy of the model, as measured on the test set Diee after it has been trained to the most recent dataset D, of task is denoted by the symbol as.j.
ACC (Higher is better): evaluates the final performance of continual learning method in terms of accuracy.
T ACC(t) = T
Σ ar
(8)
j=1
FM (Lower is better): evaluates the performance of a continual learning method in terms of stability.
FM(L) T
Σ
max 0ij
QT
16 1E(1.-T111 T-1)
LA (Higher is better): evaluates the performance of a continual learning method in terms of plasticity.
1 LA(t)
1 T
Σ ajd
(10)
17

---

Published as a conference paper at ICLR 2023
E EXPERIMENT DETAILS
We conducted all experiments using the Pytorch (Paszke et al. 2019) framework on a single RTX8000 GPU. For varying pre-trained models, implemented open python libraries. For the EfficientNet- lite (Tan & Le 2019) and ResNet18 (He et al. 2016) trained by the ImageNet (Deng et al. 2009) dataset in supervised manner, we utilized the timm library. For the ResNet50 trained by the ImageNet dataset in self- supervveed manner, we utilized the viss! library. Lastly, for the SSDlite (Howard
al.] 2019) trained by the MS COCO object detection dataset (Lin et al.] 2014). we utilized the Pytorch torchvision library.
For all experiments, we run each experiment five times using different initialization with the same sequence of tasks.
E.1 DATASET SUMMARY
We summarize the datasets used in our experiments in Table I2 on MuFAN and all baselines except for DualNet. Because DualNet utilizes sule-super learning, it contains random cropping, resizing, horizontal flipping, color jittering, and so on. In the case of MuFAN and remaining baselines, utilized random cropping, horizontal flip, and resizing. As an exception, for Split SVHN, because data is sensitive to horizontal flip, we did not utilize it.
We applied the same transformations
One difference is that we utilized data augmentation only on data from the replay buffer. That is, we applied random cropping, horizontal flip, and resizing to data from the replay buffer and only resizing to streaming data. This because we observed an increase in the final performance when data augmentation is implemented only to data from the replay buffer in general.
Table 12:
Dataset summary, including number of classes, a number of tasks, number of data, and dimensions with data augmentation.
Dataset
Classes
# Tasks
Train
Test
Dim.
Split SVHN
10
5
73,257
26,032
3 x 32 x 32
Split CIFARI00
100
20
50,000
10,000
32
32
32
Split minilMN
100
20
50,000
10,000 x 84
3 x
128
x 128
CORe50
50
10
119,984
44,971
3 x
128
x 128
E.2 BASELINES
category of CL can be divided into three families: a regularization- based approach, a dynamic architecture approach, and replay-based approach. First, the regularization-biit based approach aims at consolidating old knowledge when learning new tasks, commonly without relying on an experience replay buffer (Li & Hoiem 2017 Kirkpatrick al. [017 Lee al., 2017). For example, LwF (Li & Hoiem 2017) hinders forgetting by using outputs from previous model and EWC (Kirkpatrick al.) (017) suppresses the changes in network parameters estimated as important. Second, the dynamic architecture approach fixes or/and expands network architecture and parameters to achieve both high stability and plasticity in less limited manner (Rusu al. 2016, Mallya Lazebnik) 2018 Serra et al. 2018). Lastly, the replay- -ased approach stores a limited number of examples in previous tasks and replays them during the training new tasks (Chaudhry et al. 2019 Lopez Paz & Ranzato 2017a Chaudhry et al.) 2018b Rebuffi et al. 2017 Shin et al. 2017). This approach regularizes forgetting either by forcing output activations of the stored individual data represented by the current model and that represented by the previous model to be the same (Chaudhry al. 2019) or by projecting the gradient direction on the current task outlined by the stored data (Lopez-
Ranzato 2017a).
MuFAN is also included in the replay-based approach. Thus, in this study, we compared MuFAN with several baselines in the replay-basee approach. A brief description of the baseline (competitor) is provided as follows:
GEM (Lopez-Paz Paz & Ranzato 2017a): regularizes forgetting by projecting the gradient
direction on the current task outlined by the stored data.
18

---

Published as a conference paper at ICLR 2023
AGEM (Chaudhry et al. 2018b): efficient version of GEM, which relaxes the gradient direction as one direction.
MER (Riemer et al. 2018); uses a variant Reptile approach and optimizes the gradient inner product between each sample pair in the buffer.
ER-Ring (Chaudhry et al., 2019): standard experience replay that stores a subset of data from previous tasks and optimizes multitask loss.
DER (Buzzega al. 2020): applies &g loss to regularize the difference between current
and past logits.
DER+ (Buzzega et al. the current logit and ground-truth label.
2020): a variant of DER with an additional &2 regularizer between
MIR (Aljundi et al. 2019): variant of ER that, by sorting, chooses the samples that increase the model's s forgetting.
CTN (Pham al. 2020); models both common and task task-specific features via a lightweight controller module and bilevel optimization objective.
DualNet (Pham et al., 2021a): Inspired by complementary learning systems, consists of fast and slow networks, which focus on capturing new knowledge and learning a general representation using self- learning, respectively.
Due to the page limit, included the results of AGEM and MER showing similar performance to GEM in Appendix. Overall, showed lower or comparable performance over the replay- based methods, ER-Ring and MIR.
Table 13:
Comparison results of AGEM and MER on four CL benchmarks. The same backbone and 50 memory slots per task are used by all methods.
Split SVHN
Split CIFAR100
Method
ACC 1
FM 4
LA f
ACC f
FM 1
LA t
AGEM
83.48+22.54
35.19
0..38+00.3 92.55+1.21
57.77-+0.97
6.08+
64.22- 1.13
MER
89,45+1.21
4.55#1.49
59.57=1.12
9.4410.91
69.83++0.79
Split minilMN
CORc50
Method
ACC1
FM 4
LA f
ACC
FM +
LA
AGEM
54.92+1..2
6.4210.77
60.61
40.62+1.84
10.58
47.74
MER
58.48:0.91
6.01#1.19
64.1240.62
40.1040.78
7.06+:0.6
46.66+1.02
E.3
HYPERPARAMETES SELECTION
By following the online task incremental setting used in Pham et al. (2021a), all methods are trained over one epoch with minibatch size of 10 on Split SVHN, Split CIFARIOG 100, and Split minilMN and 32 on CORe50. We store 50 samples per task in the replay buffer and use ring-buffer management strategy Paz & Ranzato 2017b). We provide the hyperparameters of each method.
GEM
Learning rate: 0.03 (Split SVHN, Split CIFAR100), 0.05 (Split minilMN)
Gradient noise y: 0.5 (all experiments)
Number of gradient updates: (Split CIFARI00, Split minilMN), 2 (Split SVHN)
AGEM
Learning rate: 0.03 (Split SVHN), 0.1 (Split CIFAR100), 0.3 (Split minilMN)
Number of estimate gradient constraints: 850 (all experiments)
Number of gradient updates: (Split CIFAR100, Split minilMN), 2 (Split SVHN)
MER
Learning rate: 0.03 SVHNN, 0.05 (Split minilMN), 0.1 (Split CIFAR100 100)
19

---

Published as a conference paper at ICLR 2023
Replay batch size: 64 (Split SVHN, Split CIFARI00), 128 (Split minilMN)
Number of gradient updates: 2 (Split SVHN), 3 (Split CIFAR100, Split minilMN)
Across batch learning rate y.: 0.3 (all experiments)
ER-Ring
Learning rate: 0.03 (all benchmarks)
Replay batch size: 10 (Split SVHN, Split CIFARIO0, Split minilMN), 32 (CORe50)
Number of gradient updates: 2 (all benchmarks)
DER
Learning rate: 0.03 (CORe50)
Replay batch size: 64 (CORe50)
Number of gradient updates: 2 (CORe50)
DER++
Learning rate: 0.03 (CORe50)
Replay batch size: 64 (CORe50)
Trade-off strength between soft and hard labels: I (CORe50)
Number of gradient updates: 2 (CORc50)
MIR
Learning rate: 0.03 (all benchmarks)
Replay batch size: 10 (all benchmarks)
Number of gradient updates: 2 (Split SVHN), 3 (Split CIFAR100, Split minilMN, CORe50)
CTN
Inner learning rate 0.01 (all benchmarks)
Outer learning rate 3: 0.05 (all benchmarks)
Replay batch size: 64 (Split SVHN, Split CIFARI00, Split minilMN), 32 (CORe50)
Regularization strength A: 100 (all benchmarks)
Temperature т: 5 (all benchmarks)
Semantic memory size in percentage of total memory: 20% (all benchmarks) Embedding dimension of MLP layer to map the task identifiers: 16 (Split SVHN, CORe50), 64 (Split CIFAR 100, Split minilMN)
Number of inner updates: 2 (all benchmarks)
Number of outer updates: 2 (all benchmarks)
DualNet
Slow learner's SGD learning rate: 3e-4 (Split SVHN, Split CIFAR100, Split miniIMN),
le-4 (CORe50)
Slow learner's 's Look-ahead learning rate: 0.5 (all benchmarks)
Fast learner's learning rate: 0.03 (all benchmarks)
Replay batch size: 10 (Split SVHN, Split CIFAR100, Split minilMN), (CORe50)
Barlow Twins's trade-off term ABT: 2e-3 (all benchmarks)
Barlow Twins's 's moving average term: 0.3 (all benchmarks)
Fast learner's trade-off term Atrain 2.0 (all benchmarks)
Soft label loss temperature т: 2.0 (all benchmarks)
Number of inner updates: 2 (all benchmarks)
Number of outer updates: 2 (all benchmarks)
MuFAN
SGG Learning rate: 0.03 (all benchmarks)
Replay batch size: 64 (Split SVHN, Split CIFAR100, Split minilMN), 32 (CORe50)
Temperature of LD-CTN: 2 (all benchmarks)
Point-wise distillation balancing factor ÀD.CTN: 10 (all benchmarks)
20

---

Published as a conference paper at ICLR 2023
Structure-wise distillation balancing factor AD-CSD: 0.01 (all benchmarks)
Teacher and student temperature т of LD-CSD: 0.0001, 2 (all benchmarks)
Number of samples per task build the relation across tasks N: 10 (all benchmarks)
- Number of gradient updates: 2 (all benchmarks)
To test the sensitivity of the proposed total loss to balancing factors, AD-CTN and AD-CSD. we observed the results with varying values. Table [ shows that the proposed loss not largely sensitivity to those factors. In all cases, MuFAN still outperforms the recent state-of-the-an DualNet. We chose 10 for ÀD-CTN and 0.01 for ĀD-CSD. which showed good performance on average.
Table 14:
Sensitivity to balancing factors, AD-CTN and Ap-CSD, in the proposed loss. Bold fonts represent the best performance.
ApesD 0.01
AP-CIN = 10.0
Àp.CTN
Ap.
ACC
1.0
10.0
100.0
0.001
0.01
0.1
Cifar 100
75.26+00.97
75.86++0.35
74.95+0.68
75.77++0.33
75.86+0.35
75.93++0.38
minilMN
74.48:10.90
75.40:10.44
74.54:1..01
75.11:10.87
75.40:10.44
76.01-1:1.3
CORe50
65.66.12.44
6..30:11.57
64.70:12.08
67.42-12.20
67.30:1.55 1.57
66.38:12.18
F ONLINE TASK-FREE CONTINUAL LEARNING SETTING
To the scalability of the proposed cructure-wise distillation loss in online task-free CL scenario where the notion of tasks and task boundaries are unavailable during training and inference, we revised the original Eq. as follows:
m(YM///S N848 Σ
CF Σ
E EM,
(11)
where uu-) counts the number unique class, YM denotes all currently stored labels in M. denotes the criterion of new potential task and // denotes floor division. That is, with S = 10, the summation of j starts when the number of a unique class M greater than or equal to 20. Ns denotes the number of samples per class. Also, subscript denotes w a uu)lm) where represents a previously saved u()m). That is, we store the softmax output of the samples in M using the model m when the number of counted unique class is increased. In addition, the replay buffer is implemented as a reservoir buffer instead of a ring buffer, and all training techniques requiring task ID during training and inference were not used.
G VARIOUS PRE-TRAINED MODELS
In Table 15 ablated over varying pre-trained models the encoder
Table 15: Ablation study various pre-trained models the encoder (SSL: learning, and OD: Object detection). Bold fonts represent the best performance.
Self-supervised
ACC
SVHN
CIFAR100
CORe50
EfficientNet-liee
94.8+10.7
75.9110.4
67.3
ResNet18
93.9+11.0
71.5+0077
67.9+11.5
ResNer50 (SSL)
91.6111.3
72.510.7
63.64:1.5
SSDlite (OD)
94.5+0.8
74.6+0.6
66.0+1.3
DualNet
93,9+0.5
72.6+0088
57.6+1.4
Motivated by Sauer et al. (2021). investigated an EfficientNet- and ResNet18 trained by the ImageNet dataset (Deng et al., 2009) in a supervised manner, ResNet50 trained by the ImageNet dataset in self-supervised manner, and SSDlite trained by the MS COCO object detection dataset.
21

---

Published as a conference paper at ICLR 2023
We confirmed that in most cases, any pre-trained model outperformed the recent ofate-of-the-aee method. For MuFAN, we integrate an EfficientNet-ltee which consistently showed the best or near-best performance, except for the Split minilMN benchmark.
H COMPUTATION AND MEMORY COST
To further compare the model complexity between MuFAN and the main comparison methods, we counted FL.P per iteration and memory cost the final task the CORe50 benchmark. As shown in Table [6] the difference memory cost among the three methods is negligible. FLOPS per iteration of DualNet and MuFAN are 349.1B and 392.2B, respectively. Indeed, there a 43.1B increase in MuFAN compared to DualNet, however, in ACC, MuFAN also exhibits a large increase of 9.66% over DualNet.
Table 16: Comparison on FLOP per iteration and memory cost.
CORe50 ER-Ring DualNet
FLOP per iteration
Memory cost
ACC
144.3B
3419.91 MB
45.11
349.1B
3453.26 MB
57.64
MuFAN (Ours)
392.2B
3501.99 MB
67.30
I T-SNE
In Figure observed that the relation between cross-task data points is well maintained throughout the stream of tasks.
Figure 4: Visualization of the latent space of MuFAN using T-SNE on CORe50, For better visualiza- tion, we used data points with single class from each task.
J DIFFERENT DISTANCE METRICS
In addition to CS, we tested other distance metrics to check where there is a better way to penalize the distance, First, we tested with 2-norm distance follows:
d(u, v) = ||u - -=l22-
(12)
where and v vectors. Using this distance metric, revised the original Eq 3 as follows:
((m (((x)_ ( m (b)) ... Ilmm (h)-1), m; (h.) I[.).
(13)
Second, we tried a different angular distance to check whether there is better way to penalize the angular distance. In Cer et al. (2018), they proposed similarity metric that transforms CS follows: sim [u, v) = 1 arccos(< u, >)/T,
(14)
where u and are vectors and < denotes CS. This metric computes the CS of two vectors and then uses arceos to create similarity based on an angular distance. Using this similarity metric, we revised the original Eq. 3 as follows:
(m =o ((1 arccos(< mi-1 mi..))mmm.mm. mi >)/x).
(15)
(1 - arccos( < m (it m >>/>>>/)))))))))))>))>>>>)))>>
22

---

Published as a conference paper at ICLR 2023
K ADDITIONAL ABLATION STUDY ON DISTILLATION LOSS
To test the effectiveness of suppressing forgetting when another task is used to build relation across tasks rather than consecutive tasks, we revised the original Eq. 2 as follows:
[-1
FDD Σ
U(b(mi (mm M)) M,
(16)
j=2i=1
t-2 N (m)
DD ΣΣ
(m. e M.
(17)
j=2
where LD.FSD represents case that the first task is used, and LD-LSD represents a case that the most recently learned task is used.
We tested the number of samples per task N that have to be used to efficiently utilize LD-CSD, All Split CIFAR 100, Split minilMN, and CORe50 benchmarks contain different classes in each task. As shown in Table 17 there was no difference in performance between N = 10 and N = 20. There was only a slight degradation performance for 5. When N greater than equal to the number of classes in each task, it is possible to build the reliable relation across tasks for distillation.
Table 17: Ablation study on the number of samples per task, N, for LD.CSD. Bold fonts represent the best performance.
Cs (CS)
ACC
=
N
20
Cifar100
75.4
75.910.4
76.3
minilMN
75.2+0066
7..4+0.4
75.3++0.9
CORe50
67.211.6
67.311.6
67.611..
L ADDITIONAL RESULTS OF STABILITY-PLASTICIYY NORMALIZATION MODULE
We reported the standard deviation results of 6 Table 18 due to the page limit.
Table 18: Standard deviation results of Table 6 on two continual learning benchmarks considered.
Norm.
BN
RUN
IN
GN
LN
SN
CN
SPN (Oun)
1A
1A
ACC LA LA
CDR:50
To demonstrate the effectiveness of SPN on existing CL methods, we additionally tested the difference in ER-Ring with SPN and other normalization layers. Table 19 shows that SPN still superior to both single and multiple normalization layers in ACC and LA. Also. the spatial normalization layer, IN, shows the highest stability. As an ablation study, we assessed SPN using only IN or LN for the stability normalization operation. /Huang et al. (2020) and Luo et al. (2018) validated that spatial normalization layers are vulnerable in stable training because they do not utilize global context. However, because spatial normalization layers do not utilize global moment, they are more robust to catastrophic forgetting. As shown in the results of Table 20 the combination of two spatial normalization layers can adapt the characteristic of data way of enhancing stable training while maintaining high stability.
23

---

Published as a conference paper at ICLR 2023
Table 19: Effectiveness of SPN on ER-Ring. Bold fonts represent the best performance in each evaluation metric.
ER-Ring
CORe50
Norm. Layer(s)
ACC
FM +
LA f
BN
45.11=2.15
8,82::0522
18.73+
RBN
50.76=0.86
8.3411.04
56.02+1.98
IN
44.66-2..6
4.8412.59
46.00=1.000
GN
40.84=1.79
7.9611.22
47.80
LN
39.54-2.50
9.08+1.39
45.3242.18
SN
50.20=0.39
12.08+1.17
60.30+
CN
51.68=0090
9.80+
59.28:10.63
BN + IN
50.38=1.95
9.95
60.06+1.511
BN + LN
48.04=0.87
10.20
18.16
SPN
52.80-2.34
10.64+2.36
61.90-11.66
Table 20: Ablation study of SPN using only IN or LN for the stability normalization operation on DER++. Bold fonts represent the best performance in each evaluation metric.
DER+
minilMN
CORe50
Norm.
ACC
FM
LA
ACC
BN + IN 58.2-0.7
9.2-10.6
65.4-103
43.4+117
7.0::1.9
47.5=2.0
BN LN
62.7=1.1
7.2+0.7
67.4+1.4
47.5+4.0
7.64334
52.2-229
SPN
6..8+1111
9,3+0.88
72.7+0099
55.0++0.9
9.0+0.9
62.1-0.2
24


Reviews:
[{"review_id": "xciffOPdyT0", "reviewer": "Reviewer_L1XG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the problem of stability and plasticity in continual learning in a fully online manner. They do that by introducing the extraction of multi-scale feature maps from shallow and deep layers of a pre-trained model, a structure-wise distillation loss across multiple tasks, and a parallel normalization module for stability and plasticity. The proposed approach outperforms most of the current SOTA methods in regard to accuracy (ACC), stability (FM), and plasticity (LA). In addition, the authors have performed extensive experiments in their ablation study foolproofing the significance of their approach.", "summary_of_the_review": "The authors have shown improvements over SOTA methods in the online continual learning paradigm. Their claims are supported with empirical evidence. In addition, each of the components in their work are contributing to the approach. I believe this would be a good contribution. Therefore, I am leaning towards accepting this paper; however, the authors will need to clarify the issues mentioned in the weakness section.", "strength_and_weaknesses": "Strengths\n1. The paper essentially describes a novel way to address the problem of catastrophic forgetting.\n   a. First, similar to the architectures used in the domain of semantic segmentation, extracting the features from shallow to deep layers becomes essential especially when extracting deep high-level features and shallow low-level features. In this task, the extraction of multi-scale feature maps helps the model learn its intrinsic features when implying the addition of the top-down module.  \n   b. Although the significance of sequential hierarchical training in the domain of continual learning has not been explored widely, the authors have used a similar concept to force the model to learn the relationship between two consecutive tasks by maintaining an angular distance. Based on their ablation study, this has shown some improvements in stability, but not so much in accuracy and plasticity. \n   c. The proposed parallel stability-plasticity normalization module tackles the variability issues of the incoming data and problems of normalization in the inference step. For example, the inconsistency in the data during training and inference can cause a significant drop in the performance of the previous task, leading to catastrophic forgetting. Here, this method finds a balance between stability and plasticity. \n2. The authors have performed comprehensive experiments against other SOTA methods. Their extensive ablation study further solidifies their claims. \n3. The authors have made use of all the most common benchmark datasets that are widely used in this area, furthering the compatibility of their approach in regard to its reproducibility. \n4. The paper is easy to follow. \n\nWeaknesses\n1. Why are the ablation studies in Tables 3 and 4 performed on ER-Ring? Also, it seems to be the case with table 6 as well with DER++. The authors’ proposed approach is MuFAN. (It is either not clearly stated in the paper or there can be a  typo)\n2. In table 9, it can be observed that by using all three proposed components and data augmentation, only the best overall accuracy can be achieved. However, as stated by the authors, for tackling the stability-plasticity dilemma,  row-8 has shown significant improvement in handling stability whereas row-7 handles plasticity quite well. Therefore, what is affecting these components when used altogether?\n3. The authors have mentioned that the architecture is similar to that of U-Net and discussed the improvements over utilizing standard and bottom-up extraction methods; however, as mentioned in this work [1], it would be interesting to see the compatibility of the top-down (without multi-scale feature extraction) approach with auto-encoders in their ablation study.\n\n[1] A. Rannen, R. Aljundi, M. B. Blaschko, and T. Tuytelaars, “Encoder Based Lifelong Learning,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Oct. 2017, pp. 1329–1337. doi: 10.1109/ICCV.2017.148. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "wx8oteDCw9J", "reviewer": "Reviewer_zSUJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method MuFAN that strikes a better stability-plasticity tradeoff compared to existing methods in an online (single-pass) continual learning setting. The main idea of the method is to use a pre-trained model as an encoder, aggregate the feature maps from different layers, and then pass the aggregated/ fused feature map to a classifier (typically a ResNet model). During training the pre-trained encoder is kept fixed and the classifier is updated. The authors propose a structured knowledge-distillation loss that maintains the angular distance between the successive tasks. Additionally, the authors propose a new normalization technique that splits the channels at a given layer and apply a combination of BatchNorm, InstaceNorm and GroupNorm on different splits of the feature map. The experiments are reported on several supervised continual learning benchmarks. \n", "summary_of_the_review": "Please refer to the strengths and weaknesses section above for details.", "strength_and_weaknesses": "**Strengths**\n\n1. The paper studies an important problem of online continual learning. \n2. The use of a pre-trained model as an encoder and fusing features of the encoder from different scales is a nice idea which is not explored in the context of continual learning extensively. \n\n**Weaknesses**\n\n1. **Overall the method seems very ad-hoc**: While the use of a pre-trained model as an encoder that generates a fused feature map for the classifier is a nice idea, the other two contributions of the paper seem very ad-hoc. The structured-distillation loss seems like a cumbersome way of enforcing knowledge-distillation, and it is not clear if the so-called point-wise knowledge-distillation on more points would achieve similar or better results than structured distillation. The overall training objective (Eq. 4) is the combination of everything, experience-replay, point-wise knowledge distillation, and structured distillation and it is not clear if everything is needed in that objective. Second, the SP-normalization seems even more ad-hoc where half of the features are passed through BatchNorm layers, and the other half are sent to a combination of GroupNorm and InstanceNorm, without any criterion. If BatchNorm is bad for continual learning then there would be excessive forgetting through the half that was sent to the batchnorm. As for Group or InstanceNorm, if the features drift from one task to the next, how well these normalization techniques would be able to cope with forgetting. Overall, it seems that in designing SP-normalization the whole kitchen sync is thrown at the features. \n\n2. **Writing is not clear**: The writing of the paper is not clear and makes the paper a difficult read. The tuple notation for denoting data samples is incredibly cumbersome and non-intuitive. Sec 3.2 is not clear at all, terms are used without any definitions. For example, the functional form of the so-called structure-wise potential $\\psi$ is not provided until the end of the section making it difficult to understand what is happening in Eq. 2. Similarly, throughout Sec 3, intuitions are lacking and it seems that everything is combined in an ad-hoc way. What the authors refer to as knowledge-distillation based on ER, is simple experience replay, there is no knowledge-distillation there. \n\n\n3. **Experiments**:\n\n* Do the baselines, GEM, ER, MER etc also use the pre-trained model? This is not clear from the experiments. \n\n* Do all the baselines use the same codebase?\n\n* Could the authors compare the compute and memory cost of their method vs the baselines?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OETdDzLlfS", "reviewer": "Reviewer_aHBY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use a pretrained model for its multi-scale feature map for better performance in continual learning along with cross-task structure-wise distillation and a new normalization layer. The proposed method is compared to recent work including DualNet (NeurIPS 2021), CTN (ICLR 2020), MIR (NeurIPS 2019) and etc on SVHN, CIFAR100, miniImageNet and CORe50 dataset. The proposed method improves the performance over the compared methods.", "summary_of_the_review": "Given that the paper uses seemingly expensive pretraining schemes and the gain is marginal except the gain by the existing idea of using multi-scale feature of pretrained model, the method is not very interesting to be reported in the community. Moreover the description (esp., the pretraining procedure) is not clear enough to judge the value of the work.", "strength_and_weaknesses": "**Strength**\n- S1: Pretraining the model for continual learning brings a noticeable gain in performance.\n\n**Weakness**\n- W1: Pretraining using the online stream and complicated normalization scheme incurs a long pipeline for learning. One of the most important feature in continual learning is the short training time (i.e., online learning). This proposal sacrifice the efficiency for the continual learning.\n- W2: Most of the gain comes from using multi-scale feature extracted from the pretrained model, which is an existing idea (not applied to CL context yet). Gains by other proposals are marginal (at the expense of larger model than prior arts (Table. 8).\n- W3: The `pretraining' is not well suited to continual learning setup as it learns the streamed data by storing them in advance (this part is not very clear in description) for representation learning. As the continual learning setup encourages to learn the data with limited access over the tasks, the notion of pretraining may not be widely applicable in practice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ASb_vf6Hkb", "reviewer": "Reviewer_dVei", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper argues that a good continual learner should strike a different stability - plasticity tradeoff in online and offline CL, the former being more stable and the latter more plastic. The authors propose a new method MuFAN, which comprises 3 new components : the use of multi-scale features of pretrained models, an added cross-task distillation loss, and a new normalization technique. The authors evaluate their method on standard online CL benchmarks, and provide some ablation studies to highlight the effectiveness of the components. ", "summary_of_the_review": "Overall, given that the main contribution relies on proposing a new way to leverage pretrained models, but that key baselines are missing from this analysis, my opinion is that the paper should make these changes and be resubmitted. A proper evaluation of the computation complexity of the method is missing as well. While these two key issues are not properly addressed, my rating will stay the same. \n\nPost rebuttal : The authors have addressed some of my concerns, so I am raising my score accordingly.", "strength_and_weaknesses": "Strengths : \nThe paper positions itself well in the current literature. \nThe paper is clear in detailing has many ablations to evaluate the utility of the proposed components\n\nWeaknesses : \n1. *On the stability / plasticity conclusions drawn from Figure 1*. \nThe authors argue because online methods have a lower accuracy after a training task (LA), then they are less plastic. I strongly believe that this is property is not intrinsic to online learning. Rather, this property is a direct result of **the number of training iterations spent on each task.** An easy way to increase the number of training iterations in the online setting is to simply do multiple gradient optimization steps (each with different rehearsal batches) as in ([1], [2]). It would be good for the authors to redo figure 1, where the number of iterations on a datapoint is equal to the number of epochs in the offline setting (so that online and offline methods use the same compute). \n2. *On the use of pretrained models in online CL* . \nThe authors propose a novel (to the best of my knowledge) yet quite convoluted way to leverage pretrained models. Yet, highly simple and straightforward baselines (such as a nearest-class-mean classifier in the feature space of the pretrained model, or downright finetuning of such model) are omitted. Given that there is ample evidence [3,4,5] that these methods are surprisingly effective, it is crucial that one compares to such naive approaches : if one cannot beat them, then why bother with the complexity of a new method ?\nHere is a small code snippet [6] that evaluates one such simple approach using the setting described on the Split CIFAR100 setting. **Without any training, one can boost the accuracy from 75% to 85% using solely the pretrained model!** \n3. *On the added complexity of the method*\nFrom the paper, it is unclear how many additional forward passes are required at each step. Looking at 1 and 2, it seems that the full buffer is forwarded through the model at each step ? If this is the case, it is important to note that this comes at a very high additional cost in compute, and should be properly discussed. Also, what is the cost of the additional forward passes through the old model for distillation ? Given that data augmentations are used, I am assuming hidden activations cannot be cached and have to be recomputed ? \nOn a similar note, it would be much more interesting to replace table 8 with a FLOP per iteration count for several methods. \nSimilarly, what is the additional compute cost of using a top-down module ? my understanding is that upsampling the representation before feeding it to the classifier will have a significant increase in compute cost. It would be nice if the authors could analyse this. \n4. *On the task-free version of MuFAN* \nThis version uses a hyperparameter $\\mathcal{S}$, which essentially determines the size of each task (in number of unique labels). In a \"real\" online setting, the learner does not know how long the learning stream is, nor does it know how many unique labels it will see. Therefore, how would one set $\\mathcal{S}$ a priori ? Moreover, in this section you are comparing to baselines which **do not** use a pretrained model; I don't see how this is a fair comparison\n\n\n\n[1] Drinking from a Firehose: Continual Learning with Web-scale Natural Language https://arxiv.org/abs/2007.09335\n\n[2] Online Continual Learning with Maximally Interfered Retrieval https://arxiv.org/abs/1908.04742\n\n[3] REMIND Your Neural Network to Prevent Catastrophic Forgetting https://arxiv.org/abs/1910.02509\n\n[4] A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning https://arxiv.org/abs/2210.04428\n\n[5] Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis https://arxiv.org/abs/1909.01520\n[6]\n```\nimport os\nimport PIL\nimport torch\nfrom torchvision import datasets, transforms\n\nif not os.path.exists('efficientnet_lite0.pth'):\n    os.system('wget https://github.com/RangiLyu/EfficientNet-Lite/releases/download/v1.0/efficientnet_lite0.pth')\n\nif not os.path.exists('eff_net_code'):\n    os.system('git clone git@github.com:RangiLyu/EfficientNet-Lite.git eff_net_code')\n\n# load torch model\nfrom  eff_net_code.efficientnet_lite import build_efficientnet_lite\nmodel = build_efficientnet_lite('efficientnet_lite0', 1000)\nmodel.load_state_dict(torch.load('efficientnet_lite0.pth'))\nmodel = model.cuda()\nprint(f'total of {sum(x.numel() for x in model.parameters()) / 1e6 :.2f} M params.')\n\n# remove linear head\nmodel.dropout = model.fc = torch.nn.Identity()\nmodel.avgpool = torch.nn.Identity()\nDIM = model(torch.cuda.FloatTensor(2, 3, 224, 224)).size(-1)\n\n# default transforms for the model in question\ntfs = transforms.Compose([\n    transforms.Resize(224 + 32, interpolation=PIL.Image.BICUBIC),\n    transforms.CenterCrop(224),\n    transforms.ToTensor(),\n    transforms.Normalize([0.498, 0.498, 0.498], [0.502, 0.502, 0.502])\n    ])\n\nds_train = datasets.CIFAR100('./', train=True, download=True, transform=tfs)\nds_test  = datasets.CIFAR100('./', train=False, download=True, transform=tfs)\nBS, n_cls, n_way = 256, 100, 5 # use n_way == 100 for the single-head task-free result\n\n\n@torch.no_grad()\ndef get_prototypes(model, dataset, n_classes):\n    protos = torch.zeros(n_classes, DIM).cuda() \n    counts = torch.zeros(n_classes, dtype=torch.int64).cuda()\n\n    loader = torch.utils.data.DataLoader(dataset, batch_size=BS, num_workers=8, shuffle=True)\n    offset = torch.arange(DIM).cuda()\n\n    print(f'processing : {(str(type(dataset)))}')\n    for i, (x,y) in enumerate(loader):\n        print(f'{i}/{len(loader)}', end='\\r')\n        x, y = x.cuda(), y.cuda()\n        feats = model(x)\n        \n        # accumulate\n        counts.scatter_add_(0, y, torch.ones_like(y))\n        idx = offset.view(1, -1) + y.view(-1, 1) * DIM\n        protos.view(-1).scatter_add_(0, idx.view(-1), feats.view(-1)).view_as(protos)\n\n    return protos / counts.unsqueeze(-1)\n\n\n@torch.no_grad()\ndef evaluate(model, dataset, protos, n_way):\n    loader = torch.utils.data.DataLoader(dataset, batch_size=BS, num_workers=8, shuffle=True)\n    n_cls = protos.size(0)\n\n    n_ok, n_tot = 0, 0\n\n    for i, (x,y) in enumerate(loader):\n        print(f'{i}/{len(loader)}', end='\\r')\n        x, y = x.cuda(), y.cuda()\n        feats = model(x)\n\n        dist = (feats.unsqueeze(1) - protos.unsqueeze(0)).pow(2).mean(-1)\n        pred = dist.argmin(1)\n        acc  = (pred == y).float().mean()\n\n        # since in CL we typically partition and eval on a subset of classes, let's mimick this\n        _, preds_in_order = dist.sort(1, descending=False)\n        pos_of_correct_answer = torch.where(preds_in_order == y.unsqueeze(-1))[1]\n\n        n_extra = n_way - 1\n\n        # if we pick any `n_extra` classes at random, what is p(correct) ?\n        p_correct = dist.new_ones(size=y.size())\n        for it in range(n_extra):\n            n_worse_left = ((n_cls - 1) - (pos_of_correct_answer) - it).clamp_(min=0)\n            n_better = pos_of_correct_answer # fixed\n            p_correct = p_correct * (n_worse_left / (n_worse_left + n_better))\n\n        n_ok  += p_correct.sum().item()\n        n_tot += x.size(0)\n\n    print(f'\\nTest Acc : {n_ok / n_tot  * 100 :.2f}')\n\nprotos = get_prototypes(model, ds_train, n_cls)\nevaluate(model, ds_test, protos, n_way)\n```", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster