Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Authors: Filip Szatkowski
Yaoyue Zheng
Fei Yang
Tomasz Trzcinski
Bartłomiej Twardowski
Joost van de Weijer
Venue: ICML 2025 poster (2025)
Keywords: Continual Learning
Class-incremental Learning
Early-exits
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We show how intermediate representations in neural networks can be leveraged to improve both performance and efficiency in continual learning through auxiliary classifiers (ACs)

Abstract: Continual learning is crucial for applying machine learning in challenging, dynamic, and often resource-constrained environments. However, catastrophic forgetting — overwriting previously learned knowledge when new information is acquired — remains a major challenge. In this work, we examine the intermediate representations in neural network layers during continual learning and find that such representations are less prone to forgetting, highlighting their potential to accelerate computation. Motivated by these findings, we propose to use auxiliary classifiers~(ACs) to enhance performance and demonstrate that integrating ACs into various continual learning methods consistently improves accuracy across diverse evaluation settings, yielding an average 10\% relative gain. We also leverage the ACs to reduce the average cost of the inference by 10-60\% without compromising accuracy, enabling the model to return the predictions before computing all the layers. Our approach provides a scalable and efficient solution for continual learning.

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Filip Szatkowski 12 Yaoyue Zheng 34
Fei Yang
Tomasz Trzcinski 178 Bartlomiej Twardowski 74
Joost van de Weijer
Abstract
CIFAR1 CIFAR100X10 00x10 ResNet32
Continual learning is crucial for applying ma- chine learning in challenging, dynamic, and of- ten resource-constrainee environments. However, catastrophic forgetting - overwriting previously learned knowledge when new information ac- quired - remains a major challenge. this work, we examine the intermediate representations in neural network layers during continual learning and find that such representations are less prone to forgetting, highlighting their potential to accel- erate computation. Motivated by these findings, we propose to use auxiliary classifiers (ACs) to enhance performance and demonstrate that inte- grating ACs into various continual learning meth- ods consistently improves accuracy across diverse evaluation settings, yielding an average 10% rela- tive gain. We also leverage the ACs to reduce the average cost of the inference by 10-60% without compromising accuracy, enabling the model to return the predictions before computing all the layers. Our approach provides scalable and effi- cient solution for continual learning.
45
40
Accuracy
35
30
20
30
50
60
80
100
NetgaNet100x100
ResNet18
65
69%
50
Accuracy
ANCL +AC
DER
LODE
SSIL
AAC
1AC
1AC
30
40
50
60
70
80
90
100
Inference cost [%)
Figure I. We integrate auxiliary classifiers (ACs) into various CL methods, enabling dynamic inference and reducing the inference We measure their accuracy relative to the standard network at different computational budgets and show that AC-enhanced methods match the performance of the standard counterparts only 50-80% cost, and improve their performance at higher compu- tational budgets. The accuracy AC-enhanced models saturates 50-90% computation, allowing us to save 10-20% of the inference cost without sacrificing accuracy.
1. Introduction
The ability to adapt to changing environments is crucial for practical machine learning applications, particularly in resource-constraiig settings where computational effi- ciency essential. Continual learning provides the theoreti- cal foundations and algorithms for learning from non-i.i.d. data streams in such environments (De Lange al., 202
The most common continual learning scenario involves learning from sequential tasks, where the learner cannot access previously seen tasks when learning new ones. The tasks may differ in data distributions (domain-incrementa learning) or introduce new classes (class-iccremental learn- ing) and may or may not provide task identity during classi- fication (task-incremental learning) (Van de Ven & Tolias, 2019). The primary challenge in continual learning is to avoid catastrophic forgetting significant drop in perfor- mance on past tasks while learning on the new data (Mc- Closkey & Cohen, 1989; Kirkpatrick et al., 2017). Vari- ous strategies including parameter isolation (Rusu et al., 2016; Serra al., 2018; Mallya Lazebnik, 2018), weight
Warsaw University of Technology PIDEAS NCBR 'Institute of Artificial Intelligence and Robotics, Xi'an an Jiaotong Univer- sity, China *Computer Vision Center, Barcelona 'VCIP, Col- lege of Computer Science, Nankai University 'NKIARI, Shen- zhen Futian IDEAS Research Institute "Tooploox Universitttt Autonoma de Barcelona. Correspondence Fei Yang <feiyang@
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Improving Continual Learning Performance and Efficiincy with Auxiliary Classifiers
1IFAR100x10 FFT
CIFAR100x10 FT+EX
CIFAR100x10 IIC 0.00 0.98 0.96 0.98 0.98 0.90 0.98 0.95 0 0.92 0.50
10FAR100x10 LwF
AC1
0.95 0.93 0.92 87 0.83 0.77 0.35 0.64
0.92 0.88 0:87 0.98
AC2
1.92
0.89 0.89 0.57 2. 0.80 0.84 0.80 0.78
0.94 0.30 0.88 0.87 0.85 0.84 0.84 0.82
0.99 0.98 0:57 0.97 0.97 0.98 0.99 0.92 0.30 0.58 0.87 0.58
AC3
0.87 0.87 0.84 0.63 0.79 0.03
0.80 0.70
0.93 0.89 0.88 0.07 0.85 0.86 0.55 0.99
0:95 0.96 0.96 0.95
0.98 0.59 0.91 0.90
AG4
3.84 0.85 0.00 0.75 0.79 0.76 0.74
0.81 0.87 0.86 0.84 0.04 0.82 0.82
0.90 0.97 0.96 .6 0:95 0.85 0.90 0.84 0.97 0.91 0.59 0.58 0.07 0.8G 0.55
AC5
0.61
0.77 0.78
0.72
0.74 0.7%
0.86 0.59 0.90 0.78 0.78
0.96 0.96 0:93 0.99 0.52
0.95
0.87 0.56 0.84
AD6
0.62 0.63 0.65 9.57
0:61
0.77
0.73
0.71 0.70
0.93 0.91 0.85 0.87 0.56
0.91
0.79 0.77
0.75 0.76 0.74
Finel
6.41
0:41
8242 0.42 0:31
0.46
0.30
03
0.00 0.24
0:60 07 65 62
0.91 0.00 0.06 0.54 0.84 0.84 0.03
0.09 0.02 75 0.72 0.65
057
0.00
3
4
5
8
9
10
2
3
4
8
9
10
2 3
7 8
10
2
6 Task index
7
8
10
Task index
Task index
Task index
Figure 2. CKA the first task representations across different ResNet32 layers (L..BB-LL3.B5) through continual learning on CIFAR split into 10 tasks. Representations at the early layers are more similar across the continual learning. hinting at the potential for more stability that could be leveraged to improve the performance by incorporating auxiliary classifiers at the intermediate layers.
and data regularization (Kirkpatrick et al., 2017: Li & Hoiem, 2017), and rehearsal methods (Rebuffi al., 2017; Chaudhry et al., 2018) have been proposed to address this challenge. Despite these efforts, continual learning, particu- larly class-incremental learning, remains an open problem.
intermediate representations in continual learning scenarios, suggesting that utilizing additional classifiers built top of these representations could effectively reduce forgetting.
Motivated by the insights from our analysis, we adapt and integrate auxiliary classifiers (ACs) into various well- established continual learning strategies (LwF (Li & Hoiem, 2017). EWC (Kirkpatrick et al., 2017), ER (Chaudhry et al., 2019), BiC (Wu et al., 2019), SSIL (Ahn et al., 2021), ANCL (Kim et al., 2023), LODE (Liang & Li, 2024), DER++ (Buzzega et al., 2020)) and demonstrate that they consistently outperform single-classifier methods on stan- dard benchmarks such as CIFAR100 and ImageNet100. Since ACs enable dynamic inference and control of the network computation through early exit, we also explore the efficiency gains offered by their introduction. As shown in Figure 1, AC-based networks with dynamic inference can maintain the performance of single-classifier models while using only fraction of the computation. Furthermore, at 80-90% of the original network's 's computational cost, per- formance reaches a saturation point, which means that our method provides lossless acceleration. Our results demon- strate that ACs can be seamlessly integrated into common continual learning methods. yielding consistent performance improvements and computational savings without the need for extensive hyperparameter tuning.
Several studies have investigated knowledge transfer and accumulation in deep neural networks through intermediate representations (Evci et al., 2022; Jung et al., 2023), and works such Liu et al. (2020a); Ramasesh et al. (2020); Zhao et al. (2023); Masarczyk et al. (2023) revealed that later layers undergo the most significant changes during continual learning. However, these insights have yet to be leveraged to enhance continual learning performance. Inde- pendently, work on neural network efficiency has introduced early-exit classifiers (Panda et al., 2016; Teerapittaynoon et al., 2016), which utilize intermediate layers to make pre- dictions before reaching the final classifier. Such classifiers not only accelerate inference but can also improve accu- racy by addressing "overthinking" phenomenon where inputs that could be correctly classified earlier are unnec- essarily processed by deeper layers, sometimes leading to errors (Kaya et al., 2019). In this paper, we bridge these two research areas and propose use auxiliary classifiers (ACs) in continual learning to capitalize the lower forgetting observed in early layers and enable computational savings during inference.
ACS provide a high-performing alternative to standard meth- ods in scenarios where faster inference or flexible resource usage is essential. Our contributions can be summarized as:
We begin our work with a detailed analysis of intermediate representations in continual learning. First, we examine the stability of representations across different network layers using CKA (Figure 2) and find that early layer representa- tions are more stable throughout the training. Building on this insight, we explore the discriminative power of these representations and reveal that, surprisingly, auxiliary classi- fiers (ACs) trained on top of the early layers can significantly outperform the final classifier on older tasks. To dive deeper into this phenomenon, we compare overthinking in continu- ally trained networks with models trained in i.i.d. settings, discovering that overthinking is far more pronounced in continual learning. Our findings highlight the potential of
We analyze intermediate representations in continual learning (CL) and show that classifiers built on such representations are less prone to forgetting.
We propose to use auxiliary classifiers (ACs) in CL and show that our idea yields an average 10% relative improvement over single classifier alternatives across diverse benchmarks and architectures.
We leverage ACs to reduce the average cost of infer- ence by 10-60% without sacrificing accuracy by allow- ing early prediction through dynamic inference.
2

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
CIFAR1 FT
CIFAR100X10 FT+
CIFAR100X10 LWF
CIFAR100X10 |IC
35
Accursey affference
225
30
(20
25
3
15
10
2
15
2
6
1
NG1
AG2
ADD
AD4
A.G5
AC6
AG1
AG2
AG3
AC4
NG5
AGS
AG1
AG2
AG3
AG4
ACS
ADS
AC1
AC2
AC3
NC4
AC6
ACG
Figure 3. Per-task difference (only positives) in accuracy between the auxiliary classifiers (ACs), trained with linear probing on intermedi- ate layers, and the final classifier. Surprisingly, in continual learning, some intermediate classifiers can significantly outperform the final classifier on the old task data, especially for exemplar-free methods (FT and LwF).
2. Related works
2016; Kaya et al., 2019; Wójeik et al., 2023) that attach intermediate classifiers to the model were developed based on the observation that their representations could be used for classification to allow skipping later model layers and re- duce inference cost. While works such as Liu et al. (2020b); Yan et al. (2024) use multiple classifiers to improve on- line CL through specialized techniques such as ensembling, self-distillation, and contrastive learning, our approach ex- plores intermediate classifiers as a general framework that enhances offline dlass-incrementll learning performance and efficiency and can be applied to most continual learning methods and architectures.
Continual learning. Continual learning methods (Parisi et al., 2019; De Lange et al., 2021; Masana et al., 2022) can be broadly categorized into three types: regularization- based, replay-based, and purameter-isolation methods. Regularization-basse -based methods typically introduce reg- ularization term to the loss function to constrain param- eter changes for prior tasks, with subeategories of data- focused (Li & Hoiem, 2017; Kim et al., 2023) and prior- focused (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018) approaches. Recent research also enforces weight updates within the null space of feature covari- ance (Wang et al., 2021; Tang et al., 2021). Replay- based methods use memory and rehearsal mechanisms to recall past tasks during training, maintaining low loss on those tasks. Two main strategies are exemplar replay which stores selected training samples (Riemer et al., 2018; Buzzega et al., 2020; Chaudhry et al., 2018; Prabhu et al., 2020; Chaudhry et al., 2019; Liang & Li, 2024), and generative replay, where models synthesize previous data using generative models (Shin al., 2017; Wu al., 2018). Parameter isolation methods learn task-specific sub- networks within a shared network. Techniques like Piggy- back (Mallya et al., 2018), PackNet (Mallya & Lazebnik, 2018), SupSup (Wortsman et al., 2020), HAT (Serra et al., 2018), and Progressive Neural Networks (Rusu et al., 2016) allocate and combine parameters for individual task. While effective in task-aware settings, these methods are most suited for scenarios with known task sequence oracle.
3. Intermediate representations in CL
In this section, we analyze the stability of intermediate representations in continual learning and how auxiliary clas- sifiers (ACs) trained on these representations can reduce forgetting. We train neural network over the 10 tasks of split CIFAR 100 (Krizhevsky, 2009) in class- class-incremental learning setting (De Lange et al., 2021; Masana et al., 2022), aiming to classify new classes while avoiding catastrophic forgetting of previous ones without access to task identity at prediction time. For the analysis we train ResNet32 (He et al., 2016) with different approaches for continual learn- ing: naive finetuning (FT) without any additional contin- ual learning technique, finetuning with exemplars (FT+Ex), exemplar free regularization with LwF (Li & Hoiem, 2017), and BiC (Wu et al., 2019) with both regularization, exem- plars and an additional bias correction. We analyze rep- resentations at 6 intermediate layers and the final feature layer before the classifier, referring to the intermediate clas- sifiers as AC1-AC6. See Appendix A for setup details and Appendix F.3 for detailed description of our approach, further analysis, and more results.
Representations in neural networks. Understanding and comparing representations (Kornblith et al., 2019; Davari et al., 2022) at different layers in deep neural networks is an active area of research on transfer learning (Boschini et al., 2022) and continual learning (Ramasesh et al., 2020; Zhao al., 2023; Masarczyk al., 2023). Motivated by such analyses, several continual learning methods leverage intermediate layers for replay (Liu et al., 2020a; Pawlak et al., 2022) or regularization (Douillard et al., 2020). Early- exit techniques (Panda et al., 2016; Teerapittayanon et al.,
3.1. Intermediate representations are more stable
We present the representational similarity of the interme- diate and final layers in Figure 2 using Centered Kerel Alignment (CKA) (Kornblith et al., 2019). CKA measures
3

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
16
14
Oveettinkng (%6)
40
Lkique oweribbkgg
1/2
2
x
18
18
AC2
AC3
AGA
AC8
AC1
AC2
AC3
AC5
ACB
AC2
AC3
ACA
AC5
ADS
Frx
(a) Overthinking CL
(b) Overthinking per AC
(c) Unique overthinking
(d) AC training changes
Figure 4. Overthinking and AC performance analysis on CIFARI00X10. Overthinking refers to a case where samples correctly classified by early classifiers are misclassified later by the final classifier. (a) Overthinking is much more prominent in continual learning methods than in standard joint training, which indicates that the accuracy of continual learning could be greatly improved through ACs. (b) Each classifier correctly classifies significant portion of the samples misclassified by the final classifier. (c) Subsets of samples can be correctly classified only by single given AC. (d) Training ACs together with final networks improves the performance of most classifiers.
the similarity of two sets of representations by comput- ing the normalized Hilbert-Schmid Independence Criterion (HSIC) between their kernel. It is invariant to orthogonal transformations and isotropic scaling, making it suitable for comparing learned features across layers or models. We measure the similarity between the representations of the first task data learned after the initial task and the repre- sentations learned after each subsequent task. Across all methods, early layer representations change less and exhibit more stability. As expected, the most significant changes oc- cur in FT. Continual learning methods, such as FT+Ex, LwF, or BiC, show less change due to replay or regularization strategies that enforce stability, though the trend of early- layer stability remains. Our findings are consistent with the observations in previous research (Ramasesh et al., 2020; Zhao et al., 2023) and suggest that intermediate representa- tions more robust in continual learning, and classifiers built on top of those representations should suffer from less forgetting than the final classifier.
older tasks across all methods outcome not typically in i.i.d. settings. The most interesting results can be observed for the exemplar-free methods: in FT, intermedi- ate classifiers achieve the highest accuracy for most of the tasks, and in LwF, the intermediate classifiers outperform the final classifier by a significant margin on the older tasks. For exemplar-based methods, deeper ACs also surpass the final classifier, though the differences are less prominent, and earlier classifiers do not exceed the final one as in the exemplar-free cases. Generally, early classifiers exhibit more stability, while later classifiers provide better ability to discriminate between classes, Our results suggest that selecting the appropriate AC could improve performance in continual learning, especially on older tasks.
3.3. Continually trained networks overthink more
Overthinking (Kaya et al., 2019) refers to cases where inter- mediate classifiers correctly classify samples that the final classifier misclassifies. This concept emphasizes that us- ing classifiers that operate on intermediate representations can not only accelerate inference but also enhance network accuracy. To formally define overthinking, we consider a multi-classifier network evaluated under an oracle prediction rule, where a sample is considered correctly classified any of the intermediate or final classifiers predicts the correct label. We denote the accuracy under this rule as Accoractes and let Ace represent the accuracy when using only the prediction of the last classifier. Overthinking is then mea- sured as the performance gap: 0 O= ACCoracte AcCfinals and we also define relative overthinking as the normalized quantity O O/Accfinat- Initially, overthinking was studied in intermediate classifiers under i.i.d. settings. However, our prior analysis indicates that similar effect occurs in continual learning, particularly for older tasks, where aux- iliary classifiers (ACs) experience less forgetting than the final classifier. Therefore, we investigate overthinking in continual learning methods and compare it to the standard
3.2. Early layer classifiers perform better on old data
Knowing that early layers produce more stable represen- tations in class-incremental learning, we now assess their usefulness for classification. To assess the discriminative power of intermediate representations, we use linear prob- ing (Davari et al., 2022). Specifically, we attach ACs to the intermediate representations of the network and train them alongside the main model in a continual scenario. To prevent interference with the representations during the learning of the ACs, we detach the gradients between the ACS and the rest of the network. Given the high dimensionality of the representations, the ACs consist of a simple pooling layer followed by a linear layer. Then, we compare the final accu- racy of the learned ACs with that of the final classifier on per-task basis and show the differences in Figure 3.
ACs perform surprisingly well, with the penultimate classi- fier's accuracy matching or surpassing the final classifier on

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
i..dd joint training scenario in Figures 4a to 4c.
Feature Extraction Network
Less
Networks trained using different continual learning meth- ods exhibit a higher degree of overthinking compared to those trained under standard joint training (i.i.d. setting). To highlight the difference, we plot relative overthinking in
old talk data
new task datts
N-1
Figure 4a. The more pronounced overthinking in continual learning aligms with our earlier findings, where intermediate classifiers perform surprisingly well and often outperform the final classifier older tasks. In Figure 4b, show the proportion of samples misclassified by the final layer that each AC correctly classifies, that is how much given classifier contributes to overthinking. While the later layers correctly classify larger subsets of the samples, early lay- ers still can contribute towards improving the performance, especially in exemplar-free methods. Since the correctly classified subsets might overlap between the classifiers, in Figure 4c, we further investigate how diverse the learned classifiers are by measuring unique overthinking. Specifi- cally, we measure which samples, misclassified by the final layer, are correctly classified only by a specific AC, which emphasizes the diversity of the learned classifiers. The heightened overthinking continual learning suggests that ACS preserve valuable knowledge that the final classifier struggles to retain. Unlike in the i.idd. joint training case, where ACs primarily serve to speed up inference, our analy- sis suggests that in continual learning, they could also play a crucial role in improving overall performance.
9
y
Preduction
X
X
Prafiction
X
9
Auxiliary Classifier
9 Final Classifier
Figure 5. Overview of the network enhanced with ACs in continual learning. The early layers exhibit less forgetting on the old tasks, and can return the correct prediction (V) in cases where the final classifier fails (X), and save computations.
4. Enhancing CL through ACs
4.1. Combining from multiple classifier predictions
Our analysis Section 3 demonstrates that auxiliary clas- sifiers (ACs) can learn to classify different subsets of data than just standard, single-classifier network, which hints that combining their predictions should yield improved accu- racy. Therefore, we advocate the use of such multi-classifier networks in continual learning. Formally, we consider a neural network composed of backbone ʃ = J'((..((/ (x))) and final classifier g. where fi...., fN are submodules in the backbone. The standard network prediction y for a given input x can be written as y (((())). We intro- duce additional N - auxiliary classifiers 9: top of the backbone sub-module modules f2.... fN-1- During infer- ence with such muuti-classifier network, we obtain N pre- dictions: 31 = (I) = (e([{(()),,... UN-1 gN (f==) yn = =[f(x)) and select the prediction y where the class predicted by the corresponding proba- bility distribution pa has maximum confidence. Therefore, we return yarg mei.... max, (0)) where p. repre- sents the predicted probability for class in the distribution Pi. We refer this simple inference paradigm as static inference and use it in most of the experiments, it performs well across all tested settings. We show the conceptual diagram of our network in Figure 5.
3.4. End-to-end training improves ACs
Although ACs trained through linear probing show promise for improving performance in continual learning, it is un- clear how end-to-end training with enabled gradient propa- gation would affect the network. To explore this, we train the network with gradient propagation and compare the final average accuracy across all tasks with linear probing classi- fiers. We show the difference in the classifiers' performance in Figure 4d. End-to-end training improves the performance of early-stage classifiers, though it may lead to a slight de- crease in performance for the final classifier in some cases (e.g., LwF). (wF). However, in exemplar- based methods, we see significant accuracy improvements in intermediate layers with no degradation in any of the layers. The gains in the exemplar-based approach are likely due to the network's enhanced ability to retain knowledge during training, which aligns with our earlier findings in Section 3.1.
find
Our analysis shows that classifiers trained end-to-end gen- erally achieve better accuracy in non naive continual leamn ing methods (a broader comparison is provided in Ap- pendix D.3). Building these findings, the next section further investigates how end-to-end trained ACs can be uti- lized to enhance both the accuracy and efficiency of the continual learning approaches.
4.2. Dynamic inference with ACs
Inspired by the early-exit models (Panda et al., 2016; Teer- apittayanon et al., 2016; Kaya et al., 2019), we also consider using ACs as a means to reduce the average computational cost of the classification through dynamic inference. Specif- ically, we perform inference sequentially through the clas-
5

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Table 1. The final accuracy of diverse continual learning methods enhanced with auxiliary classifiers (ACs) CTFAR 100 and ImageNet benchmarks. Adding ACs improves the performance of all methods across both benchmarks, demonstrating the robustness of our idea.
Method
FT
FT+Ex
GDumb
ANCL
BiC
DER++
ER
EWC
LwF
LODE
SSIL
Avg
CIFAR100
Base
38.35 18.6
19.09;04
37.71
38.96
34.55
18.95
38.26
42.82
34.60 45.62
+AC
28. u8
38.75
23.29
39.83
20.40
43.49
39.77
28.96
40.55
49.13
48.35
39.15
+0.39
+4
+2.12
22.74
+4.53_30r
+5.22
+10.02
+2.29
+6.31_oo
+4.55 +2.72
CIFAR100x10
Base
34.51 10.27
22.22
30.69
38.54_ca _oe
32.31
10.20
29.56,.o4
38.87
42.29 1ive
30.21
+AC
34.97.4 1648
27.74gam
46.19.147 31.37_am
39.64_1 uuee
37.32
30.31 19.12
45.67_4ss
44.17
34. 34.13:4m
+6.62
+2.466am1
+5.52
+0.68
+3.31
+1.10-um
+5.01 -oom
+8.92
+0.74
+6.80
+3.91 +1.88
ImageNet100x5
Base
23.27
44.05
21.29
60.79
62.55
45.33-125
38.65
23.36-224
59.60ex27
49.88
60.54
44.48 eun
+AC
34.93
46.75 pus
25.30
62.99 1:20
65.22
54.14 100
44.46
35.09 am17
61.07=eu5
56.23 8/6
63.89 m..
50.01 cus
+11.67
42.71
+4.01 so:
+
+2.67
+8.81=1
+5.81 scow
+11.73 san
-7.47
-6.
+3.35
45.5
ImageNet100x10
Base
14,40
35, is
22.55
49.96
56.32as41
38.45-19
32.45
14.69
49.15
45.75m030
56.35
37.82a03
+AC
22.14.am
39.26.a41
52.07 25.93
57.23
45,70 ...
37.10.12
23.25
51.39__eo 19m
57.71:00
41393.4m
A
+7.74gmm
+3.32.004
+3.3819m
+2.11
+0.91 gai
-7.25_115
+4.65
48.56
+0.36
_5.64__000
+1.35g09
+4, +4.12
sifiers ②1, $2.92...... .......... 9, and at each stage i, we compute the probability distribution pr corresponding to the prediction of i-th classifier. If the confidence exceeds a threshold X, we return the corresponding prediction y;- If no prediction satisfies the threshold, we use the static inference rule to determine the prediction. Formally, we define this as:
2017), ER (Chaudhry et al., 2019), DER++ (Buzzega et al., 2020), BiC (Wu et al., 2019), SSIL (Ahn et al., 2021), ANCL (Kim et al., 2023) and LODE (Liang & Li, 2024). For all the methods, we replicate the method logic (loss) across all the classifiers and do not introduce classifier-ppecific parameters. If the original method introduces hyperparam- eter, we use the same value for this hyperparameter across all the classifiers. We also use the same batches of data for each classifier during the training. Similar to (Kaya et al., 2019), to prevent overfitting the network to the early layer classifiers we scale the total loss of each classifier according to its position so that the losses from early classifiers are weighted less than the losses for the final classifier.
1 1mii(ii Humaz P. 1 2x}
if such i exists, otherwise.
y
(1)
I max:e(1......)) maxe p! (b)
By varying the confidence threshold, one can trade off the amount of computation performed by the network for slightly lower performance, which allows such a model to be deployed in settings requiring computational adaptability.
Note that our use of ACs different from the early-exit literature, where the model accuracy usually monotonically improves when going through subsequent classifiers and the model returns the prediction of the last classifier in case no classifier can satisfy the exit threshold. As shown in Section 3, in continual learning the accuracy and quality of intermediate predictions significantly vary for different tasks, and the last classifier is not always the best one for a given subset of data. Refer to Appendix F.7 for a comparison between the performance of the standard early-exit inference rule and our method in continual learning setting.
5. Experimental results
In this section, we present the experimental results for AC- enhanced networks in CL. All our experiments are con- ducted with the FACIL (Masana et al., 2022) framework. We perform experiments on CIFARIOM (Krizhevsky, 2009) and ImageNet100 (the first 100 classes from ImageNet (Deng et al., 2009)). split into tasks containing different numbers of classes. Unless stated otherwise, we report average accuracy across all tasks at the end of the training. In Appendices F. and F22, we additionally provide the isolated accuracies of the networks on each task across the training and the final forgetting after training on all the tasks. For all exemplar- based methods (BiC, DER++ ER, GDUMB, LODE, and SSIL), we maintain a fixed-size memory budget of 2000 exemplars, updated after each task. We report results av- eraged over three random seeds. More details about our experimental setup can be found in Appendix A.
4.3. AC-enhanced CL methods
To demonstrate the effectiveness of our idea, we extend several continual learning methods with auxiliary classi- fiers (ACs) and examine their performance. In total, we investigate ACs with the following continual learning meth- ods: FT (Masana et al., 2022), GDUMB (Prabhu et al., 2020), EWC (Kirkpatrick et al., 2017), LwF (Li & Hoiem,
6

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
5.1. Improved CL with ACs
CIFAR100 ResNet32
ImagsNet100xs ResNet18
First, we demonstrate the effectiveness of our approach in standard CL settings. For our main experiments, we use
Aocurasy
50
well-established architectures: ResNet32 for CIFAR100 and
30
40
80
Drees
R (He et al., 2016) for ImageNet100. In both cases, following Section 3, we use six ACs. For ResNet32, we follow the previously described AC placement, while for ResNet18, we attach ACs to the six residual blocks between the first and last one.
Inference [1]
Inferenca [%]
Figure 6. Dynamic inference 5 tasks for AC-enhanced networks on CIFARI00 and ImageNet100. ACs improve accuracy while reducing computational cost, offering selectable orf We mark every 10% dynamic inference threshold with dots.
Additionally, in Appendices C.I and C.2, we show that our findings extend beyond the standard 5- and 10-task splits and explore warm-start continual learning (Goswami et al., 2024), where half the data is used for the first task, as well as continual learning on more challenging, fine-grained 20- and 50-task sequences.
accuracy. To unlock those savings, is essential to properly set the exit threshold X. Therefore, an important factor to consider is how sensitive to this threshold is the dynamic inference. In our experiments, AC-enhanced methods con- sistently perform better than the baselines at the confidence thresholds above 70%, and there is no noticeable drop in the network accuracy the thresholds above 90%. These results indicate that dynamic inference can be easily tuned to pro- vide meaningful computational savings. Moreover, we high- light how at the test-time ACs enable smooth adjustment of the average computation spent on the inference without the need for any further training. Further cost-accuracy plots for all methods from Table can be found in Appendix E.
Standard CL benchmarks. We evaluate our approach on 100 and ImageNet100, each split into and 10 equally sized, disjoint tasks, and present the main results in Table 1. Across all methods and settings, adding ACs consis- tently improves final performance, with the average relative improvement exceeding 10% of the baseline accuracy in ev- ery scenario tested. Interestingly, naive finetuning and EWC exhibit particularly strong gains, highlighting the potential of our simple yet effective idea. Exemplar -kemelar-basedd methods (GDumb, DER++ FR, LODE) benefit more from ACs than one with additional distillation (ACNL, BiC, SSIL), which suggests that distillation may hinder ACs' ability to diver- sify and mitigate forgetting. Overall, our approach reliably enhances performance for all methods across all scenarios. In the next paragraph, we explore how ACs can be leveraged to also improve inference efficiency.
5.2. ACs with deeper networks
In this section, we conduct experimental evaluations of our approach on larger and deeper models, specifically VGG19 (Simonyan & Zisserman, 2014) and ViT (Doso- vitskiy, 2020), which helps us gain a deeper understanding of how the impact of ACs relates to network size. Our re- sults demonstrate good scaling of the ACS with network size, highlighting the robustness of our approach.
Leveraging ACs for dynamic inference. The previous section demonstrates that ACs improve performance in con- tinual learning when performing the inference through the full network. However, ACs also can accelerate the network inference, as described in Section 4.2. To show how our approach enhances the effectiveness of continual learning methods, we evaluate selected techniques 10- and 5-task splits of CIFARI00 and ImageNet100 using dynamic infer- ence. We conduct the evaluation with À E 21, 100} % to assess the accuracy of the methods achieved at a given computational budget. We measure the inference cost in FLOPs, and report it relative to the non-AC network. Our results are presented in Figures 1 and 6.
Number of ACss with deep CNN. We investigate the im- pact of the number of ACs in the deep convolutional VGG19 model, which allows us add more classifiers. We eval- uate three AC-enhanced VGG variants: 1) 18 ACs at all 18 intermediate layers, 2) 10 ACs at every other convolu- tional layer and both fully connected layers, and 3) 6 ACs at every fourth convolutional layer and both fully connected layers. We present the final accuracy for those setups in Ta- ble 2, with dynamic inference curves for selected methods enhanced with 18 ACS displayed in Figure 7.
In all configurations, AC-enhanced methods outperform their baseline counterparts. As expected, denser AC place- ments generally lead to better performance. Using more ACs appears to be reliable strategy for improving per- formance, although some outliers (e.g. DER++) perform better with fewer ACs. Interestingly, for VGG19 archi- tecture, the ACs provide a more substantial performance
AC-enhanced networks match the accuracy of the single- classifier alternatives while using only 50%-70% of the computation on CIFARI00 and 40%-60% on ImageNet Notably, for most methods, performance stabilizes at 70%- 90% of the computational cost, which indicates that sub- stantial computation savings are possible without sacrificing
7

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Table 2. VGG19 enhanced with different numbers of ACs on CIFAR100. ACs integrate seamlessly with deeper network architectures, and the higher number of classifiers usually provides more significant gains. The brighter colors correspond to the better scores.
Method
FT
FT+Ex
GDumb
ANCL
BIC
DER+
ER
EWC
LwF
LODE
SSIL
Avg
CIFAR100X5
Base
1014
42.56_m
26.64g124
40.7310m
52.75
4
33.821800
18.7219%
39.54,05
46.69_os1
47.79_m11
+6AC
26.71
42.78
29.61
47.64
56,62
51.16
37.45
26.98
44.8
52.87 52m03
424s
+10AC
29.16gam
43.05
31.36iam
49.12
58.05
51.03gom
39.06
29.401a3:
16.51
50.39
55.30.mee
43.86gam
+18AC
31,47a044
43.53mm0
31,06+ax:
48.49
59.03e04
50.67=0a)
39.91
30.66
48. 48.22+aus
51.27
56.35
44.6
CI
Buse
9.52sm17
34.20=00
28.79 2
19,50
44,30
41.88
28.85
4.37
21. 21.044077
40.08
42.49
29.09
+6AC
16.73_am
35.29_aa
26.961118 30.
50.36
45.02_mm
32.48
17.161937
45.67 1008
47.39
34.26
+10AC
18.91=036
36.99-aus
31.55+aas
29.73++04
52,69+0us
43.94-us
33.95
19.78+
31,28+032
46,27=0as
48.29-1u2
35.76+aus
+18AC
21.16=uu
37,54-auv
52,56
43.94-0sz
34.86
20.54+02a
32,54-002
47.62-m04
49.68
e6.79
boost than for ResNet32, and the best 18-AC setup yields average relative improvements of approximately 20% and 25% for and 10 tasks, respectively. Notably, for 10 tasks, VGGI9 with ACS achieves a 2.5% accuracy improvement over AC-ResNet32 across all methods, even though the non-AC baseline VGG19 is average 1% worse than the baseline ResNet32. This suggests the potential gains from ACS are higher in deeper networks. We also hypothesize that VGG may develop more diverse and robust classifiers than ResNet due to the absence of residual connections, which further benefits our approach.
CIFAR100x10 VGG19
ImageNett ViT-baae
35
Accuracy
30
30
25
20
ANKCL
LODE
591
+6G
-40
20
40
60
800
100
20
40
00
80
100
Inference co5t (75
Inference cost (N)
Figure 7. Dynamic inference plots for selected CL methods ex- tended with ACs for VGG19 and ViT trained for 10 tasks from scratch on CIFARI00 and ImageNet100. Dynamic inference with ACs is even more beneficial with bigger networks. We show the results for all the methods in Appendices E.2 and E.3.
In Figure 7, we show that the VGG19 enhanced with 18 ACS is also well-suited for dynamic inference, matching baseline performance at just 20-20044000 40% of the full model's 's computation for non-naive methods and achieving the maximum accuracy at as few as 40% of the computation. We present detailed re- sults for all VGG19 methods in Appendix E.2. Additionally, in Appendices D.1 and D.2, we explore AC placement and architecture for ResNet32, and in Appendix B we quantify the training overhead of our method. We find that the impact of using more ACs is more pronounced in larger networks, where the potential for reducing inference cost is higher.
(2022), ViT is notoriously data-hungry. Therefore, when trained from scratch in continual learning scenarios, it tends to perform less effectively than ResNet18. However, ACs still integrate effectively with the ViT architecture, deliver- ing superior results compared to the baseline counterparts across all methods. AC-enhanced methods achieve perfor- mance on par with their standard counterparts at particularly low 20-40% computation and quickly reach the saturation point, attaining full performance at 50% of the computa- tional budget. These findings highlight the scalability of our approach and its compatibility with larger models, where the benefits and computational savings are even more pro- nounced due to richer representations.
ACS in Vision Transformers. We further explore the com- patibility of ACs with larger models by continually training base Vision Transformer (ViT'). ). To ensure a fair comparison with prior experiments, we train ViT from scratch, incorpo- rating ACs into each transformer block (11 blocks in total). Following the standard classifier design in ViT, we perform classification using the [CLS] token and implement ACs with single LayerNorm followed by a linear classification layer. This AC architecture is notably lightweight compared to the standard transformer block, which makes integrating ACS with transformers particularly seamless. In Figure 7, we present dynamic inference results for ViT with a selected subset of methods on the 10-task split of TeageNer100.
5.3. ACs with wider networks
Wider networks offer richer, higher-dimensional feature rep- resentations, but they may also impose greater overhead on attached classifiers. This makes it particularly interesting to assess how network width affects the applicability of our approach. To this end, evaluate AC-enhanced models using WedeRes (Zagoruyko & Komodakis, 2016) on CIFAR 100. This architecture is shallower than ResNet- 32 but twice as wide, with approximately 0.7M parameters compared to 0.46M in ResNet-32. To measure performance under our framework, we attach five additional classifiers
As highlighted in works like Pan et al. (2022); Wang et al.
For simplicity, we refer to ViT-b-16 as VIT in this work.
8

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
C/FAR100X5 Nide
CIFAR100x10 WideResNet16
robust across tasks and capable of correctly classifying dis- tinct subsets of data, even if their overall accuracy is lower. As a result, using multiple ACs introduces both diversity and redundancy in predictions if one classifier forgets, an- other may still succeed. This redundancy, combined with confidence-bass early exits, makes ACs a viable tool for continual learning and helps explain how they can improve not only efficiency but also performance.
50
Accunery
40
10
142
1231
20
[5]
Inferience
Figure 8. Dynamic inference results with WideResNet16.
Reproducibility. The code used to run experiments in this paper is publicly available at ht.ps: tht../...mtooo.omm/.. Dezatkowski/cll http//cclaxxxlaaaa www.liarr-clasiiiiee..
to each ResNet block in the model. Results for the main continual learning methods are presented in Figure& 8, with complementary results for other methods provided in Ap- pendix E.4.
Limitations. ACS add slight training and memory over- head, which varies by method and architecture. This study focused on computer vision and classification, leaving op- portunitics for exploration in other modalities and tasks.
Consistent with the previous experiments for other architec- tures, AC-enhanced variants outperform their corresponding baselines across all configurations. Interestingly, in line with the recent work (Mirzadeh et al., 2022a;b) which suggests that wider networks may be less prone to forgetting, the final accuracies with Wede are better than with ResNet32. These findings further underscore the general- ity of our method, demonstrating its effectiveness even in architectures less commonly used for continual learning.
Impact Statement
This work aims to advance the scientific understanding of machine learning, particularly in the field of continual learn- ing. Our primary contribution is to enhance the fundamental principles of Al research, fostering progress within the aca- demic and scientific community. While our work has the potential influence broader applications, we emphasize that its intended purpose is purely scientific. We do not take responsibility for any unintended consequences that may arise from its application beyond this scope, especially in an era of rapid AI and deep learning advancements.
6. Conclusions
In this work, we show how intermediate representations in neural networks can be leveraged to improve both perfor- mance and efficiency in continual learning through auxiliary classifiers (ACs). Our analysis reveals that representations in the early layers are more stable and less prone to for- getting compared to those in the final classifier. Based on these findings, we propose attaching lightweight ACs to intermediate layers to boost network performance while enabling faster inference through dynamic layer skipping. Our experiments across diverse continual learning methods show an average 10% relative performance improvement over standard single-classifier models on CIFAR 100 and ImageNet100, consistent with different architectures such as ResNets, VGGI9, and ViT. Additionally, through dynamic inference, AC-enhanced models reduce the computational cost of the inference by up 60% without sacrificing accu- racy. Overall, our approach provides a promising solution to address both performance and efficiency challenges in resource-constraine continual learning environments.
A Acknowledgements
This work is supported by National Centre of Sci- ence (NCP, Poland) Grants No. 2022/45/B/ST6/0281,, 2024/53///ST6/0008.. and 2023/55/D/ST6/02846 Yaoyue Zheng acknowledges the China Scholarship Council (CSC) No.202406280387. This work was supported by Horizon Europe Programme under GA no. 101120237, project "ELIAS: European Lighthouse of AI for Sustainability". We gratefully acknowledge Polish high-performance com- puting infrastructure PL.rrdd (HPC Center: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2024/0113885. We acknowl- edge the Spanish project PID2022-14325778 by MCIN/AEI/1O.. 13039/501 10001 1033 and FEDER, and Funded by the European Union ELLIOT project. Bartlomiej Twardowski acknowledges the grant RYC2021-032766
financed
Discussion. Our findings demonstrate that adding ACs improves overall continual learning performance. The gains from ACS might seem counterintuitive at first, especially since early ACS are less accurate and also not immune to forgetting. However, different ACs learn to classify samples based on different features. In particular, early classifiers rely on shared, general features, which makes them more
9

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
References
Evei, U., Dumoulin, V., Larochelle, H., and Mozer, M. C. Head2toe: Utilizing intermediate representations for bet- ter transfer learning. In International Conference Machine Learning, pp. 6009 6033, PMLR, 2022.
Ahn, H., Kwak, J., Lim, S., Bang, H., Kim, H., and Moon, T. Ss-il: Separated softmax for incremental learning. In Proceedings of the IEEE/CVF International conference on computer vision, pp. 844-853, 2021.
Goswami, D., Liu, Y., Twardowski, B., and van de Weijer, J. Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. Advances in Neural Information Processing Systems, 36, 2024.
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, Т. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pp. 139-154, 2018.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn ing for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Boschini, M., Bonicelli, L., Porrello, A., Bellitto, G.. Pen- nisi, M., Palazzo, S., Spampinato, C., and Calderara, S. Transfer without forgetting. In European conference on computer vision, pp. 692-709. Springer, 2022.
Jung, D., Lee, D., Hong, S., Jang, H., Bac, H., and Yoon, S. New insights for the plassicy dilemma in online continual learning. arXiv preprint SKGV:2302.08744. 2023.
Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. Dark experience for general continual learn- ing: strong, simple baseline. Advances in neural infor- mation processing systems, 33:15920-15930, 2020.
Kaya, Y., Hong, S., and Dumitras, T. Shallow-deep net- works: Understanding and mitigating network overthink- ing. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Pro- ceedings of the 36th International Conference on Ma- chine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3301-3310. PMLR, 09-15 15 Jun 2019. hww.hhtts:///proceeo v97
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a-gem. Interna- tional Conference on Learning Representations, 2018.
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H., and Ranzato, M. On tiny episodic memories in continual learning. arXiv preprint a90iv 10486, 2019.
ngs ml press/
Kim, S., Noci, L., Orvieto, A., and Hofmann, T. Achieving a better sability-pllsticiny trade-off via auxiliary networks in continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1130-11939, 2023.
Davari, M., Asadi, N., Mudur, S., Aljundi, R., and Belilovsky, E. Probing representation forgetting super- vised and unsupervised continual learning. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 16712-16721, 2022.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., A., al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-355 2017.
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. A continual learning survey: Defying forgetting in classifi- cation tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-333 3385, 2021.
Korbblith, S., Norouzi, M., Lee, H., and Hinton, G. E. Similarity of neural network representations revisited. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3519 -5299. PMLR, 2019, URL http: http:://pocceeddeee mlr press/ v v 337/kornblithl 9a. html.
Deng, J., Dong, W., Socher, R., Li, I.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.1 / 11929, 2020.
Krizhevsky, A. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Douillard, A., Cord, M., Ollion, C., Robert, T., and Valle, E. Podnet: Pooled outputs distillation for small-tasks incre- mental learning. European Conference on Computer Vision, pp. 86-102, 2020.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-299 2947, 2017.
10

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Liang, Y.S. and Li, W.-J Loss decoupling for task-agnostic continual learning. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Pan, X., Jin, X., He, Y., Song, S., Huang, G., al. Bud- geted training for vision transformer. In The Eleventh International Conference on Learning Representations, 2022.
Liu, X., Wu, C., Menta, M.. Herranz, L.,, Raducanu, B., Bagdanov, A. D., Jui, S., and de Weijer, J. v. Generative feature replay for class-incremental learning. In Proceed- ings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition Workshops, pp. 226-227, 2020a.
Panda, P., Sengupta, A., and Roy, K. Conditional deep learn- ing for energy-efficce and enhanced pattern recognition. In 2016 Design, Automation & Test in Europe Conference & Exhibition ( (DATE), pp. 475-480, IEEE, 2016.
Liu, Y., Parisot, S., Slabaugh, G., Jia, X.,, Leonardis, A., and Tuytelaars, T. More classifiers, less forgetting: A generic culti-classifier paradigm for incremental learning. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16, pp. 699-716. Springer, 2020b.
Parisi, G. I., Kemker, R., Part, J. I.., Kanan, C., and Wermter, S. Continual lifelong learning with neural networks: A review. Neural Networks, 1 11::54-1,, 2019.
Pawlak, S., Szatkowski, F., Bortkiewiez, M., Dubinski, J., and Trzciński, T. Progressive latent replay for efficient generative rehearsal. In International Conference on Neural Information Processing, pp. 457-467. Springer, 2022.
Magistri, S., Trinci, T., Soutif, A., van de Weijer, J., and Bagdanov, A. D. Elastic feature consolidation for cold start exemplar-free incremental learning. In The Twelfth International Conference on Learning Representations, 2024.
Prabhu, A.. Torr, P. H., and Dokania, P. K. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision-ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 524-540. Springer, 2020.
Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Ramasesh, V. V., Dyer, E., and Raghu, M. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Rep- resentations, 2020.
Mallya, A., Davis, D., and Lazebnik, S. Piggyback: Adapt- ing a single network to multiple tasks by learning mask weights. In Proceedings of the European Conference Computer Vision (ECCV), pp. 67-82, 2018.
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Masana, M., Liu, X., Twardowski, B.. Menta, M., Bagdanov, A. D., and Van De Weijer, J. Class-increment inssemnacteett learning: survey and performance evaluation on image classifica- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-5533, 2022.
Riemer, M., Cases, I., Ajemian, R., Liu, M., Rish, I., Tu, Y., and Tesauro, G. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2018.
Masarczyk, W., Ostaszewski, M., Imani, E.,, Pascanu, R., Milos, P., and Trzcinski, T. The tunnel effect: Build- ing data representations in deep neural networks. arXiv preprint 2rXiv:2305.197 2023.
Rusu, A. A., Rabinowitz, N. C.,, Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks. arXiv preprint NtG.:v660004677 2016.
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 109-165. Elsevier, 1989.
Serra, J., Suris, D., Miron, M., and Karatzoglou, A. Over- coming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp. 4548-4557. PMLR, 2018.
Mirzadeh, S. I., Chaudhry, A., Yin, D., Hu, H., Pascanu, R., Gorur, D., and Farajtabar, M. Wide neural networks forget less catastrophically. In International conference on machine learning, pp. 15699-15717. PMLR, 2022a.
Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. Advances in neural informa- tion processing systems, 30, 2017.
Mirzadeh, S. I., Chaudhry, A., Yin, D., Nguyen, T., Pascanu, R., Gorur, D., and Farajtabar, M. Architecture matters in continual learning. arXiv preprint 27X10:2202.00275, 2022b.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2014.

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Tang, S., Chen, D., Zhu, J., Yu, S., and Ouyang, W. Layer- wise optimization by gradient decomposition for contin- ual learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 9634- 9643, 2021.
Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In International conference machine learning, pp. 3987-3995. PMLR, 2017.
Zhao, H., Zhou, T., Long, G., Jiang, J., and Zhang, C. Does continual learning equally forget all parameters? arXiv preprint arXiv:2304.04/555 2023.
Teerapittaynnon, S., McDanel, B., and Kung, H. Branchynet: Fast inference via early exiting from deep neural net- works. 2016 23rd International Conference on Pat- tern Recognition (ICPR). pp. 2464-2469, 2016. doi:
Van de Ven, G. M. and Tolias, A. S. Three scenarios for continual learning. arXiv preprint arXiv: 1904.073444 2019.
Wang, S., Li, X., Sun, J., and Xu, Z. Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition, pp. 184-193, 2021.
Wang, Z., Liu, L., Duan, Y., Kong, Y., and Tao, D. Continual learning with lifelong vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 171 171-181, 2022.
Wójcik, B., Przewie2likowski, M., Szatkowski, F., Wolczyk, M., Balazy, K., Krzepkowski, B., Podolak, I., Tabor, J., Smieja, M., and Trzciński, T. Zero time waste in pre- trained early exit neural networks. Neural Networks, 168: 580-601, 2023.
Wortsman, M., Ramanujan, V., Liu, R., Kembhavi, A., Rastegari, M., Yosinski, J., and Farhadi, A. Supermasks in superposition. Advances in Neural Information Pro- cessing Systems, 33:15173-15184 2020.
Wu, C., Herranz, L., Liu, X., Van De Weijer, J., Raducanu, B., et al. Memory replay gans: Learning to generate new categories without forgetting. Advances in Neural Information Processing Systems, 31, 2018.
Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., and Fu, Y. Large scale incremental learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 374-382. 2019.
Yan, H., Wang, L.., Ma, K., and Zhong, Y. Orchestrate latent expertise: Advancing online continual learning with multi-level supervision and reverse self-distillation. Computer Vision and Pattern Recognition, 2024. doi:
00.1109/CYPP5273..224..027
Zagoruyko, S. and Komodakis, N. Wide residual networks. British Machine Vision Conference, 2016. doi: 10.5244/ C.30.87.
12

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Appendix
A. AC architecture and training details
In our main experiments, we follow the insights from (Kaya et al., 2019) and (Wójcik et al., 2023) in our design of multi-classifier networks. We place the ACS after layers that perform roughly 15,, 30%, 45% 60%, 75%, 90% of the computations of the full network (L1.B3, LI.B5, L2.B2, L2.B4, L3.BI, L3.B3, L3.B5 for ResNet32, and blocks B2-B7 for ResNet18). For convolutional networks, ACs are composed of pooling layers to reduce the input size for the fully connected networks that produce the predictions. For experiments on ViT, we apply fully connected classifier on top of the LayerNorm layer on the first token. All the classifiers in our model are composed of heads for each task, and we add new head upon encountering a new task.
Our main objective used for training the network on any given task is a weighted sum of losses for each classifier. For continual learning methods, we use the additional losses alongside the cross-entropy and weigh the total loss. We train the model for each task jointly with all the ACs, updating all the parameters of the network. We follow the weight scheduler from (Kaya et al., 2019) and progressively increase loss weights for different ACs over the training phase to the values matching their computational cost (e.g. the weight for the first classifier for ResNet32 would increase up O. 15, for the second classifier to 0.30, and on). For ResNet18 we use 6 ACs and weights [0.3, 0.4, 0.55, 0.65, 0.8, 0.9), the network contains only 8 blocks whose computational cost distributes approximately like that. For experiments with 12 ACs, we attach classifiers to all blocks L1.B3-L3. B4 and interpolate the weights from the standard setting. For 3 ACs, we use blocks L1.B3, L.2.2,, and L3.B1 with weights [0.15, 0.45, 0.75]. When training ViT or VGG networks, for each model block we use multiplies of given base weight (e.g. 0.08 for ViT-base, 0.05 or 0.09 for 18 and 10 AC setup for VGG19), For example, we set the AC weights for 11 ACs in VIT as [0.08, 0.16, 0.80, 0.88). Different loss weights for each AC serve to stabilize the training and mitigate overfitting the earlier layers, which may have lower learning capacity.
We train the ResNet32 models on CIFAR100 for 200 epochs each task, using SGD optimizer with batch size of 128 with a learning rate initialized to 0.1 and decayed by rate of 0. at the 60th, 120th, and 160th epochs. For training ResNet18 on ImageNet100 100, we change the scheduler to cosine with linear warmup and train for 100 epochs with 5 epochs of warmup, as we find to converge to similar results in a shorter time. For ViT, we use Adam W and train each task for 100 epochs with learning rate of 0.01 and batch size of 64. We also use cosine scheduler with a linear warmup for 5 epochs. We use a fixed memory of 2000 exemplars selected with herding (Rebuffi al., 2017). For ER each batch is balanced between old and new data, and for SSIL we use 4:1 ratio of new old data. Otherwise, for other exemplar-based methods, we follow the standard FACIL procedure for exemplars and just add them to the training data without any balancing.
13

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
B. Training time and memory overhead disucssion
Introducing Auxiliary Classifiers (ACs) into the network introduces additional computational and memory overhead during training, due to the extra gradient computations and memory required by the classifiers. In this section, we quantify this overhead by evaluating the impact of ACs on training ResNet-32 on CIFAR-100 with various CL methods. We train the model with different numbers of ACs, where "O ACS" corresponds to the baseline model without any auxiliary classifiers. For each configuration, we report average training time (in hours) and peak GPU memory usage (in GB) across three random seeds. Training time is measured separately for 5-task and 10-task incremental setups, while peak memory usage is independent of the number of tasks. We present the results in Tables 3 and 4.
In the standard setup used in the main paper (6 ACs), we observe roughly 50% increase in training time a 10% increase peak memory usage compared the baseline. However, is important note that these results represent the worst-case overhead across all experiments, since ResNet-32 is the smallest model we evaluate, and the relative impact of the added ACs parameter count and computational load is the most pronounced for this model. As shown in Figures 1 and 7, the total inference cost with ACs exceeds the baseline by approximately 3% for ResNet-32, whereas for larger models like ViT, this overhead reduces to around 1%. While these overheads are non-negligible, they are not prohibitive in practical settings. Class-incremental learning is typically performed in an offline setting without real-time constraints, making the additional training costs acceptable in exchange for the performance gains achieved by ACs. Moreover, we did not specifically optimize for training efficiency, and these overheads could be further reduced if necessary in the downstream applications.
Table 3. Training time (in hours) for CIFAR-100 under 5-task and 10-task setups, reported (5-task / 10-task).
ACs
FT
FT+Ex
GDumb
ANCL.
BiC
ER
EWC
LwF
LODE
SSIL
Avg
0
1.4
1.7
0.6/ 1.2
2.1/288
1,4/200
2.5/
1.2/14
1,2/14
3.0/3.3
1.7 1.7/188
1.6/ 1.9
3
1.6
1.5
0.8 1.5
2.7/3.6
1.8/
2.8/3.2
1.9
1.5/19
3.6/4.2
2.1/2.3
2.0/2.5
6
1.6/19
1.7/2.4
1./17
3.3/4.4
2.1/3.0
3.3/3.8
1,8/22
1.8/24
4.3/5.2
2.5/2.9
2,4/3.0
12
2.4
2.3
1,5/233
4.5
2.9/4.2
4.0/4.8
2.5
2.5/3.5
5.5/6.9
3.2/3.9
3.1/4.1
Table 4. Peak GPU memory usage (in GB) across methods and AC configurations.
ACS
FT
GDumb FT+Ex
ANCL.
BiC
ER
EWC
LwF
LODE
SSII.
Avg
0
2.10
2.10
2.39
2.10
2.09
2.10
2.10
2.10
2.10
2.18
2.14
3
2.25
2.18
2.56
2.29
2.22
2.18
2.56
2.22
2.22
2.29
2.30
6
2.41
2.27
2.73
2.47
2.32
2.27
2.73
2.30
2.31
2.41
2.42
12
2.72
2.43
3.03
2.82
2.50
2.43
3.02
2.50
2.49
2.65
2.66
14

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
C. ACs in more CL settings
In this section, we prove the robustness of our idea on additional benchmarks in warm-start continual learning and two setups with more tasks (20 and 50) on CIFARI00.
C.1. Warm-stard continual learning
Table Adding auxiliary classifiers (ACs) is beneficial to the final network accuracy when training CIFARIO0 warm-start scenario with 50 classes in the first task.
Method
FT
FT+Ex
GDumb
ANCL
BiC
DER+
ER
EWC
LwF
LODE
SSIL
Avg
CIFARIONA
Base
16.18
40.38
17.381am
42.85
16.60
38.49
38,11
17.08
42.72
42.281e46
46.78
35135_em
+AC
22.37
38.12g0m
22.60
43.
48.75
43.55
37.81 gass
25.49_4m
43.45
44,95
48,97
38.18
+6. 12
-2.26
+5.22
+1.12
+2.15
+5.06
-0.30
+8.40
+0.72
+2.67
+2.19
+2.83
CIFAR100x
Base
7,90eams
16.4
16.55
+3.86-an
42.38
35.24
34.86
8.01 moas
32.13 e
38.17 18.17
41.46-1a4
29.73
+AC
11.91
36.80
22.73g4x4
34.94:4s
45.37
40.46
36.80
16.05
35.31
40,97
45.70
33.37
+4.00s:44
+0.39
+6.1750m
+1.08
+2.99
+5.22=1.
+1.94s109
+8.0 -8.04
+3.18
+2.80
+4.24=
-3364
CIFAR100X6 ResNet32
CIFAR100X11 ResNet32
50
Accuracy
30
ANCL
BIC
DER
LODE
SSIL
20
+AC
+AC
+AC
+AC
+AC
20
30
40
50
60
70
80
90
100
20
30
40
50
60
70
80
90
100
Inference cost [%6]
Inference cost [%]
CIFAR100X6
ResNet32
CIFAR100x11
ResNet32
40
Accuracy
30
20
20
CR
EWC
FT+Ex
GDumb
LwF
+AC
+C
+AC
+AC
+AC
30
40
80
90
100
30
40
50
80
90
Inference cost [%]
Inference cost [%]]
Figure 9. Dynamic inference results for CL methods enhanced with ACs in warm start scenario CIFAR100.
common scenario in continual learning warm-start (Magistri et al., 2024: Goswami al., 2024), which simulates starting from a pre-trained model. This scenario an interesting study for continual learning due the practical benefits of pre-trained models and the differing learning dynamics. To evaluate our model in warm-start scenario, train CIFAR 100, using 50 classes for the first task to simulate a pre-training phase. The remaining classes are split evenly into 5 or 10 tasks, referred to as the 6-task and 1t-kk splits. We conduct experiments with a new task split and results are shown in Table 5. We observe a general improvement in the warm-start setting, except for ER and FT+Ex, which show different performances on the 6-task and 11-aask splits. In Figure 9, we also show the effectiveness dynamic inference under the warm-start scenario.
15

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Table 6. Additional results for AC-enhanced methods with longer sequences of tasks on CTFARIO0.
Method
FT
FT+Ex
GDumb
ANCL
BiC
DER+
ER
EWC
LwF
LODE
SSIL
Avg
CIFARI00x20
Base
4.72gams
23.682104
19.34
38.8
34.5
30,94
5.5
18.97
37.90
36.86212
+AC
7_33
37.
30.11:40
20.87
42.03 2uLs
36.45
36.10
9.70
19.63
41.85
39.78
29.18; n.
+2.61
+4.8
46.43eus
+1.53
+3.22
+1.95-u8
+5.16
44
+0.66
+3,95
+2.92+108
+3.40en2
CON5RI00N50
Base
2.97
21 1o.60
13.48
5.88 1821
24.01
16.79_1.
18.35
1.63
1.09
23.56
22.66.u0
14.00gam
+AC
1.64au
26.39
17.63
5,54
29,39
22.59-6u9
22.96
2.12
5,40=am
28.53=1.2
25.77
=7.09
+0.67
+4.1 10
0.34
25.38go
=5
44.61
+0.491am
20.31
+4.97
+3.11
g3.08gmmm
CIFAR 100x20 ResNet32
CIFAR100x50
ResNet32
Accuracy
30
20
10
ANCL
BIC
DER+
LODE
SSIL
+AC
+AC
+AC
+AC
AAC
20
30
40
50
60
70
80
90
100
20
80
100
Inference cost [%]]
Inference cost [%]
CIFAR100X20
ResNet32
CIFAR100x50 ResNet32
30
20
Accuracy
20
10
ER
EWC
FT+Ex
GDumb
LwF
+AC
+AC
+AC
+AC
+AC
+AC
20
30
40
50
60
70
80
90
100
20
40
60
80
100
Inference cost [%]
Inference cost [%]]
Figure 10. Dynamic inference results for CL methods enhanced with ACS on CIFAR100 split into long, 20 and 50 tasks sequences.
C.2. AC-enhanced methods on longer task sequences
In Table 6 present results for experiments on 20 and 50 equally split tasks on CIFAR100, following the setup from Section 5.1. For the 50-task split, we use growing memory of 20 exemplars instead of a constant memory of 2000 due to the early tasks containing less samples than the memory limit. Non-replay-based methods perform poorly on longer sequences of tasks, especially on 50 tasks, but ACs robustly enhance the method performance in all tested scenarios. In Figure 10, we also demonstrate the performance of AC-enhanced methods with dynamic inference.
16

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
D. A Ablation studies on AC design
In this section, we perform ablation studies on AC design in the setup from Section 5.1 for CIFARIOO. In Appendices D.I and D.2 we explore the impact of the number of the ACs and alternative AC architectures from Wójeik et al. (2023). In Appendix D.3 we compare models trained end-to-end and obtained through linear probing, and in Appendix D.4 we show the performance of networks with only a single classifier to provide the intuition behind the classifier behaviour.
D.I. Number of ACs
We vary the number of classifiers for the AC setup from the main paper: we either drop half of them or insert an extra one between the existing classifiers. We measure the improvement obtained upon the baseline and the additional computational cost incurred by 3, 6, and 12 ACs and show the results in Table 7 and Figure 11. While the optimal setup varies across different continual learning methods, the addition of ACs is universally beneficial to performance.
Table Difference w.r.t. bascline singlo-classifier methods when using a different number of auxiliary classifiers (ACs). ACs robustly improve the final accuracy of continual learning methods, regardless of the number of classifiers used.
FLOPS
FT
FT+Ex
GDumb
ANCL
BiC
ER
EWC
LwF
LODE
SSIL
Avg
NDAC
69.90M (1x)
CIFAR100<5
BAC
70.72M (1.0xx)
+7.61
a0
+5.08
+2.14
+2.62
+4.63
+8.94
+0.95
+4.19 +4.19a0a
+2.65 jn.
+3.95 1039
6AC
71.55M (1.02x)
+9.49.a
+0.39
+4.20
+2.12
+2.74_0s3
+5.22
+10.02
1m
+2.29.tts
+6.31 geat
+4.55 +2.72
12AC
72.97M (1.04x)
+9,09
+1.67-12s
+5.15
+2.45=am4
+3.57=447
+4.46=u44
+2.75-12s +9.97=633
+5.43
+2.92
+4.74
CIFARI00x10
3AC
70.84M
(1.01x)
+6,48sus
+3.05=a5
+5.74
+1.71
+2.85=21
+4.44=ue
+7.92=txi
+0.53
+6.13 13
+2.02=1.
+4.09
6AC
71.77M (1.03x)
+6.62.1m
+2.46-am
+3.52
+0.68
+3.31-2a
+5.01
+8.92
+0.74
+6.80 eess
+1.88
+4.20
12AC
73.36M (1.05x)
+6.12 +2.59
+1.85 140
+3.98 2s
+4.95
+6.57
+1.14
+6.68 1070
+1.98 1m
+4.05 1000
CIFAR100x5| ResNe132
CIFAR100x10 ResNeI32
CIFAR100s5
ReeNet32
10010100x10 ResNet32
40
50
Apquracy
30
ANO.
+3AC
Acturayy
40
40
B.C
134C
*5AG
30
30
+5AC
+13AC
20
40
80
50
100
20
40
80
80
100
20
40
80
60
160
20
40
60
80
100
Inferonce cool [N]
Infaronce cost [N]
Inforenca cost [%]
Inforence cost [%)
CIFAR10045 ResNe132
CIFAR100x10 ResNel32
CIFAR100&5 ResNet32
CIFAR100x10 ResNet32
Aocuray
Accuracy
30
30
10
20
20
+13MC
Inference
Inference
Inferenca
Inference
CIFAR100K5| ResNet32
CIFAR100x10 ResNet32
CIFAR100x5 ResNet32
10010100x10 ResNe132
10
30
Aasuncy
ER
Accuaay
EWC
+3AC
1040
15AC
10AC
106G
+1240
20
40
60
100
20
40
80
50
100
20
40
60
10
100
20
40
60
80
100
Inference cool [N]
Inferwnoe cosl [%]
Infererca cosl
[%]
Inference cosl [%]
CIFAR100x5 ResNet32
CIFAR100x10 | ResNe132
CIFAR100k5 ResNet32
CIFAR 100x10 ResNet32
50
Accuraty
22.5
25
Gouns
Accuraty
40
40
20.0
30
17.5
Inference cuet
Inference (5%
Infererce cosl
Inferense cosl
CIFAR100X5 ResNet32
CIFAR100x10/ ReaNet32
CIFAR100K5 ReaNet32
CIFAR ResNet32
Accuvarc
40
30
26
+3AC
1340
80
30
11260
1124C
20
40
60
80
100
20
40
60
60
100
40
60
80
100
40
60
80
100
Inference cosl [o5]
Inference cosl
Inforence cost [%]
Inforence cost [%]
Figure 11. Dynamic inference experiments for ResNet32 on CIFAR100 110 enhanced with different number of ACs.
17

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
D.2. AC architecture
In our main work, we investigate a simple setup with independent classifiers. Early-exit works such as (Wójeik et al., 2023) propose more complex dynamic architectures, where subsequent classifiers connected and their predictions are combined through weighted ensemble. Those architectures induce only slight parameter and computation overhead, but in standard supervised learning setting can improve the performance of intermediate classifiers through sharing the knowledge between them. We investigate those architectures in continual learning on the set of methods analyzed in previous sections on split CIFARIO0 benchmarks and present the results in Table 8, alongside dynamic inference plots in Figure 12. Similar to the AC density ablation, we do not observe a clear improvement from changing the setup. We hypothesize that connecting the classifiers makes them no longer independent, which negates the benefits yielded in continual learning by the classifier diversity.
Table 8. Difference w.r.t. baseline single-classiffer methods when using different auxiliary classifier architecture: cascading (C) and ensebling (E) from Wójeik et al. (2023). Similar to Table 7, ACs universally improve the accuracy with small differences in performance between the architectures.
Method
FT
FT+Ex
GDumb
ANCL
BiC
ER
EWC
LwF
LODE
SSIL
Avg
CIFAR100<5
AC
+5.70+521
+0.24
+2.52+230
+1.27-137
+1.65=1s1
+3.13+1m
+6.01+55)
+1.37+127
+3.79+1.0
+1.63=1.8
+2.73+2s1
AC+C
=4.4
-0.72gmm
+2.49
+1.20
+1.16
+2.57
+6. 16am
+0.662am
+1.2621a3 +2.89
+2.31
AC+E
+6.47
+0.38
+2.00
+1.28
+1.30
+2.32
+6.58
+1.14
+2.79
+1.29
+2.56=2n
CIFAR 100x10
AC
+6.62+106
+2.462031
+5.52=1.1
+0
+3.31=26
+5.01=a95
+8.92+105
+0.74
+6.80
+1.88=0.7
+4.2020.7
AC+C
+6.981um
+2.63 Love
45.56
+1.60_am
+3.63
+5.
+8.35 1ock
+1.27
+5.4510m
+2.53 1um
+4.32 in
AC+E
+7.35+014
+3.27=074
+5.09=0.0
+2.28-021
+3.68=216
+4.77+052
+8.62+1.12
+1.18+030
+5.85 =0.00
+1.53=0077
+4.36=033
CIFAR100X5 ResNet32
C/FAR100x10 ResNeB3
CIFAR10055 ResNat32
100x01 ResNet32
40
Avarany
Acauraty
40
30
MD-C
30
MC-C
MDHe
AACE
60
60
100
100
Inference cosl
Inference cos!
Infererce cosl
Inferercce cost [S)
CIFAR10045| ResNel32
CIFAR100x10 | ResNel32
CIFAR100x5| ResNet32
10011100x10 ResNe132
Avtrrsy
Acturacy
30
Fi+ex
MC
AC
MD-G
+AC-C
MD+E
20
+ACHE
40
60
80
100
20
40
80
80
100
20
40
80
90
100
20
40
60
50
100
Inforence cost 1%)
Inference cost (%)
Inforence cost [%]
Inforence cost [%]
CIFAR100x5| ResNe132
C/FAR100x10 ResNel32
CIFAR100x5 ResNet32
1FF01000100 ResNel32
40
Avcrrsy
Asainay
Rs
EWC
15
30
20
MDHCC
10
60
Inferonce
Inferonce
Interance [S]
Inferance
CIFAR18015 ResNet32
1IFAR100x10 ResNet32
CIFAR100:5 ReaNet32
CIFAR100X10 ResNet32
Accurasy
25
C2une
40
Loce
20.0
AAC
17.5
20
+5044
30
30
ADCH
ACCCC
20
40
60
80
100
20
40
80
80
100
40
50
80
100
20
40
60
50
100
Inference cost [%)
Inference coel (5)
Inforence cosl
[%]
Inferenize cosl [95]
CIFAR100K5 ResNet32
CIFAR100x10 ReaNet32
CIFAR100:5 ResNet32
CIFAR 100x10 | ResNet32
40
Accurtty
Acturan aty
40
40
55L
4AG
20
30
30
60
60
100
100
100
Inference cosl [S]
Inference cosl [N]
Inforence cost
Inference cost
Figure 12. Dynamic inference experiments for ResNet32 on CIFARI00 enhanced with different types of classifiers
18

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
D.3. Comparison between end-to-end training and linear probing
In Section 3.4, we advocate for training the network and ACs jointly with enabled gradient propagation, as it leads to better performance of individual classifiers. In this section, we investigate the final performance of linear probing classifiers in comparison with jointly trained ACs CIFAR100. For ACS use the same setup in Section 5.1, and in the case of linear probing the only difference is that the classifiers are trained without gradient propagation. We show the final performance of both settings in Table 9, and also demonstrate their cost-accuracy characteristics in Figure 13, Aside from distillation-basse exemplar-free methods, ACs outperform probing accuracy but probing also outperforms baselines most of the time. Dynamic accuracy curves also highlight that end-to-end training generally allows AC methods to achieve greater accuracy at lower computational costs due to the ability to learn better early classifiers.
Table 9. Comparison between final results when using intermediate classifiers trained together with the network (AC) or trained with linear probing (LP). Training classifiers together generally yields better performance, with the only noticeable exception being exemplar free distillation-bass methods (ANCL and LwF), which could be caused by significant variance in per-task accuracy of intermediate classifiers.
Method
FT
FT+Ex
GDumb
ANCL
BIC
ER
EWC
LwF
LODE
SSIL
Avg
CIFARI00x5
Base
18.68
38.35
19.09=0.4
37.71=114
47.66-0.41
34.55
18,95
38. 38.2
42.82+as4
45.62+a.s
34.17+a27
+LP
26.82 36.83
22.56:003
43.60
49.6 .9.62..mm
38.47
28.13
41.13
46.35
47.33
38.09
+AC
28.18
38.75
23.29
39.83=112
50.40
39.77=431
28.96a1.
40.5
49.13
28.35
38.72
CIFAR 100x
Base
10,27
20.5
22.22:2072
30.69
42.87=151
32.31
10.20
29, 29,56.204
38.87+045
42.299
29.38=n26
+LP
17.77.1.
35.62.mm
25.60
33.72.1..
44.74_2.
35.78.0.
18.84
31.88 1mm1
43.37
43.33 goos
33.06
+AC
16.88118
36.97 1,.99
27.74
31.37
46.19
37.32.
19.12
30.31
45.671952
44.17
33.57..m2
10Dx5 ResNat32
CIFAR100X10 ResNat32
CIFAR100X55 ResNel32
110XR100X10 ResNat32
Accurany
30
Accuraty
40
40
16
30
20
20
100
100
40
80
40
80
Inference cost
Inferencee cost
Inference cost [%)
Inference cost
(%)
CIFAR10065| ResNet32
CIFAR100x10 ResNet32
CIFAR100K5 ResNet32
CIFAR100X10 | ResNel32
Accursay
15
Asourey
30
8
20
20
MC
10
UP
30
100
20
40
60
100
20
40
60
80
100
20
40
60
50
100
Inforence cosl [%]
Inforence cost [%]
Inforence post [%]
Inforence sost (%)
13
CIFAR100x5 ResNet32
CIFAR100x10 10| ResNet32
CIFAR100x5 ResNet32
CIFAR100x 10 ResNet32
Accersay
Assurens
15
20
10
100
20
Inferance cost
Inferance cost
Inference [1)
Inference cost
100AR ResNat32
CIFAR100X10 ResNet32
CIFAR100x5 ResNet32
CIFAR 100x10 ResNet32
22.5
25
Accurany
40
40
20.0
Bburb
-40
-AC
17.5
20
40
00
100
40
50
80
100
20
40
50
80
100
20
40
60
50
100
Inforance cost (%)
Inforance cost [%)
Infaronce cost [%]
Infercnce cost (N]
CIFAR10Ds5 ReaNet32
CIFAR100X10 ResNat32
CIFAR 00xx5 ResNet32
C1FAR100X10 ResNet32
Accurasy
Acourery
36
%
30
20
100
100
100
Inference cost
Inferemce cost
Inference coot [8]
Inference cool (8)
Figure 13. Dynamic inference plots for several continual learning methods extended with auxiliary classifiers when using auxiliary classifiers with enabled gradient propagation (AC) or without (I.P).
19

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
D.4. Leave-one-AC ablation
In this section, we perform leave-one-out ablation of the setting explored in Section 5.1 for CIFARIO0 methods explored in Section 3. Namely, train the model with only one auxiliary classifier out of the original six, with the classifier weight equal to 1. during training and present dynamic inference results in Figure 14. For non-naive methods, either the 5th or the 6th AC achieves the best performance. Interestingly for finetuning, later ACs yield lower performance, which is consistent with our observations on more native stability in early layers. All tested AC setups achieve comparable performance at the full computational budget, but compared to our based setup of using all ACs they tend to underperform at lower compute budget. Slightly better performance of single AC setup for FT+Ex and LwF hints that AC placement in our work could be further optimized. However, overall similar performance across all the tested scenarios prove the robustness of our idea.
observe that
CIF5R100X5
CIFAR100G FT+EX
DIFAR100x3 LwF
CIFAR100x5
BIC
30
40
50
Accuracy
25
30
35
40
30
20
20
25
30
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
CIFAR100X10 FT
CIFAR100x10 FT+EX
CIFAR100X10 LwF
CIFAR100x10 BIC
18
Accuracy
16
30
40
14
AllACs
AC
5th
12
20
20
AC
8th AC
3rd AC
10
0.25
0.50
0.75
1.00
0.25
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
Cost
Cost
Cost
Cost
Figure 14. Leave one out AC ablation for FT, FT+Ex, LwF and BiC.
20

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
E. Additional dynamic inference results
This section presents dynamic inference plots for experiments performed Section or complementary to those experiments.
E.1. Standard benchmarks
In Figure 6, we present dynamic inference results for and 10 task splits of CIFAR100 and ImageNer100 with the methods omitted in the main section due to space constraints. As the previous sections, ACs demonstrate robustness and allow for significant reduction of average computation.
CIFAR100x10 | ResNet32
ImageNet1 00x10 | ResNet18
00x110
40
Accuracy
50
30
40
ANCL
BIC
DER+
LODE
SSIL
20
+AC
+AC
+AC
+AC
30
20
40
60
100
40
60
80
100
Inference cost [%]
Inference cost
[%]
CIFAR1 00x5
ResNet32
ImageNet100x5 ResNet18
40
60
50
Accuracy
30
40
30
20
ER
...
EWC
FT+Ex ...
GDumb
LwF
+AC
AC
20
20
40
60
80
100
40
60
80
100
Inference cost [%]
Inference cost
[%]
CIFAR100X10 O | ResNet32
ImageNet100x10 ResNet18
50
Accuracy
30
40
30
20
20
EWC
FT+Ex
GDumb
10
20
40
60
80
100
40
60
80
100
Inference cost [%]
Inference cost [%]
Figure Dynamic inference plots as in Figure additional continual learning methods extended with auxiliary classifiers on CIFAR10 100 and ImageNet1 00 split into 5 and 10 tasks.
21

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
E.2. VGG19
In Figure 7 this section, we present dynamic inference plots for VGGI9 on CIFARI00 with all the methods considered in the main paper that correspond to the results from Table 2. AC-enhanced methods outperform the baselines by even bigger margins than ResNets used in main analysis and match their accuracy at smaller fraction of the compute.
CIDDR100x5 VVG11
C/FAR100x10 VGG19
100
VVG11
CIFAR100X10 VGG18
60
Azurwey
40
Accurnty
50
ANCL 1946
40
BCH0SC
ANCL 1844
20
40
60
100
20
40
60
80
100
20
40
60
50
100
20
40
53
50
100
Inference cosl [%]]
Inference cosl [%]]
Infererze coel [8]
Infereniee cosl [8]
CIFAR100x5 VVG19
CIFAR100X10 VGG19
CIFAR 100x5 VGG19
CIFAR100X10 VGG19
Arcueey
Ansuraty
100
+10.2
20
coel
Inference cost
cost
CIFAR100X5 VVG19
C/FAR100x10 VGG19
CIFAR100GC VVG11
CIFARM00X10 VGG19
Ancuraa
40
30
20
20
40
80
100
20
40
60
100
20
40
60
100
20
40
60
50
100
Inference cosl [4]
Inference cosl [N]
Inforence cost [N]
Inforence COST [N]
CIFAR100x5 VGG19
CIFAR10bx10 VGG19
CIFAR 100x5 VVG11
CIFAR100X10 VGG19
Ausanecy
Ascunsyy
30
50
11AC
20
Inference cosl [4]
Inferonce cos1 [N]
Inforenca cost
Inforenca COST
CIFAR100VE VGG19
10x10
Accurecy
Accunay
40
30
30
20
100
20
40
20
40
60
50
100
20
40
60
100
Inference cosl [5]
Infarencc cost [N]
Infenenca cosl [%)
Inferenca cost [%)
Figure 16. Dynamic inference experiments on CIFARI00 with VGG19 network enhanced with different number of ACs.
22

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
E.3. VIT
In Figure 17, we show dynamic inference results with all AC-enhanced methods for ViT trained on ImageNet100. Due computational constraints, contrary to the main experiments, only single seed for these experiments. While the overall performance of VIT is below the results achieve with ResNet18 models due to issues with training ViTs, already discussed in the paper, for all methods, AC versions outperform the baseline. Interestingly, for certain methods, performance peaks at the middle computational budgets and starts to degrade bit, indicating insufficient calibration of Transformers trained with those methods.
ImageNet1 00x5 ViT-base
Image ViT-base
40
Accuracy
30
20
FT+Ex
LODE
LwF
SSIL
10
+AC
+AC
+AC
+AC
10
20
40
60
80
100
20
40
60
80
100
Inference cost [%]
Inference cost [%]
ImageNet1 00x5 ViT-base
Image base
40
30
Accuracy
30
20
ANCL
BIC
ER
GDumb
10
10
+AC
+AC
+AC
+AC
Inference cost [%]]
Inference cost [%]
ImageNet1 00x5 ViT-base
ImageNet100x10 ViT-base
25
17.5
Accuracy
20
15.0
12.5
10.0
DER+
EWC
+AC
+AC
+AC
20
40
60
80
100
20
40
60
80
100
Inference cost [%)]
Inference cost [%]
Figure 17. Dynamic inference plots for ViT-base on ImageNet 100.
23

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
E.4. WideResNet-2
In this section, we provide dynamic inference plots for WideResNet-2 for continual learning methods missing from Figure in the main paper.
CIFAR100X5 WideResNet16
CIFAR100X10 WideResNet16
40
Accuracy
30
30
20
20
ER
EWC
-
FT+Ex
a. GDumb
LwF
10
AC
-..
AAC
AC
AC
..
AA
AC
20
40
60
80
100
20
40
60
80
100
Inference cost [%]
Inference cost [%]
Figure 18. Dynamic inference plots for WideResNet-2 on CIFAR100.
24

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
F. Additiona analysis
In this section, we present additional analysis multi-classifier networks. Furthermore, we provide results complementary the experiments from Section for task split of CIFARI00, as well as the results of those experiments for the models with enabled gradient propagation.
F.I. Forgetting AC-enhanced models
To better understand the impact of the addition of ACs, we further analyse how they affect the network forgetting. We define average forgetting across training on total tasks as T' Ei= on the i-th task after training j-th task. The final task is excluded from the average, it has zero forgetting under this definition. In Table 10, we report average forgetting values at the end of training for the main settings we consider in our experimental section (ResNet32 on CIFAR 100 and 8 on ImageNet100). The addition of ACs generally leads to reduced forgetting. While there are isolated cases where AC-enhanced models show slightly higher forgetting, these are exceptions, and even those cases the final accuracy still higher with ACs, as evidenced in Table 1. These results indicate that ACs enable more effective knowledge accumulation during continual learning.
Acct)) where Acc; refers the accuracy
Table 10. Forgetting values (difference between average accuracy the end of the first task and the end of the training) corresponding the experiments in Table 1. AC-enhanced models generally exhibit lower forgetting, and even in cases where the forgetting is higher, the
ACs still lead better final accuracy. This indicates that addition the ACs improves the network's ability to accumulate knowledge.
Method
FT
PT
GDamm
ANCL
BCC
DER+
HR
BWC
L.F
LODE
SSIL
Avg
CIFARIOOSS
Bose +AC
67
44.0010.79
12.0110.63
36.9120.79
9.33-2.89
74224117
17.2510.36
61.6521.13
21.2111.69
21.61
24.80=1.06
32/41:
1..
45.5111066
10.00.10.47
26.32:1115
9.26-4.00
3.68::.05
40.3110044
48.04:1.44
19.9510.57
20.61
19.95-1.12
15.90
CIFAR100<10
Base
70:35+2242
49.57++1.25
14.11+1008
40.49=1..55
9.31=2.43
11.88-4.27
51.77+111
08.40+274
24.26+22.12
1.549
21.44-2,74
34.55-21.51
+AC
26.40:22.97
4.003+1.05
1068:1444
30.60:1.466
6.37-3.81
8.39-5.10
46.62:1.33
54.08:2.300
29.12:+1.01
20.24:2.112
16.71-1.96
29.75-17.96
ImageNetL0005
74.3520.63
52.0010.775
16.7310.82
9.1
10.37=11.70
42.74.7.04
56.141:227
21.90=0.88
37,58:223.0 08.0
+AC
61.45:130
49.9711000
15.9811007
13.70.0.99
9.76
14.45
49.651125
61.77.0611
17.4910.76
31.8312.64
0.80
19.43=19.40
ImageNet100x10
57.50+11440
12.73-3.38
28.54455.66
19.75=1.69
39.65=222.73
+AC
68.44:1.554
54.67+1..73
18.12+11477
27.44=1.14
12.84=2.71
28.3--5.6
56.06:11.74
63113:1.18
18.17:00.0
31.15++2.20
18.90-1.80
36.57 20.12
25

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
F.2. Accuracies across the training on CIFARIO0 for separate tasks
In this section, we provide more detailed results for our experiments on CIFARI00, where we compare the performance of base and AC-enhanced networks with different continual learning methods on per-task basis. The results for each method are shown in Figures 19 to 29.
CIFAR100X5| ANCL
CIFAR100X10 ANCL
80
60
Accuracy
Acourscy
40
40
20
20
0
Current Task
Current Task
Figure 19. Per-task accuracy on CIFAR100X5 (left) and CIFARI00X10 (right) for ANCL.
CIFAR100x5| BIC
CIFAR100x10 BIC
80
Tea
65
Wethod Toss -AC
60
60
Accuracy
Accuracy
50
65
50
30
20
45
0
2
3
6
8
Current Task
Current Task
Figure 20. Per-task accuracy on CIFAR100x5 (left) and CIFAR100x10 10 (right) for BiC.
CIFAR100X5 DER+
CIFAR100x10 DER+
80
65
60
Mathol +AC
70
Vebol
Accuracy
65
Accuracy
60
50
50
45
40
40
35
30
3
8
Current Task
Current Task
Figure 21. Per-task accuracy on CIFAR100x5 (left) and CIFAR100xID (right) for DER++
26

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
CIFAR100x5 ER
CIFAR100x10 ER
70
80
60
Accuracy
Accurscy
660
50
40
20
20
Current Task
Current Task
Figure 22. Per-task accuracy on CIFAR100X5 (left) CIFARI00x10 (right) for ER.
CIFAR100X5 EWC
CIFAR100x10 EWC
60
60
Accuracy
Accuracy
40
Metiod
40
Does
20
Dec
20
0
Current Task
Current Task
Figure 23. Per-task accuracy on CIFARI00XS (left) and CIFARI00X (right) EWC.
CIFAR100X5 FT+EX
CIFAR100X10 FT+Ex
80
80
70
Accuracy
60
Accuracy
60
60
40
TeaLD
20
Current Task
Current Task
Figure 24, Per-task accuracy on CIFAR100:5 (left) and CIFAR100x10 (right) for FT+Ex.
CIFAR1 00x5 FT
CIFAR100x10 FT
60
Accuracy
Accuracy
40
40
Dosd
20
20
Current Task
Current Task
Figure Per-task accuracy on CIFAR100X5 (left) and CIFAR100x10 (right) for FT.
27

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
CIFAR100X5 GDumb
CIFAR100X10 | GDumb
45
Took
40
Accuracy
35
Accurscy
50
30
40
25
30
20
15
Current Task
Current Task
Figure 26. Per-task accuracy on CIFAR100X5 (left) and CIFAR100X10 (right) for GDumb.
CIFAR100X5 LODE
CIFAR100X10 LODE
80
70
70
Accuracy
80
Accuracy
60
50
50
Teb
Dec
40
40
30
Current Task
Current Task
Figure 27, Per-task accuracy on CIFAR100X5 (left) and CIFAR100x10 (right) for LODE.
CIFAR100X5 LWF
CIFAR100X10 LwF
80
60
60
Accuracy
50
Accuracy
40
40
TeaLD
30
20
20
3
Current Task
Current Task
Figure 28. Per-task accuracy on CIFARI00X5 (left) and CIFAR100X10 (right) for LWF.
CIFAR100X5 SSIL
CIFAR100X10 SSIL
70
70
Accuracy
60
Accuracy
600
50
50
Matroc
40
40
30
Current Task
Current Task
Figure 29, Per-task accuracy on CIFAR100X5 (left) and CIFARI00x10 (right) for SSIL,
28

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
F.3. Results complementary Section 3
F.3.1. CKA
CIFAR100x5 FT
CIFAR100K5
FT+Ex
CIFAR100X5 LwF
CIFAR1 00x5 BIC
AC1
0.36
C.94
0.30
0.87
0.96
0.35
0.92
0.81
0.99
0.89
0.98
0.98
0.99
0.88
0.56
0.34
AC2
0.53
0.91
0.87
0.97
AC3
AC4
0.92
0.57
ACS
0.88
0.81
0.78
0.77
0.80
0.85
0.82
0.81
0.97
0.96
0.98
0.85
0.95
0.82
0.89
0.57
AC6
0.74
0.80
68
0.65
0.79
0.74
0.72
7.
0.94
0.52
0.91
0.90
0.91
0.85
0.81
0.78
Finel
0.57
0.51
3.51
0.51
0.73
0.56
0.63
3.61
0.91
0.88
0.88
0.85
0.86
0.79
0.74
0.71
2
3
4
2
4
5
2
Task index
Task index
Task index
Task index
Figure CKA of task representations across ResNet32 layers through continual learning on CIFARIOO split into tasks. ACs are trained without gradient propagation.
CIFAR100x10
100AR100x10 FT+EX
CIFAR100x10 LwF
CIFAR100x10 BIC 0.99 0.97 0:95 0.93 0.83 0.91 0.90 0.89 0.86
AC1
0.86 0.82 0.88
0.85 0.52 0.75 0.50
0.77
0.76
0.96 0.91
0.88 0.56 0.84 0.51 0.82 0.79 0.77
0.59 0.99 0.89 0.99 0.98 0.80 0.99 0.88 0.97
AC2
0.89
0.83 0.77
0.77
0.87
0.98 98 0.90 0.99 0.90 0.97
0.96 0.82 0.91 0 0.86
AC3
0.97 0.90 0.89
0.85
0.80
0.95
0.85
AC5
0.72
0.99
0.89 0.99
0.57 0.62 0.90
0.63 0.52
0.86 0.81
0.79
0.78
3.76 0.76 0.76 0.75
0.74
0.96 0.95 0.96 94 O. 94 0.59 0.93 0.83 0.82
0.94 0.90 0.87 C.85 0.84 0.83 0.81 0.80 0.78
AC6
0.58
0.55 0.56 0.58 0.56 0.47 0.53
0.51
0.52
0.81
0.76
0.72 0.72 0.70 0.09 0.08 0.67
0.00
0.94 82 0.91 0.90 0.89 0.89 0.89 0.80 0.87
0.92 0.96 0.83 0.80 0.77 0.7 O 0.72
Final
0.41
0.38
0.41
0.43 0.43 0.30 0.42
0.36
0.41
0.50 0.74 0.68 0.58 0.65 0.55 0.62 0.51
0.51
0.52
0.90 0.88 0.87 0.86 0.86 0.85 0.85
0.84
0.90
0.93
0.79
0.75 0.73
0.71
0.71
0868 0.67 0.66
2
3
4
5
7
8
9
10
10
2
3
4
5
6
8
9
10
2
3
4
5
6
7
8
9
10
Task index
Task index
Task index
Task index
Figure CKA of first task representations across ResNet32 layers through continual learning on CIFAR100 split into 10 tasks. ACs are trained with gradient propagation enabled.
CIFAR100x5 FT
CIFAR100x5
FT+Ex
CIFAR100x5
CIFAR1 00x5 BiC
AC1
0.95
0.91
0.86
0.05
0.96
0.52
0.89
0.86
1.00
0.89
0.99
0.99
0.98
0.97
0.96
0.94
AC2
0.92
0.88
0.84
0.81
0.94
0.80
0.88
0.83
0.99
0.88
0.98
0.90
0.98
0.98
0.88
0.91
AC3
0.51
0.38
0.98
AC4
AC5
0.80
0.77
0.73
0.71
0.85
0.81
0.78
0.77
0.97
0.96
0.95
0.95
0.94
0.90
0.87
0.85
AC6
0.67
0.83
80
0.58
0.77
0.71
0.88
0.67
0.94
0.52
0.91
0.90
0.80
0.83
0.78
0.76
Final
0.53
0.47
0.48
0.49
0.72
0.55
0.63
3.61
0.91
0.88
0.87
0.85
0.85
0.79
0.74
0.71
2
3
4
5
2
Task index
Task index
Task index
Task index
Figure CKA of the task representations across ResNet32 layers through continual learning on CIFARIO0 split into tasks. ACs are trained with gradient propagation enabled.
29

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
F.3.2. AC PERFORMANCE DIFFERENCE OVER FINAL CLASSIFIER
CIFAR100x5 FT
CIFAR100x5 FT+Ex
CIFAR100x5 LWF
CIFAR100x5
Accuracy difference
3.5
20
2.5
15
10
2.0
10
1.0
1.5
0.5
5
1.0
0.0
0.0
AC1
AC2
ACS
AC4
ACS
AD8
AC1
AC2
AC5
AD4
AC5
ADS
AC1
AC2
AC3
AC4
ACS
AD5
AC1
AG2
AC3
AC4
ACS
ACB
Figure 33, Per-task difference between the accuracy (only positives) of the ACs trained with lincar probing for 5 task split of CIFAR100.
CIFAR1 FT
CIFAR100x10 FT+Ex
CIFAR100x10 LwF
CIFAR100x10 BIC
25
Accuracy diference
3.
15
10
3
2
AC1
AC2
ACS
AD4
A.55
AD8
AC1
AC2
AC3
AD4
AC5
ACS
AC1
AC2
AC3
AC4
AC5
AC5
AC1
AC2
AC3
AC4
ACS
AC5
Figure Per-task difference between accuracy (only positives) of the ACs trained with gradient propagation for 10 task split CIFAR100.
CIFAR1004 FT
CIFAR100x5 FT+EX
CIFAR100x5 LwF
CIFAR100x5 BIC
3.0
7
4.0
Accuracy difference
3.5
2.5
80
2.5
15
1.5
3
2.0
2
1.0
0.5
0.0
0.0
Figure 35. Per-task difference between the accuracy (only positives) of the ACs trained with gradient propagation for 5 task split of CIFAR100.
30

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
F.3.3. OVERTHINKING
20.0
17.5
Overhining DM.
40
Unquue overtiiking
Vibos samp
60
20
5.0
2.5
INF
AC1
AC2
AC3
AG2
ACS
ACS
0.0
AC1
AC3
AC4
ADK
(a) Overthinking
(b) Overthinking per AC
(c) Unique overthinking
Figure 36, Overthinking for 5 task split of CIFARI00, without gradient propagation.
50
Unique overthnking (S)
15.0
Oowethiniing [S]
Mixd: samples
40
10.0
20
5.0
2,5
10
J184
FT
FT-Ex
LAF
AC1
AG2
AC5
AD4
AC5
ADS
0.0
AC1
AC2
AG2
AC4
ACS
AD8
(a) Overthinking
(b) Overthinking per AC
(c) Unique overthinking
Pigure 37, Overthinking for 10 task split of CIFARIO0, with gradient propagation enabled.
Unique overthinkigg (90)
12
Ovvetteeking [N]
100
Medss samges
40
20
20
10
J:44
F7
FT-Ex
LAF
8C
AC1
AC1
ACS
(a) Overthinking
(b) Overthinking per AC
(c) Unique overthinking
Figure 38. Overthinking for task split of CIFARI00, with gradient propagation enabled.
F.3.4. TRAINING ACCURACY CHANGE FOR CIFAR100 TASK SPLIT
18
Accursyy change
+2
Figure 39, Difference between classifiers performance after enabling gradient propagation on CIFAR100 split into 5 tasks.
31

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
F.4. Classifier selection patterns and accuracy during inference
In Figures 40 and 41, we show the classifier selection distribution and accuracy for static inference in the CIFARI00 experiments from Table 1. While early classifiers are rarely selected, intermediate classifiers often match outperform the final classifier. We hope these results help explain the performance improvements from adding the ACs.
CIARRT0555, ANCL Choccfer Selection
CIFART COc5, ANC Arrurryy
CITARTODEES, AND Clateeeer Selectios
CENATOnA10 ANCL Accurwyy
Ta
Accunry
Accurnty
TD
Ta
Taxt
CIARLOOKS. EK| Classfler Solectinn
CIARROONS, BR| Accuracc
CH ER| Sellectiion
CIFAROOOELD Accuracy
TabiD
T
CIRAR10X55, EMC| Clasifier Sdection
21RRR1006, BWC| Accurccy
CHAR100 EWC Cussifirr Solacioo
CRAR100110, EWC| Accuracy
213
471
114
11
26
314
94
264
122
344
342
223
22
314
84
443
123
13
321
382
144
127
252
303
312
114
331
253
121
277
24
2IFRR2005 FT| Sustffersitctiinn
CIARLDOKS, FT| Accuracy
CIFAR100K10 FT| Classifar Silection
CIFARL00100 FT| Accuracy
12
3BC
115
45
48
43
33
18
13
103
30
3M
313
132
131 33 114 15C
033
301
U1
11
43
14
303
92
14A
141
114
176
344
224
23
314
12
314
183
113
342
152 124 113
122
117
540
113
154
224
25
117
14
354
134
224
354
155 134 113 532 111
140
13
214
112
14
81
127
124
154
154 194
282
213
103
43
32
111
351
515 134 1M 123
10
Figure 40. Distribution and accuracy of classifiers for ANCL, BIC, ER, EWC and FT.
32

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
CINALOONS, FT+Ex| (henderr Selection
CEPAR1OAS, FT+EK] Accunecy
CIPAR100110, FT+CA
Caassfer Seletton
CIFARIO0ED, FT+ Arcurery
24
X4
41
22
12
21
44
24
32
11
11
11C
30
122
174
K4
84
214
212
244
214
22
124
22
114
122
124
130
151
112
10
221
CEARTREES GDumb Cleeet anvoterSetee Selection
CEAATOOES, SDumb| Arrizaty
15410, cDunh|
CRAEE Selection
20470, GDumb Accureyy
351
212
64
18
214
341
344
24R
144
316
13
244
540
64
1H
134
32
543
54
381
134
M5
3M4
16
19
114
123
24
123
211
331
312
234
197
113
548
548 153
12
153
11
314
253
643
111
13
10
141
A
CIAAR10be, LODE
Chesserr Se wction
OIFRRODDSS LOOT Arturary
CITARTADeES, (oDr Dhedter Selection
CFXA1On210, LOGE Accurwyy
TakkD
Tax
LW Accuncy
Accurncy
TD
TabiD
Tax
2005 SEL| | Dasetter Selection
CIARLONS SSI| Accuncty
CEARE 00:10, 5M1 Clocdller Selection
GFAR100510, SUL Accurary
Kg
Ta
Tax
Figure 41. Distribution and accuracy of classifiers for FT+Ex, GDumb, LODE, LwF and SSI...
33

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
E.5. Diversity of the auxiliary classifier predictions
To investigate the diversity among the auxiliary classifiers, we measure unique accuracy - the percentage of the samples correctly predicted only by this given classifier and misclassified by all other classifiers and present the results in Figures 42 to 45, The intermediate classifiers learn to specialize to some degree across all analyzed methods and all intermediate classifiers, especially on the older tasks, with the trend more visible for the naive and exemplar-fre free settings.
CIFAR100x5
C0FAR100x5
FX
CIFAR100x5
CIFAR10DS
BIC
L1.B3
2.55
2:13
2
19b
0.12
1.57
8.25
0.31
0:15
0.30
0.42
0:35
5./8
1.10
1.00
1br
0.tb
1.15
0.32
0:95
0.27
0:03
1,32
0.64
L1.B5
3.32
192
1.77
1.37
0.35
1.78
0.27
0.57
0.37
0.30
0.57
0:41
0.69
0.97
1.68
1:55
1.92
1.37
0.72
0.78
0.32
1.20
0.66
0.65
L2.B2
3.85
1:63
0.20
3.29
0.60
0.63
0:50
0:47
9:23
0:4G
0:30
0.83
0.73
0.75
1.02
0.77
0.53
0.76
L2.84
0.78
0:35
0.72
6.79
0.64
L3.B1
0:62
1:68
1.77
2.20
1.40
0.78
L3.B3
0-33
2.05
2.47
4.80
1.41
2.27
3.00
2.48
15
3.10
121
2.40
4.15
1.82
1.00
2.05
0:
2.67
1:53
1.87
1.07
3.33
2.27
Final
0:33
0.43
1000
2:25
10.00
2.48
3.83
3.93
4.10
521
2.3/
3.00
5:33
1.533
4.12
6.17
11:32
4.77
3:50
2.97
2.03
3:08
4.40
3.84
2
5
Avg
1
2
3
Avg
Avg
1
2
3
4
Avg
Task
Task
Task
Task
Figure 42. Percentage unique samples that only a single given classifier classifies correctly for different tasks on CIFARIO0 split into 5 tasks. The classifiers are trained without gradient propagation.
CIFAR1 10 FT
CIFAR1 00x | FT+EX
MIFAR100X11 LwF
CIFAR BIC
L1.85
4.33
2:20 480 2. 218
0:93
53
0.54
0.77
901
250
1.00
0.83 27
1:7
0.50
L2.B2
4.77
174444
2:20
5.67 4.30 23 27.000
7.70
0:60
80
0 1.77 167 03 0:00 130
10..43..0000 43
1044
0. 8:
0 00.0 1 10 0
L2.84
2.00
00
00
327
3474
2323 2
13)
0.00000
1.00
0:90
17 A2
17 -0 1211112000000 2000
0.25 1:11
1:00
2317
102 0000 1
1.13..44777777 144
1.13. 07
07 01
1 17 1:00 130
L3.B1
3.00
180
487 3:30 7:4
0.09
3.57
3.07
60 2.7 250 2.23
2.69
0.002.00
6.13 2.13
1.80
2.00
87
0:30
0
0.10
1.80 2.27 199 245 1.50 2.04
L3.83
0.08
0.30 33 73 2.69
2.63
2.0
0.33
1:29:3.0
5.07
17
1.53
2.10 157 2.0
2.27 2692.4
9.000.37
2
6 Task
Avg
8
2
6 Task
10
Task
Task
Figure 43. Percentage of unique samples that only a single given classifier classifies correctly for different tasks on CIFARIO0 split into 10 tasks. The classifiers are trained without gradient propagation.
CIFAR100x5 FT
100x5 FT+EX
CIFAR100x5
LwF
CIFAR100x5 BIC
L1.B3
2:48
3.25
2007
2:47
0.50
2:15
0:
1B
0:18
027
0.00
C.42
0.22
5.66
0:72
1:47
100
0:72
1.07
0:45
0:61
1:33
0:15
0:47
0 T0
L1.B5
0.
2:25
3.15
2.38
0:4b
1.54
0.42
0.60
0:18
0.5
0.67
0.45
0.53
1:30
0.m0
0.52
0.63
0.73
0:00
0.85
0.66
L2.82
0.65
0.88
0.83
0:38
1.25
0.58
0.67
1.00
2.50
0.77
1.02
0.92
1.4G
L3.B1
2.51
9.
3.7
3.00
0:68
2.58
2.37
2.63
2.07
185
0:42
7.50
3.10
1:20
0.87
1.02
1:10
1:51
1:37
1:43
1.75
L3.83
0.47
1:22
1.43
3.b
0:38
1.39
1.80
1.80
1.30
2.12
0.93
1.67
0.82
2.53
1:13
25r
1.822
1.F7
1.25
1.42
1.43
1.23
1.30
1.61
Final
0:31
0.32
0.90
1.02
6:38
1.51
2.48
2.60
2.85
2
1.40
2.45
2.10
1.87
2.15
2.52
3.02
2.51
1:95
2.75
2:42
2:05
4.70
2.79
2
3
5
Avg
1
2
3
Avg
Avg
1
2
3
5
Avg
Task
Task
Task
Task
Figure 44. Percentage unique samples that only a single given elassifier classifies correctly for different tasks on CIFARI00 split into tasks. The classifiers are trained with gradient propagation enabled.
F.6. Upper bound analysis of multi-classifier networks
To quantify the potential of the auxiliary classifiers, we consider an oracle multi-classifier network as an upper bound for our method. When evaluating such an oracle, we obtain predictions from all of its classifiers and consider a prediction correct when least one classifier (auxiliary classifier or the original network classifier) is correct. Therefore, the oracle has an
34

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
CIFAR100X10
1IFAR100x10
FT+EX
1IFAR100x10 LwF
CIFAR100x10 BIC
L1.B3
3:003.03330
1.04
2.80
C33
C/6C
30
0122
0.r 107 30
L1.B5
2,7
522
2.23
0.10
25
9.7
6.77 5.00
083
6:50
85
1:17
60
90
0.30
0.60
0.57 045 0.63
37 8803
1:3
1,2
1:51
100
055
170..00 8.00
1.09
1-17
0.70 0.73 0.57
0.90 100 0.67 0.57 0.S 5.00. 5.76
L2.82
4.24
3.0
5:1
2:00
23333
3
80
20
1
1:77
60 1.71500 1.63
777 73
1.10
13
0.
18
0.70 1.00
1.42
0.0
23.0
6177771111 18 0.731.0
L2.B4
17
2010
1:57
70 123
50000.00
6.
07
L3.B1
10
59.5100.00
1:30
1,23
1:10
1.00
1:31
2.47
2021
1:10 1:23
Final
93 001
22
1,40
1/32
1: 103 01
2.50
192222
1:30
2.102.00
250 2:20 2.922.
Avg
10.Avg
Avg
6
A0.
Task
Task
Task
Task
Figure 45. Percentage of unique samples that only single given classifier classifies correctly for different tasks CIFARI00 split into 10 tasks. The classifiers are trained with gradient propagation enabled.
ideal algorithm for combining classifier predictions and always returns the 'best case' prediction from all the classifiers. We measure the difference between the accuracy of such an oracle network and the accuracy of a standard single-classifie network and present the results for first task data, last task data, and average over all the tasks in Table 11 and Table 12 for both linear probing and ACs, As in our previous analysis in Section 3, exemplar-free methods show more variance in the performance across tasks. However, the average difference across all tasks is also significant for exemplar- based methods, with the oracle for the best-performing method BiC achieving approximately a 30-40% relative increase in overall accuracy. Those results indicate that, while our simple setup achieves consistent improvement, it is still leaves room for improvement in the future work.
Table I. Upper bound on accuracy improvement on 5 tasks of CIFAR100 when using oracle multi-classifier network, trained with linear probing and auxiliary classificrs.
Task 1
Task
Task 3
Task
Task
Avg
CIFAR100X5, Linear Probing
FT
31.82:12.23
46.90:11.00
54.22:0.59
43.30=1.24
6,57400.6
36.5611.52
FT+EX
12.2840.10
14.58+0.33
12.7241.80
13.87+0.93
11.37:40.1
12.96:10.62
LwF
35.07:42.03
28.88::2.51
20.92:+2.01
1..25=1113
9,55:+447
21.93:+.200
BIC
15.15:00.11
17.004-2.33
18.73+1.50
17.93+2.26
16.03++2.39
16.97:
CIFARI00K5, Auxiliary Classifiers
FT'
24.65:+2.82
45.07:15577
56.87+3.68
48.82-14.36
9.05-10.65
36.89::1.3
FT+Ex
14.42:00.60
17.42+1.12
15.70=
15.50=
10.88:+0.40
14.74:10.66
LWF 18.52-11.89
22.57:2..8
21,5720.93
18.57=0.95
15.07:22.08
19.26:10.61
BIC
15.88++0.96
18.42:0.466
18.83++2.29
18.72=#0.06
15.43-43.06
17.46++0.61
Table 12. Upper bound on accuracy improvement on tasks of CIFARI00 when using oracle multi-classifée network, trained with linear probing and auxiliary classifiers.
Task
Task 2
Task 3
Task
Task 5
Task 6
Task 7
Task
Task 9
Task 10
Avg
C1FAR100100, Linear Probing
FT 30.00=7.52
16.10+3.900
40.47:46.03
23.37=4.57
48.70=0.72
41.43+-5.5 52
32.80+1..8
44.40:14.83
28.87=0.97
9.03+11.22
31.52:0.0
FT+Ex
17.17-1.24
16.63+-0.22
18.40+2.72
18.10+1.44
17.6740.78
16.30+-1.31
18.90+0.79
15.7741.30
17:00-0.44
11,90+1.32
16.78+0.38
LoF BIC
48.93=6.24
30.30+3.00
33.6343.33
28. 28.93+55.05
25.60+2.13
21.50+55.50
13.30+2..17
1..47=1.50
11.33+1.24
8.40+1.54
23.64 1.29
19.03=1.10
20.73::2.04
20.5042.74
20.83+2.377
17.83=1.12
19.1740.67
16.83+33116
18.90=0.53
20.77=4.80
17.7344.31
19.23:10.70
CIFAR100x10, Auxiliary Classifiers
FT
12.93-0.38
14.30.14.69
35.07.10221
25.70.14.90
46.27
38.70112.7
32.50
43.77.1..3
34.73.3.46
9.20
19.92
FT+Ex
20.03=1.56
17.7741447
18.8710.60
20.33:1.70
18.13:2..25
20.5010.78
18.9741995
17.47=1.36
19.13=248
12.3010.80
18.35:10001
LoF
23.80-1.46
12.60.14.88
22.93.11.65
21.50.12.91
28.3011880
25.33
12.70.11.49
21.53.1.31
18.0015.12
15.2712.77
20.10.10.74
18.87=2.76
20.60::0:4
21.53:22.4
22.70:2.52
18.6021.51
19.87+2.01
17.1741.88
19.7720.57
18.7728.36
17.7347.03
19.3620.78
35

---

Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
E.7. Dynamic inference rule ablation
In Table 13 we demonstrate the accuracy of variants of dynamic inference for different confidence thresholds À for CIFARI00. We compare the standard, early-exit paradigm,, where the network returns final classifier prediction in case no classifier meets the exit rule, and the paradigm used in experiments where the network defaults to the most confident prediction. Using the most confident prediction outperforms the standard early-exit rule, which is consistent with our analysis that showed that the last classifier is not always the best in continual learning and that the early classifiers exhibit lower forgetting for carlier task data.
Table 13. Comparison between dynamic inference performance with different confidence thresholds when using maximum confidence prediction (MC) and final classifier prediction (1 aast as the default output for multi-classifier networks trained with linear probing (I.P) jointly with the network with gradient propagation (AC). Using max confidence prediction yields hetter accuracy.
FT (LP)
FT+Ex (LP)
LwF (LP)
BiC (LP)
FT (AC)
FT+Ex (AC) LwF (AC)
BiC (AC)
CIFARIO0XS
Last
24.14+1.33
28.43
37.34+0.09
48.35++0.23
27.64+0.97
28.87+0.32
38.08
48. 08.44000.0
0.5
MC
14.73+1.48
28.3340096
36.95=0.200
48.48+0.40
18.10+t.03
28.85+0.31
38.36+0.59
4..37=0.31
Last
23.67
3..60+1.26
40.21=0.08
49.25=00.3
26.00
36.27+0.28
1..1
0..45=0.73
0.75
MC
26.66.1 1.70
35.09.11.38
39.70
49.74.10.48
28.9111.07
36.18.10.17
40.33
50,19
Last
20.98
37.27111.10
39.97-10.27
49.18
22.38.10.33
38.22.10.19
39.28
49.35
0.9
MC
26.70.11.55
36.5711..3
40.07:10.10
49.89-10.52
28.4411.00
38.2711..13
40.50
50.39
Last
19.9
37.48+0.98
39.62:t0.23
49.15=0.31
20.69++0224
38.50+0.17
19.25=1.36
09.24=0063
0.95
MC
26.74+t 1.40
36.77+1.29
40.07++0.08
49.89-0.52
28.25++1005
38.62+0.22
40.53++0.44
50.40=00666
Last
19.19+0.227
37.46+00.2
39.40=t0221
49.14+0..2
09.49
38.55+0..8
39.22+
0..24
0.98
MC
26.83:11.23
36.8111.27
40.10.10.07
49.89-10.52
28. 1.11
38.72.10244
40.55
50.40
Last
18.9410.15
37.44.10.88
39.32.10.22
19.14-10.32
19.08
38.55110.44
39.22:11.35
49.23
0.99
MC
26.82.11.22
36.8311.277
40.10:10.07
49.89-0.52
28.20111.09
38.7510.27
40.55
50.40-10.68
Last
18.39+0008
37.43+0.85
39.11=0.226
49.14-00.32
18.35+0.30
38.51+0.43
1..2
0..22+0065
1.0
MC
16.82+1.19
36.83+1.277
40.10+0.07
49.89+0.52
18.18+1.07
38.75+0.26
40.55+
50.40++066
CIFAR100x10
Last
15.22+1.64
27.03
28 28.69:0.779
42.37=1.60
15.85
27.6811042
28.96#1.29
12.40
0.5
MC
16.05
27.04
28.06:10.33
42.62+1..75
16.79+1.34
27.7210.37
29.09=1.10
42.67:1144
Last
14.12:10.98
33.6911.20
31.1010.87
43.95=1.69
13.31
34.4010.47
28.65#1.72
44.93-10.3
0.75
MC
1..47+1..11
33.84.1..00
30.17
44.59
16.7741122
34.6410.41
29.72+1.19
45.83+11.51
Last
12.19:+0.51
34.61+1.005
31.03-+0.92
13.87++1.3
11.25++1.04
35.83+0.51
28.30+
44.55=0062
0.9
MC
1..71+11.3
35.41
30.54
14.82+1..11
16.82+1.114
36.59+1.52
29.79+1.21
46.1411..66
Last
11.25:+0555
34.4911.110
30.63+1.10
43.79=1.722
10.62++0.83
35.63+:0.43
28.27=1.79
44.33-10.3
0.95
MC
1..74+1.31
30.58
30.59=0.67
44.84=1.72
16.89
36.92::0.99
29.79#1.21
16.17=1.45
Last
10.51+00.35
34.27+11.13
30.17+1.19
43.77=11.72
10.09
35.29400.99
28.22+11811
4..23
0.98
MC
1..77+1..0
35.61
30.60-10.68
44.84
16.88
36.97-10.40
29.79+1.21
46. 16.19
Last
10.19400.7
34.26.11.10
30.00+1.200
43.76-11.72
9.92-10.69
35.10:4..45
28.20
44.21
0.99
MC
1..7
35.61
30.60.10.68
44.84
16.88.1..08
36.98.10.40
46.19 29.7941.21
Last
9.79:10.3
34,22+1.088
29.65+1.19
43.75=11.72
9.76+00.33
34.9310.40
18.1
44.19=0.51
1.0
MC
1..77+1.30
35.62++0.89
30.60=0.66
44.84=1.73
16.88
36.97+0.39
29.79+1.21
46. 1.47
36


Reviews:
[{"review_id": "CU9u0tKNjx", "reviewer": "Reviewer_FvJb", "summary": "This paper introduces Auxiliary Classifiers (ACs) to enhance performance and efficiency in Continual Learning (CL) by leveraging intermediate representations in neural networks. The primary challenge is catastrophic forgetting, where new knowledge acquisition disrupts previously learned information. The key conceptual idea is that intermediate neural network representations exhibit higher stability than final-layer representations, making them less prone to forgetting. The authors propose attaching auxiliary classifiers (ACs) to intermediate layers to Improve accuracy by reducing overfitting in later layers, accelerate inference through early classification when confidence is high, and enhance generalization by leveraging more stable feature representations. The paper evaluates ACs across multiple continual learning methods using CIFAR100 and ImageNet100 datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for all the suggestions. Below, we address the points mentioned by the Reviewer. We will include the clarifications and the Reviewer’s suggestions in the updated version of our paper.\n\n## Accuracy metrics and backward transfer (BWT)\n\nIn all our experiments, we report the average accuracy across all classes after the final task. \n\nBWT can be seen as the negative of forgetting, which we discussed in the response to the Reviewer Y8XT. It is hard to observe BWT consistently in our setting across the whole training course and interpret this metric, so we do not report it like most class-incremental learning works (the metric is mostly used in task-incremental settings). However, BWT can be derived from the per-task accuracy plots we added in response to Reviewer Y8XT ([link](https://imgur.com/a/0O3btHy)). \n\n## Early vs later layer representation stability\n\nWe agree with the reviewer that feature-sharing is a useful concept when explaining the results observed in our paper. Early-layer representations tend to capture shared, low-level features that are less task-specific, making them more stable across tasks and hence suffer less from forgetting. The more complex, later-layer features, are more task-specific and consequently suffer from more forgetting (since they might not be required for current task data). As a consequence of this phenomenon, during continual learning the ACs based on lower layers can potentially outperform those of the later layers. \n\n## Forgetting in the ACs\n\nAs noted by the Reviewer, our ACs are still classifiers and as such are also susceptible to forgetting. Our approach leverages the stability of earlier features shown in Figure 2 but also benefits from the diversity of features used by different ACs. Since we classify based on the information aggregated across multiple ACs, it is enough if just one classifier returns a confident prediction for a given sample to make the correct prediction. Empirical results indicate that AC-based networks generally exhibit lower forgetting on a per-task basis across continual learning (see the forgetting plots under the [link](https://imgur.com/a/0O3btHy) and the response to Reviewer Y8XT), supporting our intuition that using multiple ACs results in more robust classification.\n\n## Thresholding strategy\n\nThe Reviewer correctly points out that our inference strategy is influenced by prediction confidence, which varies for each AC and the task. While confidence-aware inference strategies could improve performance, they are complex enough to warrant separate papers (e.g., Meronen et al., *“Fixing Overconfidence in Dynamic Neural Networks”*, WACV 2024) and often rely on calibration on a holdout dataset, which limits their applicability to exemplar-based methods. To maintain simplicity and generalizability, we use a shared threshold. The fact that our approach performs well without per-AC adaptation further highlights its robustness and broad applicability.\n\n## Overthinking definition\n\nOur use of the term “overthinking” follows the SDN paper and is well-established in the literature, so we skipped it to keep the paper concise. Figures 4b and 4c are supposed to provide a more nuanced analysis of the phenomenon.\n\n## Unique overthinking\n\n\"Unique overthinking\" indicates how many samples can be correctly classified by a unique AC. It highlights that each classifier specializes in different subsets of the data in a non-redundant way.\n\n## How are the ACs initialized?\n\nWe follow the same FACIL protocol for both the main classifier and the auxiliary classifiers (ACs) - we add a new head for the classes introduced in each task. \n\n## Gradient propagation in Section 3 \n\nWe mention that gradients from ACs are detached from the backbone in Section 3.2 (L#201-204); in Section 3.4, we compare this with a setup where gradient propagation is enabled end-to-end to empirically evaluate which approach yields better results. Then in all following experiments, we used the setup with enabled gradient propagation.\n\nGradient propagation can sometimes negatively affect individual classifier performance. However, while some individual classifiers may degrade slightly, the overall benefit outweighs these effects. In most cases, gradient propagation training leads to better results (see Appendix C.3.). In Appendix E, we also show that findings from our analysis are consistent across with both detached and enabled gradients.\n\n## Training times with ACs\n\nSee the response to the Reviewer dDEJ.\n\n## Figures clarification\n\nFigure 3 is intended to highlight which auxiliary classifiers outperform the final classifier.\n\nThe marks in Figure 5 are supposed to represent correct and incorrect predictions. \n\nThe accuracy reported in Figures 1 and 6 is the average accuracy after the final task.", "claims_and_evidence": "* The paper is mostly clearly written, making it easy to follow the authors' claims and methodologies. However,\n* Figure 3 is unclear. It appears that all auxiliary classifiers outperform the final classifier (non-negative accuracy differences). If this is not the intended message, the figure needs clarification or revision.\n* Figure 5 is not clearly presentated. Specifically, what do the check marks and x-marks represent? Clarification is also needed on how this figure illustrates static inference as described in the accompanying paragraph.", "methods_and_evaluation_criteria": "* The experimental design is thorough, valid, and effectively covers relevant scenarios.\n* Continual Learning literature commonly reports metrics such as average accuracy and average backward transfer (BWT). However, this paper primarily focuses on final-task accuracy. I believe Tables 1 and 2, and Figures 1 and 6, should includ average accuracy and BWT.\n* The authors provide inference costs, but training costs should also be reported, given that adding auxiliary classifiers, both with and without gradient detachment, could significantly impact computational overhead.", "theoretical_claims": "* The paper does not present explicit theoretical claims or proofs.", "experimental_designs_or_analyses": "* The experimental design is sound and comprehensive, covering several baseline comparisons and tasks relevant to continual learning.\n* However, clarity regarding the training procedures of auxiliary classifiers (e.g., incremental class additions, gradient propagation handling) needs improvement. For example, how are new classes incrementally added for each AC? Are ACs initialized with a fixed number of classes, or do classes expand progressively? Additionally, the authors mention \"enabled gradient propagation\" in Section 3.2, but it is unclear whether this explicitly means gradients are not detached. If this is the case, could the authors clarify the reasoning behind this choice? Moreover, could gradient propagation negatively affect intermediate representations, potentially causing performance degradation, as suggested by the LwF case in Figure 4(d)?", "relation_to_broader_scientific_literature": "The paper distinguishes the novel usage of intermediate auxiliary classifiers from existing CL techniques.", "essential_references_not_discussed": "No essential missing references were identified in the current version of the paper.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "* Figures and their corresponding textual explanations often appear on separate pages, negatively impacting readability. Repositioning figures closer to their descriptions would greatly enhance readability.\n* For figure 1 and 6, it should be stated if the accuracy is average task accuracy or final task accuracy.\n* Including explicit accuracy values for each AC in Figure 4 would be helpful, as parts (b) and (c) currently show only relative performance.\n* Defining overthinking as \"cases where samples correctly classified by early classifiers are misclassified by the final classifier\" is intuitive but may oversimplify the phenomenon. A more nuanced justification or further context would strengthen this definition.", "questions_for_authors": "* Do ACs also suffer catastrophic forgetting, trained with and without graident detach? If ACs are continually trained alongside the final classifier, it seems plausible that ACs would also forget older tasks. Could you discuss or demonstrate the robustness of ACs across sequential tasks?\n* While the paper demonstrates robustness in early representations, it also acknowledges later layers’ superior ability to classify larger subsets. What do you believe explains this phenomenon?\n* In Figure 4, \"unique overthinking\" appears relatively minor (around 10%). Could the authors elaborate further on the significance or implications of this proportion?\n* If confidence measures play a key role in your inference strategy, it would be valuable to analyze and report confidence metrics explicitly for each AC.\n* A common viewpoint in deep learning literature suggests that early layers learn general representations, while later layers acquire task-specific representations. While authors claim that early layers do not \"forget old tasks,\" it could be also possible that the apparent stability of early layers might reflect learned shared characteristics (e.g., color, shape) rather than true resistance to forgetting. What is your perspective on this interpretation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "iPkFwGs6Zy", "reviewer": "Reviewer_dDEJ", "summary": "The manuscript analyzes the effect of class-incremental training on the parameters of a neural network, finding that deeper layers tend to be more affected from catastrophic forgetting. To exploit this it introduces a series of Auxiliary Classifiers (AC), finding that their combination can significantly make the predictions more robust.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the time spent on our work and the suggestions for improving its quality. We appreciate the Reviewer’s mention of *Boschini et al., \"Transfer without forgetting.\" ECCV 2022*;  we will include this paper in the updated related works. Below, we address the points raised by the Reviewer.\n\n> I found the use of a single seeded run for the results of the ViT-based models (lines 301-302) to be concerning. In my opinion, for a proper evaluation the tests should be repeated across multiple runs without seeds.\n\nWe appreciate the Reviewer’s concern. Due to computational constraints, we initially conducted these experiments with a single seed, as ViT-based models are particularly expensive to train. In response to the Reviewer’s suggestion, we have now rerun the ViT experiments from Figure 7 using two additional seeds (three in total). The updated figure, including confidence intervals, is available at [link](https://imgur.com/JTuI5wn), demonstrating that our findings remain consistent across multiple runs. We will incorporate these results into the revised version of our paper.\n\n> While the manuscript investigates the change in the intermediate representations for a backbone trained from scratch, it could be interesting to analyze the effect of AC on pre-trained backbones, especially in light of [1] which conducts the same initial tests with CKA for pre-trained networks.\n\nThank you for the interesting suggestion. We conducted warm-start continual learning experiments (Appendix B.1) as a controlled alternative to pretraining, empirically demonstrating that our approach remains effective in such a setting. We consider the investigation of pre-trained models with ACs a promising future work direction.\n\n> How much is the overhead of computing all gradients for the losses of the ACs (e.g., in terms of required GPU memory and training time with respect to the baseline)?\n\nWe appreciate the Reviewer’s question. We measured training times and peak memory usage for training ResNet32 with different numbers of ACs on CIFAR-100, summarized in the tables below. \"0 ACs\" refers to a standard network. We report training times in hours (mean from three runs) and peak memory usage in GB. For memory, we include one table, as it does not depend on the data split.\n\n## Training times (in hours) - 5 tasks\n|ACs|ANCL|BiC|ER|EWC|FT|FT+Ex|GDUMB|LODE|LwF|SSIL|Avg|\n|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n|0|2.1|1.4|2.4|1.2|1.1|1.3|0.6|3|1.2|1.7|1.6|\n|3|2.7|1.8|2.8|1.5|1.3|1.5|0.8|3.6|1.5|2.1|2|\n|6|3.3|2.1|3.3|1.8|1.6|1.7|1.1|4.3|1.8|2.5|2.4|\n|12|4.5|2.9|4|2.5|2|2.3|1.5|5.5|2.5|3.2|3.1|\n\n## Training times (in hours) - 10 tasks\n|ACs|ANCL|BiC|ER|EWC|FT|FT+Ex|GDUMB|LODE|LwF|SSIL|Avg|\n|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n|0|2.8|2|2.5|1.4|1.4|1.7|1.2|3.3|1.4|1.8|1.9|\n|3|3.6|2.5|3.2|1.9|1.6|2|1.5|4.2|1.9|2.3|2.5|\n|6|4.4|3|3.8|2.2|1.9|2.4|1.7|5.2|2.4|2.9|3|\n|12|6.8|4.2|4.8|3|2.4|3.3|2.3|6.9|3.5|3.9|4.1|\n\n# Peak GPU memory usage (GB)\n|ACs|ANCL|BiC|ER|EWC|FT|FT+Ex|GDUMB|LODE|LwF|SSIL|Avg|\n|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n|0|2.1|2.09|2.1|2.1|2.1|2.1|2.39|2.1|2.1|2.18|2.14|\n|3|2.29|2.22|2.18|2.56|2.25|2.18|2.56|2.22|2.22|2.29|2.3|\n|6|2.47|2.32|2.27|2.73|2.41|2.27|2.73|2.31|2.3|2.41|2.42|\n|12|2.82|2.5|2.43|3.02|2.72|2.43|3.03|2.49|2.5|2.65|2.66|\n\nOur standard setup (6 ACs) results in approximately 50% training time and 10% memory overhead. The exact overheads will depend on the model, AC architecture, and training environment. However, we do not consider this overhead prohibitive for real-world use cases, particularly in offline class-incremental learning, where training resource constraints are not usually strict; therefore, we also did not optimize the code towards such constraints. As our primary concerns are efficiency and performance during the inference time, we consider the discussed overheads acceptable. We appreciate the Reviewer raising this important point, and we will incorporate this discussion in the paper.", "claims_and_evidence": "The claims are convincing.", "methods_and_evaluation_criteria": "The choice of benchmarks and competitors seems adequate.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The choice of benchmarks and competitors is adequate. However, I found the use of a single seeded run for the results of the ViT-based models (lines 301-302) to be concerning. In my opinion, for a proper evaluation the tests should be repeated across multiple runs without seeds.", "relation_to_broader_scientific_literature": "While the analysis on the change in the intermediate representations in a class-incremental environment has been conducted in several other works [1,2,3] (with [1,2] being already cited in the manuscript), the use of auxiliary classifiers to take advantage of the smaller changes in parameters of shallower layers seems novel to me. \n\n[1]: Ramasesh, V. V. et al. “Anatomy of catastrophic forgetting: Hidden representations and task semantics.” In ICLR 2020.  \n[2]: Zhao, Haiyan et al. “Does continual learning equally forget all parameters?.\" ICML 2023.  \n[3]: Boschini et al. \"Transfer without forgetting.\" ECCV 2022.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- While the manuscript investigates the change in the intermediate representations for a backbone trained from scratch, it could be interesting to analyze the effect of AC on pre-trained backbones, especially in light of [1] which conducts the same initial tests with CKA for pre-trained networks.  \n- How much is the overhead of computing all gradients for the losses of the ACs (e.g., in terms of required GPU memory and training time with respect to the baseline)?   \n\n[1]: Boschini et al. \"Transfer without forgetting.\" ECCV 2022.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See section on \"Other Strengths And Weaknesses\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SvxRdbMe3f", "reviewer": "Reviewer_Y8XT", "summary": "The paper shows that intermediate representations are less prone to forgetting in the continual learning paradigm, which is aligned with previous observations in the literature. Based on this finding, they propose using auxiliary classifiers in the intermediate layers and showed that the method consistently outperforms the single classifier, when combined with various continual learning techniques.\n\n## Update after rebuttal:\nThe authors have addressed my comments. I will keep my positive score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the time spent on our work and the suggestions for improving its quality. Below, we address the points raised by the Reviewer.\n\n> Since the method focuses on forgetting, I expect to see a metric to measure it in the experimental results.\n\nWe appreciate the Reviewer’s suggestion and have now included forgetting metrics for CIFAR-100. Our method generally helps in mitigating forgetting. Even in cases where we see more forgetting with ACs, it still outperforms the baseline in terms of accuracy (see Table 1). \n\n|||FT|FT+Ex|GDumb|ANCL|BiC|DER++|ER|EWC|LwF|LODE|SSIL|Avg|\n|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|\n|CIFAR100x5|Base|64.85$\\pm$1.27|**44.80$\\pm$0.79**|12.01$\\pm$0.63|36.94$\\pm$0.79|9.33$\\pm$2.89|7.42$\\pm$4.17|47.25$\\pm$0.36|64.65$\\pm$1.13|24.21$\\pm$2.69|**20.61$\\pm$1.14**|24.80$\\pm$1.08|32.44$\\pm$19.83|\n||+AC|**50.04$\\pm$1.18**|45.51$\\pm$0.66|**10.00$\\pm$0.47**|**26.32$\\pm$1.15**|**9.26$\\pm$4.00**|**3.68$\\pm$4.05**|**40.31$\\pm$0.84**|**48.04$\\pm$1.34**|**19.95$\\pm$0.57**|**20.61$\\pm$0.96**|**19.95$\\pm$1.12**|**26.70$\\pm$15.90**|\n|CIFAR100x10|Base|70.35$\\pm$2.42|49.57$\\pm$1.25|14.11$\\pm$1.08|40.49$\\pm$1.55|9.31$\\pm$2.43|11.88$\\pm$4.27|51.77$\\pm$1.11|68.40$\\pm$2.74|**24.26$\\pm$2.12**|**18.49$\\pm$1.50**|21.44$\\pm$2.74|34.55$\\pm$21.51|\n||+AC|**56.40$\\pm$2.97**|**48.03$\\pm$1.05**|**10.68$\\pm$1.44**|**30.60$\\pm$1.46**|**6.37$\\pm$3.81**|**8.39$\\pm$5.10**|**46.62$\\pm$1.33**|**54.08$\\pm$2.30**|29.12$\\pm$1.01|20.24$\\pm$2.12|**16.71$\\pm$1.96**|**29.75$\\pm$17.96**|\n\nAssuming we train on total of $T$ tasks, we compute forgetting as $\\frac{1}{T-1} \\sum_{t=1}^{T-1} (Acc_{t}^{t} - Acc_{t}^{T})$, where $Acc_{i}^{j}$ refers to the accuracy on the i-th task after training on j-th task (we skip T-th task, as it has zero forgetting according to this definition). We will add these results to the paper. \n\n> One comparison that could make it stronger is the comparison against shallow architecture (with perhaps wider layers).\n\nThank you for the suggestion. To address this, we evaluated our method using WideResNet16 [1] on CIFAR100, which is shallower than the ResNet-32, but twice wider. Consistent with previous results, the AC-enhanced methods outperform the baselines.\n\n|||FT|FT+Ex|GDumb|ANCL|BiC|DER++|ER|EWC|LwF|LODE|SSIL|Avg|\n|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|\n|CIFAR100x5|Base|19.40$\\pm$0.30|38.50$\\pm$0.64|19.17$\\pm$0.95|44.18$\\pm$0.86|50.73$\\pm$1.40|34.88$\\pm$7.28|33.56$\\pm$0.66|19.09$\\pm$0.17|43.17$\\pm$0.35|43.72$\\pm$0.85|48.95$\\pm$0.43|35.94$\\pm$0.62|\n||+AC|**29.79$\\pm$1.94**|**40.67$\\pm$0.37**|**24.09$\\pm$1.16**|**46.09$\\pm$0.75**|**53.38$\\pm$0.68**|**46.02$\\pm$1.08**|**39.61$\\pm$0.64**|**32.58$\\pm$0.37**|**44.26$\\pm$0.02**|**50.83$\\pm$0.30**|**51.35$\\pm$0.53**|**41.70$\\pm$0.23**|\n|CIFAR100x10|Base|9.13$\\pm$0.74|34.39$\\pm$0.80|21.94$\\pm$0.26|33.32$\\pm$0.50|46.02$\\pm$0.29|34.28$\\pm$3.80|30.56$\\pm$0.50|10.10$\\pm$0.35|32.59$\\pm$1.05|40.22$\\pm$0.27|45.58$\\pm$0.28|30.74$\\pm$0.52|\n||+AC|**16.77$\\pm$1.51**|**37.92$\\pm$0.12**|**27.81$\\pm$0.46**|**35.28$\\pm$0.34**|**48.79$\\pm$0.32**|**43.57$\\pm$0.63**|**36.32$\\pm$0.74**|**21.37$\\pm$1.90**|**34.05$\\pm$0.94**|**46.28$\\pm$0.20**|**47.01$\\pm$0.66**|**35.92$\\pm$0.24**|\n\n> Since the performance relies on earlier layers, reporting the achieved accuracy of task t at time t is useful.\n\nThank you for this suggestion. For the Reviewer’s convenience, we plotted such accuracies ([link](https://imgur.com/a/0O3btHy)). We will add those plots to the paper appendix.\n\n> CKA is mentioned in Section 1 without definition. Figure 2 is not clear (...).\n\nThank you for the feedback. While CKA is a well-established metric in the literature, we acknowledge that adding a brief definition will improve clarity, and we will include it as suggested.\n\n> Do you think that these findings generalize to LLMs? And where to place ACs in that case?\n\nPrior work, such as [2], explores the usae of early exits in LLMs, suggesting that our approach could, in principle, be applicable in that context. The placement and architecture of ACs in LLMs could follow a similar strategy to what we describe for ViTs (see Appendix A). \n\nHowever, we are cautious about making direct claims regarding the transferability of our findings to LLMs. Our work explores class-incremental learning in vision, which differs fundamentally from language modeling, where tokenization constrains the class space. ACs might be applied to domain-incremental scenarios in LLMs, but investigating this requires further research beyond our current scope.\n\n**References**:\n\n[1] Zagoruyko et al., *“Wide Residual Networks”*, BMVC 2016\n\n[2] Schuster et al., *“Confident Adaptive Language Modeling”*, NeurIPS 2022", "claims_and_evidence": "Yes, the paper provides empirical evidence and analyses for claims. The method is combined with various continual learning techniques and evaluated on two benchmarks with different numbers of tasks. Different architectures are also analyzed.", "methods_and_evaluation_criteria": "Yes, though since the method focuses on forgetting, I expect to see a metric to measure it in the experimental results. However, only the final accuracy is reported.", "theoretical_claims": "The paper doesn’t include theoretical claims.", "experimental_designs_or_analyses": "Yes. The experimental design is reasonable. One comparison that could make it stronger is the comparison against shallow architecture (with perhaps wider layers), as it is also less prone to forgetting.", "relation_to_broader_scientific_literature": "Up to my knowledge, exploring the idea of early exit classifiers in the continual learning paradigm is novel, but exists in other paradigms as acknowledged in the paper.", "essential_references_not_discussed": "Up to my knowledge, the paper covers the essential references.", "other_strengths_and_weaknesses": "Strengths:\n- The method is simple, yet effective\n- Extensive experiments are provided \n- Generalization across different architectures is analyzed \n- Paper is well-written \n\nWeakness: \n- Forgetting metric is missing in the results\n- Since the performance relies on earlier layers, reporting the achieved accuracy of task t at time t is useful.", "other_comments_or_suggestions": "- CKA is mentioned in Section 1 without definition.\n- Figure 2 is not clear until one reads the details in Section 3.1 (despite been discussed in Section 1).", "questions_for_authors": "1. Have you compared against shallow networks?\n2. Can you report the forgetting?\n3. Do you think that these findings generalize to LLMs? And where to place ACs in that case?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HmDMt8hPTZ", "reviewer": "Reviewer_urVj", "summary": "The paper proposes a new method for continual learning, which trains auxiliary classifiers on top of the intermediate features of the deep neural network along with the main neural network. The method is motivated by the observation that, in the process of class incremental continual learning, intermediate features change less drastically than the final layer(s), and some of the auxiliary classifiers can achieve better performance than the final classifier. Results show that auxiliary classifiers can improve the performance of a number of competitive continual learning approaches, including replay, regularization, and architectural methods. With dynamic layer skipping, the ACs can also enable cheaper inference cost while keeping the same continual learning performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the time spent on our work and the suggestions for improving its quality. Below, we address the points raised by the Reviewer.\n\n> If we just care about the optimal final accuracy (the point at 100% inference cost), the amount of improvement provided by auxiliary classifiers seems limited for more recent methods, especially considering the additional training overhead of these auxiliary classifiers.\n\nWe do not consider ACs a standalone method, but rather a addon-method that enhances existing approaches. Our experiments demonstrate that ACs enable inference cost savings up to while providing moderate performance improvements, even for newer methods such as LODE, where we both improve the performance and save up to 20% computation without any performance degradation, or save 50% compute while keeping the same performance as the baseline (see Figure 1). \n\n> The observation that intermediate features are more stable in continual learning methods is not a novel one, as mentioned in the paper in lines 186-189.\n\nAs the Reviewer notes, we acknowledge prior work on the stability of intermediate features and we treat it as a motivation for our approach. To our knowledge, the idea of leveraging intermediate features to enhance both efficiency and performance in continual learning has not been explored before. Our work builds upon the known observations and introduces a novel approach to utilize it effectively.\n\n> The Riemer et al., 2018 paper is referred to as \"ER\" in the paper, which is a little bit confusing. (...)I would like the authors to confirm that the \"ER\" columns in the tables actually refer to the method proposed in the Riemer et al., 2018 paper.\n\nWe thank the reviewer for pointing out this inaccuracy. The \"ER\" in our work refers to Experience Replay as described in [1], and we will update the paper to make this clear.\n\nTo clarify the distinction between FT+Ex and ER, FT+Ex represents the most naive form of replay, where stored data is simply added to the training dataset for each task without modifying batching or sampling procedures. In contrast, ER ensures that each batch maintains a balanced number of samples from both old and new classes and involves oversampling from the memory. Therefore, ER is a more sophisticated replay method.\n\n>  I have some difficulty understanding what figure 4a is plotting exactly (\"we plot overthinking normalized by the accuracy of the final classifier\" seems a bit unclear). (...)\n\nWe appreciate the Reviewer’s feedback. Given the oracle accuracy $Acc_{oracle}$ and final classifier accuracy $Acc_{final}$, normalized accuracy in Figure 4a is simply $\\frac{Acc_{oracle} - Acc_{final}}{Acc_{final}}$. We will clarify this in the updated version of the paper.\n\n> In figure 2, the earlier layers (such as AC1) changes least during the process of continual learning, while AC6 changes most (closet to the final layer). However, in figure 3, we notice that AC6 provides most performance improvement for a majority of the tasks in non-trivial CL methods. Is there an intuition on why this is the case? (...)\n\nThank you for the question. Early feature stability for older tasks is a motivation behind our approach, but it does not conflict with the better discriminative abilities of the classifiers attached to the later layers, especially on the new data. Figure 2 demonstrates classifier performance in isolation, thus later classifiers will exhibit higher downstream accuracy. Our approach leverages both of those properties to enhance overall performance in CL. Note how with different ACs we learn to classify the samples based on diverse features (e.g. more primitive features in early layers, and more sophisticated features in the later ones) and this improves the robustness of our method (as evidenced by our analysis of overthinking). \n\nWe will elaborate this in more detail in the updated version of the paper.\n\n> Is there any justification on using the confidence score to choose which auxiliary classifier to use for the final prediction (...), especially given that neural networks are often not very well calibrated?\n\nOur choice of the dynamic inference aggregation rule was primarily empirical. Methods that combine multiple predictions (e.g., voting, weighted ensembling) introduce additional complexity without yielding clear improvements (see our analysis of cascading and ensembling classifiers in Appendix C.2). Therefore, we opted for the simpler approach of using a single classifier’s output. We conducted an ablation study comparing the use of the final classifier prediction versus selecting the prediction with the highest confidence (Appendix E.5), and opted to use the best-performing method. See also response to the Reviewer FvJb about thresholding strategies.\n\n**References**\n\n[1] Chaudhry et al., *\"Continual learning with tiny episodic memories.\"*, Workshop on Multi-Task and Lifelong Reinforcement Learning (2019).", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and/or evaluation criteria make sense for the problem at hand. CIFAR 100x10 and ImageNet 100x10 are popular and standard evaluation benchmarks for class incremental continual learning in computer vision and classification. The baseline methods that the authors choose are good representatives of different types of continual learning algorithms.", "theoretical_claims": "N/A. The paper does not include any proofs for theoretical claims in the main text.", "experimental_designs_or_analyses": "The experimental designs and analysis in the paper are sound and valid. The main experiment shown in table 1 demonstrate the effectiveness of the auxiliary classifiers across a range of different CL methods. The additional experiments shows in table 2 and figure 7 demonstrate the effectiveness of the auxiliary classifiers across different neural network architectures.", "relation_to_broader_scientific_literature": "The proposed method is highly related to previous works on overthinking in neural networks and early-exit classifiers for cheaper inference cost in neural networks. This paper studies these topic in the context of continual learning. The observations that overthinking is more apparent in continual learning motivates the auxiliary classifier method. The early-exit classifiers motivates the dynamic layer skipping approach proposed in the paper for inference cost - accuracy tradeoff.\n\nThe paper is also highly related to previous works on using intermediate features in continual learning. The prior finding that intermediate features are more stable motivates the auxiliary classifier method.", "essential_references_not_discussed": "The related works are sufficiently discussed in the paper.", "other_strengths_and_weaknesses": "Strengths;\n- The proposed method is well motivated, and draws inspirations from prior works in continual learning, neural network overthinking, and early-exit classifiers. It provides a natural tradeoff between final accuracy and inference cost.\n- The analysis on overthinking of continually-learned models is a novel contribution.\n\nWeaknesses:\n- If we just care about the optimal final accuracy (the point at 100% inference cost), the amount of improvement provided by auxiliary classifiers seems limited for more recent methods, especially considering the additional training overhead of these auxiliary classifiers.\n- (minor) The observation that intermediate features are more stable in continual learning methods is not a novel one, as mentioned in the paper in lines 186-189.", "other_comments_or_suggestions": "1. The Riemer et al., 2018 paper is referred to as \"ER\" in the paper, which is a little bit confusing. I believe \"ER\" usually refers to naive Experience Replay (the FT+Ex setting in this paper), while the Riemer et al., 2018 paper is referred to as MER (Meta Experience Replay). In the provided anonymous code base the \"er.py\" file also seems to only contain code for naive experience replay. I would like the authors to confirm that the \"ER\" columns in the tables actually refer to the method proposed in the Riemer et al., 2018 paper.\n2. I have some difficulty understanding what figure 4a is plotting exactly (\"we plot overthinking normalized by the accuracy of the final classifier\" seems a bit unclear). I would like the authors to more clearly define how the numbers plotted in Figure 4a is calculated.", "questions_for_authors": "1. In figure 2, the earlier layers (such as AC1) changes least during the process of continual learning, while AC6 changes most (closet to the final layer). However, in figure 3, we notice that AC6 provides most performance improvement for a majority of the tasks in non-trivial CL methods. Is there an intuition on why this is the case? This seems to contradict with the motivation provided in section 3.1 that the intermediate feature are more useful for classification in CL because they are more stable.\n2. Is there any justification on using the confidence score to choose which auxiliary classifier to use for the final prediction (instead of say a popular vote of all the auxiliary classifiers), especially given that neural networks are often not very well calibrated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper found that intermediate layers' representation is less prone to forgetting, leading to better performance. The paper empirically shows that the intermediate layers' representation is effective for less forgetting by proposing an auxiliary classifiers that use the intermediate layers' representations. All reviewers appreciate the benefit of the proposed method. Most of reviewers discuss with the authors with the rebuttal. Agreeing to the reviewers' suggestion, the AC recommends to accept the paper to ICML 2025.
Decision: Accept (poster)