Auxiliary Classifiers Improve Stability and Efficiency in Continual Learning

Authors: Filip Szatkowski, Fei Yang, Tomasz Trzcinski, Bartłomiej Twardowski, Joost van de Weijer
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: continual learning, class incremental learning, auxiliary classifiers

Abstract:
Continual learning is crucial for applications in dynamic environments, where machine learning models must adapt to changing data distributions while retaining knowledge of previous tasks. Despite significant advancements, catastrophic forgetting — where performance on earlier tasks degrades as new information is learned — remains a key challenge. In this work, we investigate the stability of intermediate neural network layers during continual learning and explore how auxiliary classifiers (ACs) can leverage this stability to improve performance. We show that early network layers remain more stable during learning, particularly for older tasks, and that ACs applied to these layers can outperform standard classifiers on past tasks. By integrating ACs into several continual learning algorithms, we demonstrate consistent and significant performance improvements on standard benchmarks. Additionally, we explore dynamic inference, showing that AC-augmented continual learning methods can reduce computational costs by up to 60\% while maintaining or exceeding the accuracy of standard methods. Our findings suggest that ACs offer a promising avenue for enhancing continual learning models, providing both improved performance and the ability to adapt the network computation in environments where such flexibility might be required.

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729497639037

thread_type:
official_review

<thread_object>
creator: reviewer_jvvk

<content>
summary:
This paper investigated the stability of intermediate neural network layers during continual learning, where early network layers tend to be more stable. The authors then proposed to integrate auxiliary classifiers (ACs) into intermediate layers and ensemble them for improving continual learning. The authors then provided extensive experiments to demonstrate the effectiveness of the proposed ACs.

strengths:
1. This paper is essentially well-organized and easy to follow.

2. The proposed ACs seem to be easy to implement and provide significant improvements over a range of continual learning baselines.

3. The proposed ACs may also reduce the computation through dynamic inference.

weaknesses:
1. The authors claimed that “no work has yet explored the use of intermediate classifiers in the continual learning setting”. However, there are at least two papers focusing on using multiple ACs in continual learning. [1] proposed to use multiple side classifiers on the top of regularization-based methods. [2] added multiple ACs to the intermediate outputs and integrated their outputs for online continual learning.

2. The entire work is essentially based on the observations that the intermediate outputs behave differently and may outperform the final outputs in some cases. Is it possible to provide some mechanistic explanation for this phenomenon? Also, the advantages of intermediate outputs in unique accuracy (Figure 3) seem to be marginal for continual learning baselines. I'm not sure this is the main reason for the improved performance of the ACs.

3. The authors claimed that the dynamic inference can reduce the computation. Does this mean training costs and/or testing costs? From my understanding, the proposed ACs still need to train the entire model while skip some layers for inference.

4. The experiments are mainly performed with ResNet-based architectures. Do the proposed ACs also apply to the intermediate outputs of transformer-based architectures?

[1] More classifiers, less forgetting: A generic multi-classifier paradigm for incremental learning. ECCV 2020.

[2] Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation. CVPR 2024.

questions:
Please refer to the Weaknesses.

--------------------------

I agree that this paper is essentially of borderline quality in terms of novelty and empirical contribution.

**Novelty**: This paper is based on the empirical observations that the earlier layers tend to be more stable in continual learning, which is not surprising because the earlier layers often capture more general features that are potentially shared by all tasks. Inspired by this, the authors then employ auxiliary classifiers (ACs) to improve offline class-incremental learning. As acknowledged by the authors, the idea of ACs is borrowed from the early-exit model, a widely used strategy in improving performance and efficiency of deep neural networks. Some advantageous properties of this work, such as reducing inference time, stem from the early-exit model rather than the authors’ own contribution. Also, I have provided two papers in continual learning that implement similar ACs in a parallel or sequential manner, respectively. The authors’ clarification of their differences, such as different continual learning setting, cannot convince me that the use of ACs in this work is completely novel.

**Empirical contribution**: I acknowledge that the authors have provided a lot of experiments, but increasing the amount of experimental results does not necessarily add to the quality of this work. This work is essentially based on the empirical connections between layer-wise stability and continual learning performance. It does not have theoretical basis to ensure applicability, which is a limitation, although I think it’s not a big problem for such a borderline paper. However, all analyses are limited to offline class-incremental learning and training from scratch (i.e., one of the most basic continual learning setting). All continual learning methods achieve very limited performance (e.g., less than 40% on the simple CIFAR100) and the benefits of ACs are remarkably more significant on the simplest FT, which further add the concerns of the applicability of this intuitive idea in realistic continual learning scenarios. Although the authors provide a “pre-train” state of ResNet and also provide ViT results with training from scratch, the overall performance is even worse. Further considering the limited continual learning scenarios and the extra parameter costs of the ACs, I think such empirical contribution is not significant enough.

In summary, I understand that the authors did a lot of experiments. However, “doing more experiments” does not necessarily mean that the quality is improved. These efforts only help to improve the understanding of the work. I think this work has been clearly demonstrated, but it’s novelty and empirical contribution remain (slightly lower than) a borderline quality.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730559778670

thread_type:
official_review

<thread_object>
creator: reviewer_zon8

<content>
summary:
This paper investigates the stability of intermediate neural network layers and addresses the catastrophic forgetting problem in continual learning (CL) by utilizing features from these layers to train auxiliary classifiers (ACs). The proposed approach is novel and aims to enhance the robustness of existing CL methods.

strengths:
**Originality:** The focus on leveraging intermediate layer features to train ACs as a means to combat catastrophic forgetting is an innovative contribution to the field.  
**Quality:** The experimental results demonstrate that the proposed ACs significantly improves the performance of current CL methods, validating the effectiveness of the approach.  
**Clarity:** The paper is well-organized and easy to follow.

weaknesses:
1. The paper lacks a detailed analysis of time complexity and computational overhead. Specifically, how much additional time and memory are required for training and inference with the introduced ACs? This is a significant concern, as the practicality of the proposed method may be limited by increased resource requirements.  
2. The description of how to train the ACs is unclear. Are the same strategies used for training all classifiers? What is the architecture of each classifier?  
3. The choice of static inference, where the classifier with the maximum probability is selected, lacks further analysis and justification. More explanation is needed on this decision-making process.  
4. In Figure 5, what does the x-axis labeled "cost" represent? Additionally, what value of $\lambda$ was used in the reported results for dynamic inference?

questions:
1. What is the distribution of the final selected classifiers during inference? 
2. The paper observes only six intermediate layers; it would be interesting to know if similar results apply to other layers as well.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660484516

thread_type:
official_review

<thread_object>
creator: reviewer_cj85

<content>
summary:
This paper aims to target catastrophic forgetting in continual learning as the problem statement. They
Introduce auxiliary classifiers (ACs) as a mechanism to improve performance in continual learning. The study provides analysis using linear probes and then proposes adding classifiers to intermediate layers, leveraging the fact that earlier layers of neural networks exhibit more stability. The results are shown with different Methods, naive fine-tuning , replay-based and regularizer based CL methods.

strengths:
- Catastrophic forgetting is a key challenge in continual learning and this paper aims to address this critical issue
- The use of linear probing to assess accuracy at different network layers is interesting and offers insights
- The paper is well-organized and generally easy to follow

weaknesses:
- The paper’s objective is bit ambiguous. It’s unclear whether the goal is to fully mitigate catastrophic forgetting or simply to offer additional accuracy sources through auxiliary classifiers. Because, forgetting still occurs, with the method seemingly redistributing accuracy rather than eliminating forgetting. This distinction needs clarification, particularly around Line 190, where the claim that the method is "less prone to forgetting" may need more evidence.

- Previous studies have already shown that early layers capture more generic features, while later layers capture task-specific semantics, so just early layers alone are often insufficient for reliable predictions. Further, though the paper incorporates auxiliary classifiers across layers, this approach introduces computational overhead. The lack of consistent patterns in the ablation studies also leaves it unclear how to optimally position these classifiers for a more efficient solution.

- The motivation to introduce auxiliary classifiers (ACs) stems from empirical analysis, but the results show inconsistent patterns across different continual learning methods. For instance, in replay-based methods, weights remain relatively stable even without ACs, suggesting that the benefits of ACs may not be as universal as claimed. This raises the question of whether adding classifiers could be unnecessary overhead for certain methods.

- LP works on frozen networks, however the hypothesis in Line 253, aims to train all classifiers, and the criteria changes. Training multiple classifiers concurrently may impact the final classifier's performance by diluting its specificity and potentially reducing network plasticity. Hence the training and the final classifier accuracy and the patterns learnt to make the prediction, can get affected ?

- Empirical analysis could be more detailed. There’s limited discussion on the scalability of this method to larger networks or more extended task sequences. The claim of reduced forgetting (Line 190) would benefit from testing on longer task sequences (>10)  and more complex (deeper) architectures. Also does the phase of training play a part, during initial epochs vs near the end of the final epochs for a task? 

- Other accuracy criteria such as stability and plasticity or forward/backward transfer is not provided which are important for assessing the method's full impact on continual learning.

- Will this work when classes overlap, say in domain incremental learning?

questions:
- Figure 1 is not clear, the colors blend together. In general few figures need improvement.
- Can you explain LP analysis? The classifiers at each layer are trained after the whole network is trained on all tasks and frozen?
- Line 187, is this claim correct? There are no analysis for longer tasks (more than 10)
- Can we visualize a pattern of which classifiers are being used? With multiple ACs, how is the final classifier’s predictive power affected?  Could this architecture reduce overall network plasticity?
- Line 471 -  The lack of a clear impact from varying AC numbers and positioning is surprising. This makes it difficult to form a clear intuition about the impact. Thoughts on this ablation?
- While replay and regularization methods are considered in results, parameter isolation methods such as PNN.. are not considered. Also, such as DER ++ (logit replay) are not considered?
- Line 283 - was any other criterion tried before choosing maximum confidence? 
- How is threshold calculated for dynamic inference? Does it depend on arch or complexity of data or tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732060599807

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary response

comment:
We thank all the Reviwers for the time spent on our work and their comments that helped us improve the paper. 

We updated the paper according to the Reviewers' suggestions (changes marked with olive color) and provided additional results such as experiments with longer (20 and 50) task sequences, deeper architectures (VGG19), and additional baseline in DER++. In addition, wherever suitable, we tried to point the Reviewers to sections in the Appendix of the original version of the paper that were initially overlooked by them (e.g. experimental details and results for Vision Transformers). We hope our answers can improve the Reviewers’ opinion about the robustness and quality of our work. 

We uploaded an updated paper and appendix to OpenReview. For ease of discussion, we add all the new results to the end of the appendices during the discussion phase, but we are open to reorganizing the sections in the paper later following the Reviewers' advice.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732725309313

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of discussion and changes to the paper during the rebuttal so far

comment:
We have finalized the revisions to our work during the rebuttal phase, which improved the quality of our submission. We have provided thorough responses to each Reviewer and addressed the raised concerns. However, despite our efforts, we are disappointed by the Reviewers' unwillingness to engage in the discussion and re-evaluate the scores for our submission after improvements, especially given its initial borderline rating. 

Below, we summarize all the content added during the rebuttal for the reviewers and readers:
* We revised the structure of the paper to improve its clarity and flow.
* We included additional results demonstrating the robustness of our method on longer task sequences.
* We added evaluation for a deeper convolutional network VGG19.
* We incorporated an additional baseline comparison with DER++.
* We moved the results with ViT models requested by the Reviewers from the appendix to the main paper. We highlight that these results were already present in the appendix of the original submission.
* We provided further details on the training and inference processes for the models, addressing Reviewers' queries.
* We added classifier selection statistics from our experiments and provided additional ablation on the classifier placement.
* We slightly updated the related works section to include the works highlighted by the Reviewers.

In a separate comment below, we also address the common concerns expressed by the Reviewers. We kindly ask the Reviewers to consider reevaluating our paperwork and engage in a more thoughtful discussion, as we feel our effort invested in both the paper and the rebuttal warrants a fair and respectful response.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733300295597

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We are deeply disappointed with the quality of the reviews and the discussion phase at ICLR 2025. The reviews we have received were very low-effort and overlooked a lot of our work, and despite all the efforts we made during the rebuttal the reviewers refused to engage with us in any meaningful discussion. This was especially disheartening as our work was initially rated borderline, with reviewers posing numerous questions we thoroughly addressed. Although the conference extended the discussion period by a week, this additional time led to no engagement from the reviewers, leaving us frustrated and disheartened. Such disregard for any meaningful dialogue feels deeply disrespectful to the effort we invested in responding to the reviewers — if discussion is not taken seriously by the reviewers, why pose all those questions in the first place? Our experience at ICLR 2025 raises serious concerns about the fairness and rigor of the review process, which falls far below the standards expected of a such conference.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes to utilize Auxiliary Classifiers (AC) to benefit continual learning, particularly offline class-incremental learning. The proposed method is simple -- attach AC to the intermediate layers so that more stable generic features could contribute to improved continual learning performance. The empirical results show the benefits of the proposed method. However, despite the positive empirical results, they were mostly done on small scale datasets, CIFAR100 and ImageNet100, which make it less certain about the scalability of the proposed method. Particularly, in modern ML applications with much larger computation and data scale, it would be **necessary to show the benefit of the proposed method on larger scale dataset**, at least at the level of ImageNet100, for offline class incremental learning setting. (Several previous work, e.g., SSIL, already has results on those setting.) Moreover, the gain the proposed method achieves tends to diminish as the base method accuracy increases, so again it is not clear about how much benefit the proposed method will bring provided the increased computational costs. To that end, AC believes the current submission is not sufficient for a publication at ICLR, yet, and the decision is Reject.