Title: Learning Expandable and Adaptable Representations for Continual Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Mixture model

Continual Learning
Abstract: Extant studies predominantly address catastrophic forgetting within a simplified continual learning paradigm, typically confined to a singular data domain. Conversely, real-world applications frequently encompass multiple, evolving data domains, wherein models often struggle to retain many critical past information, thereby leading to performance degradation. This paper addresses this complex scenario by introducing a novel dynamic expansion approach called Learning Expandable and Adaptable Representations (LEAR). This framework orchestrates a collaborative backbone structure, comprising global and local backbones, designed to capture both general and task-specific representations. Leveraging this collaborative backbone, the proposed framework dynamically create a lightweight expert to delineate decision boundaries for each novel task, thereby facilitating the prediction process. To enhance new task learning, we introduce a novel Mutual Information-Based Prediction Alignment approach, which incrementally optimizes the global backbone via a mutual information metric, ensuring consistency in the prediction patterns of historical experts throughout the optimization phase. To mitigate network forgetting, we propose a Kullback–Leibler (KL) Divergence-Based Feature Alignment approach, which employs a probabilistic distance measure to prevent significant shifts in critical local representations. Furthermore, we introduce a novel Hilbert-Schmidt Independence Criterion (HSIC)-Based Collaborative Optimization approach, which encourages the local and global backbones to capture distinct semantic information in a collaborative manner, thereby mitigating information redundancy and enhancing model performance. Moreover, to accelerate new task learning, we propose a novel Expert Selection Mechanism that automatically identifies the most relevant expert based on data characteristics. This selected expert is then utilized to initialize a new expert, thereby fostering positive knowledge transfer. This approach also enables expert selection during the testing phase without requring any task information. Empirical results demonstrate that the proposed framework achieves state-of-the-art performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on addressing catastrophic forgetting on evolving data domains of continual learning, which is multi-domain continual learning. This work introduces dynamic expansion approach comprising global and local backbones. The global backbone is trained using a mutual information metric. To mitigate forgetting, HSIC-based method is used. Also, this framework includes expert selection to identify the most relevant expert according to the input distribution.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.\tThe writing is clear. \n2.\tThe motivation of the global and local backbones seems acceptable.\n\nWeaknesses\n\n1.\tIn Section 3.4, the KL divergence is applied between two feature distributions. I don’t think it is a good idea to use KL divergence as a metric between two feature distributions, because the KL divergence is usually used on predictions (or logits, probabilities). Features are in high dimensions, I don’t know how you get the distributions of the features exactly (Actually I went through the code in the supplementary material, and did not find the specific implementation), but if you just take the softmax on the features to get the probabilities, most of the values would be close to zero. This provides little information. I think this section needs more elaboration.\n2.\tThe results in Table 1 shows that the framework achieve better performance than other methods by a large margin. But as I see the results on the last three rows, I wonder if there are results of another baseline method that does not use any of the three alignments. Because it seems the three alignments are not too significant in terms of achieving the competitive results. This leads to another concern that is the main contribution of the improvement comes from the extra pre-trained backbone, the larger pre-trained backbone or the alignments?\n\nQUESTIONS:\n1. Which pre-trained ViT is used in the proposed framework, does the comparing methods using the same pre-trained model?\n\n2. In Table 3, the number of training parameters of the proposed framework is among the largest, but how does the GPU utilization is the least? The section that refers to Table 3 is only reiterating the superiority of the framework. I don’t see an explanation of this in the text.\n\n3. Also, how would it be that the proposed framework has the almost comparable number of training parameters with DER++ and CLS-ER? From my understanding, the proposed framework only trains the lightweight experts and the last layers of the backbones.\n\n4. On line 175, what is the “concentration operation”? Isn’t it “concatenation”? If it is concatenation, the symbol should be $\\bigoplus$.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LEAR to address multi-domain continual learning that jointly addresses the stability–plasticity trade-off. The approach incorporates a collaborative backbone along with three key components: MIBPA and KLDBFA for maintaining historical knowledge consistency and task-specific feature alignment, and HSICBCO for learning disentangled representations. An expert selection module (ESM) dynamically activates relevant experts to enable scalable network expansion and efficient inference. Experimental results validate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n+ The paper addresses a new and relatively unexplored problem setup.\n+ The proposed method appears to outperform prior methods significantly under the new setting.\n+ The paper is well-structured and includes a detailed empirical study.\n\n### Weaknesses:\n+ Despite the claim of proposing several new techniques, many of the components seem to be straightforward applications of existing techniques in continual learning. For instance:\n   + The use of a dual-branch architecture is a well-established strategy in the continual learning literature.\n   + The proposed LEAR module, which learns task-specific experts or adaptors, closely resembles techniques used to build expandable representations across tasks.\n   + The idea behind MIBPA (maintaining consistency with historical predictions) is a well-known strategy for mitigating forgetting in continual learning. In this work, the authors adopt mutual information as the consistency metric instead of the more common cross-entropy. However, it's unclear whether this choice offers any real advantage. Since mutual information is symmetric and generally less sensitive to low-confidence outputs than cross-entropy, it's not obvious why it would be preferable in this context. A theoretical or empirical comparison between the two would help clarify whether this design decision is justified.\n+ Based on the discussion above, I encourage the authors to clearly delineate which components are truly novel and which are adaptations of known techniques.\n\n+ Equation (7) treats the outputs of the two branches as random variables, but the formulation lacks detail. What assumptions are made about their distributions? Is there a specific probabilistic model behind this formulation?\n+ The task definition needs further clarification. Does each task correspond to a specific domain or dataset? Or, is the number of classes per task fixed? If the former, I would also be interested in how the method performs under the latter setup, which arguably reflects a more practical scenario.\n+ The ablation in Table 1 could be more informative. Instead of removing components in parallel, it would be better to add or remove them step by step. This would make it clearer how each module contributes to the final performance. The current setup doesn’t fully show the relative importance of each part.\n+ Although the authors append code in the supplementary materials, the provided code cannot run in its current version , which undermines reproducibility.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nThe authors have addressed the limitations and potential negative societal impact of their work in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the limitations of existing class-incremental learning (CIL) methods, which often operate within a single domain and fail to reflect real-world scenarios. To overcome this, the authors introduce a new Multi-Domain Continual Learning (MDCL) setting, where each task corresponds to a distinct dataset, introducing both class and domain shift. To tackle MDCL, they propose a collaborative architecture consisting of a global backbone and a local backbone to capture general and task-specific information. To enhance performance and reduce forgetting, the approach integrates several components such as Mutual Information-Based Prediction Alignment (MIBPA), Kullback–Leibler Divergence-Based Feature Alignment (KLDBFA), Hilbert-Schmidt Independence Criterion-Based Collaborative Optimization (HSICCO), and Expert Selection Mechanism (ESM).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The MDCL setting that the paper presents more accurately represents the real-world CL scenario and can introduce a more challenging benchmark for the community to pursue, given many CIL methods already achieve commendable results on a single domain.\n2. The experiments show that many current methods fail to handle MDCL, yet the proposed method shows great performance, surpassing state-of-the-art baselines.\n\nWeaknesses:\n1. The paper is hard to follow due to the overwhelming introduction of multiple components without sufficient context. It would be clearer if the authors provided more in-depth discussion before each component explaining the motivation. For example, for mitigating forgetting, the authors say they introduce KLDBFA—it would have been better to briefly reference options like regularization and distillation. By making this connection to previous work, they could make it easier to understand and then mention why they decided to go with KL. The same approach could have been applied to each component.\n2.  Before the MDCL setting, it would be better to also compare the proposed approach on the standard single-domain CIL setting. The proposed method should also work on a single domain, so having that comparison helps establish the method’s performance, as all literature reports results in that scenario. It would be easier to compare, and it also strengthens the work by showing it can handle a single domain. You could then move to MDCL to show that current methods fail on that setting.\n3. The ablation studies on different model components should be more thorough. There is w/o MI, KL, and HSIC, which show method performance without each of these components. However, ESM is not discussed (there is a visualization in Table 2b, but accuracy without ESM is not given). Additionally, HSIC in Table 2 shows no meaningful improvement—the model is only 0.11% worse when removing it, which is not statistically significant, so given its complexity, its inclusion may not be justified. The results reported in Table 2 are much better than any other baselines, but it is unclear where these improvements originate from. To provide more insight, the authors should first present results for the collaborative architecture with only the global and local backbones, then progressively add a component or a combination of components to demonstrate their contributions and possible interactions. Having the joint fine-tuning results, where the model is fine-tuned in a non-continual setting on all the data, would also serve as a valuable upper bound for comparison.\n\nI would reconsider my evaluation if my concerns regarding the experiments are addressed. I would also like to review the comments from other reviewers.\n\nQUESTIONS:\nQuestions:\n1. Could you clarify what is stored for each task after training, specifically regarding the parts of the local backbone that need to be stored?\n2. What would be the total number of parameters of this pipeline, not just the trainable parameters, but all the parameters that need to be stored, including those from the experts, task-specific covariances, etc.?\n3. Could you please provide the performance of the collaborative architecture with the global and local backbones and without any other components?\n4. Could you also provide the performance on standard single-domain CIL benchmarks for comparison, such as CIFAR-100, CUB, or ImageNet-R with 10-task splits?\n\nLIMITATIONS:\nYes, but a more thorough analysis of the number of parameters could have been provided.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces LEAR, a buffer-free framework for multi-domain continual learning (MDCL). Specifically, to prevent forgetting while adapting to new tasks, LEAR (i) matches old vs new expert logits with a mutual-info loss (MIBPA) for the global side, (ii) constrains drift in the local feature space with a KL-divergence loss (KL-DBFA), and (iii) pushes the two branches to carry different feature representations with an HSIC independence loss. After each task, it stores only the mean vector and covariance matrix of that expert’s transformed features, computed in a fully frozen ViT feature space; when the next task arrives, it picks the closest old expert by Mahalanobis distance and clones its weights to start learning from a good initial point. Results show the proposed LEAR boosts average accuracy and decreases forgetting on three challenging domain-shuffled image benchmarks compared to prior works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This work addresses a more practical and challenging scenario, multi-domain continual learning, as opposed to the class-incremental learning that most prior methods have focused on.\n\n2. The experimental results demonstrate significant accuracy improvements over existing methods.\n\nWeakness:\n1. The quality of the writing needs improvement. The paper makes it difficult to grasp the overall continual learning process, and Figure 1 is hard to understand until reaching the final subsection (Section 3.7) of the Methodology.\n2. While the proposed method achieves notable accuracy gains over previous approaches, the underlying reasons for its success remain unclear. Please refer to the detailed comments under “Questions” below.\n\nQUESTIONS:\n1.My primary concern is that it remains unclear why the proposed method achieves such significant accuracy improvements over prior continual learning approaches.\n\n1.1 Although the authors provide an ablation study examining the three designed regularization terms, it is difficult to discern their individual contributions. In particular, the feature separation term (HSIC) does not appear to yield a meaningful accuracy gain.\n\n1.2 Rather than focusing solely on the regularization terms, it would be more informative to demonstrate the effectiveness of the collaborative backbone design, which is the key contribution of this work.\n\n1.3 It is not well explained why the proposed MIBPA approach mitigates catastrophic forgetting, nor how effectively it accomplishes this. \n\n2.In line 175, what does the \"concentration operation\" stand for? \n\n3.Which layers are frozen in the auxiliary model? Lines 185–186 state that the auxiliary model freezes the final three layers of the global backbone. However, it appears that in the rest of the backbone, all layers except the last three are frozen. Could you clarify this point? Additionally, why are the last three layers chosen specifically, rather than a different number of layers?\n\n4.What is the exact model architecture of the lightweight expert?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}