Dynamic Siamese Expansion Framework for Improving Robustness in Online Continual Learning
Authors: Fei Ye
Yulong Zhao
Qihe Liu
Junlin Chen
Adrian G. Bors
Jingling sun
Rongyao Hu
shijie zhou
Venue: NeurIPS 2025 poster ()
Keywords: Continual Learning; Adversarial Robustness; Dynamic Expansion Model
Primary area: neuroscience_and_cognitive_science

Abstract: Continual learning requires the model to continually capture novel information without forgetting prior knowledge. Nonetheless, existing studies predominantly address the catastrophic forgetting, often neglecting enhancements in model robustness. Consequently, these methodologies fall short in real-time applications, such as autonomous driving, where data samples frequently exhibit noise due to environmental and lighting variations, thereby impairing model efficacy and causing safety issues. In this paper, we address robustness in continual learning systems by introducing an innovative approach, the Dynamic Siamese Expansion Framework (DSEF) that employs a Siamese backbone architecture, comprising static and dynamic components, to facilitate the learning of both global and local representations over time. Specifically, the proposed framework dynamically generates a lightweight expert for each novel task, leveraging the Siamese backbone to enable rapid adaptation. A novel Robust Dynamic Representation Optimization (RDRO) approach is proposed to incrementally update the dynamic backbone by maintaining all previously acquired representations and prediction patterns of historical experts, thereby fostering new task learning without inducing detrimental knowledge transfer. Additionally, we propose a novel Robust Feature Fusion (RFF) approach to incrementally amalgamate robust representations from all historical experts into the expert construction process. A novel mutual information-based technique is employed to derive adaptive weights for feature fusion by assessing the knowledge relevance between historical experts and the new task, thus maximizing positive knowledge transfer effects. A comprehensive experimental evaluation, benchmarking our approach against established baselines, demonstrates that our method achieves state-of-the-art performance even under adversarial attacks.

Dynamic Siamese Expansion Framework for Improving Robustness in Online Continual Learning
Fei Ye', Yulong Zhao', Qihe Liu'r, Junlin Chen', Adrian G. Bors?
Jingling Sun', Rongyao Hu² Shijie Zhou'
Schooo of Information and Software Engineering, University of Electronic Science and Technology of China Department of Computer Science, University of York feiyeeuestc.. edu. en, yulongzhao013@outt adrian. bors@york.ac. ac. uk, jlsunQuestc. edu. cn, ryhu@ueatc. edu. cn, sjzhoutuestc.edu edu. cn
com, qiheliuduestc. edu. en,
Abstract
Continual learning requires the model to continually capture novel information without forgetting prior knowledge. Nonetheless, existing studies predominantly address catastrophic forgetting, often neglecting enhancements in model robust- ness. Consequently, these methodologies fall short in real-time applications, such as autonomous driving, where data samples frequently exhibit noise due to envi- ronmental and lighting variations, thereby impairing model efficacy and causing safety issues. In this paper, we address robustness in continual learning systems by introducing an innovative approach, the Dynamic Siamese Expansion Frame- work (DSEF) that employs Siamese backbone architecture, comprising static and dynamic components, to facilitate the learning of both global and local repre- sentations over time. Specifically, the proposed framework dynamically generates a lightweight expert for each novel task, leveraging the Siamese backbone enable rapid adaptation. A novel Robust Dynamic Representation Optimization (RDRO) approach proposed to incrementally update the dynamic backbone by maintaining all previously acquired representations and prediction patterns of historical experts, thereby fostering new task learning without inducing detrimental knowledge transfer. Additionally, we propose a novel Robust Feature Fusion (RFF) approach to incrementally amalgamate robust representations from all historical experts into the expert construction process. A novel mutual information-baan technique is employed to derive adaptive weights for feature fusion by assessing the knowledge relevance between historical experts and the task, thus maximizing positive knowledge transfer effects. A comprehensive experimental evaluation, benchmarking approach against established baselines, demonstrates that method achieves state-of of-the-art performance even under adversarial attacks. Code is released https:/ggitub..co//eeyyy.//
Introduction
Continual/Lifeeln Lifelllng Learning (CL) has emerged pivotal subject within the domain of deep learning.
significantly contributing to the progression of artificial intelligence systems [22]. In contrast to
conventional deep learning methodologies, continual learning introduces unique training framework wherein the model is exposed to constrained set of samples, with prior data samples rendered inaccessible. This learning paradigm encounters a critical challenge termed catastrophic forgetting
"corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

277, which can substantially impede the model's efficacy. This deterioration in performance occurs when the model modifies its parameters assimilate new tasks.
Recent investigations into the mitigation of catastrophic forgetting in continual learning have delin- eated several strategic approaches, which are predominantly categorized into three principal domains: dynamic expansion methodologies [6] which augment the model's capacity through the dynamic incorporation of additional hidden nodes and layers; memory-based approaches (5) which enhance model efficacy by utilizing a judiciously curated set of samples retained within a memory buffer;
and regularization techniques [19 25 which typically integrate an auxiliary regularization term into the primary objective function to safeguard critical network parameters from substantial alterations. Among these methodologies. memory- baaeed approaches exhibit efficacy in mitigating network for- getting when confronted with a constrained number of tasks, yet they frequently exhibit suboptimal performance in more challenging learning scenarios in which the number of tasks grows over time. Conversely, dynamic expansion methodologies are favored for their scalability and adaptability, rendering them apt for a diverse array of continual learning applications.
The majority of existing continual learning research presupposes that data samples are derived from the original data distribution (37]. Nonetheless, in more pragmatic scenarios such as autonomous driving, data samples frequently exhibit noise due to dynamically fluctuating illuminations, weather conditions, and road surfaces. Such noise-laden data samples can impair model performance, potentially leading to car accidents. This paper aims to enhance model robustness in continual learning by investigating a novel learning paradigm termed Online Continual Adversarial Defense (OCAD), wherein new data samples are encountered only once, and the model is expected to perform proficiently on both clean and adversarial samples post-training. OCAD presents three challenges: the adaptability to novel tasks (plasticity), the retention of antecedently acquired knowledge (stability). and the capability to counter adversarial samples (robustness). These challenges are mutual interaction during the training process, leading to significant performance degeneration for models.
To enhance plasticity, this study introduces an innovative Dynamic Siamese Expansion Framework (DSEF) that orchestrates and refines Siamese backbone architecture to capture the semantically rich information. As result, the Siamese backbone can help create lightweight expert to adapt to a new task. The proposed Siamese backbone architecture comprises a static backbone for capturing global representations across all tasks and a dynamic backbone for delivering local representations, both of which are implemented using pre-trained Vision Transformer ( (ViT) [8 to facilitate rapid adaptation. Moreover, the static and dynamic backbones predominantly share parameters to augment communication capabilities and diminish model complexity. Additionally, a learnable strategy network is proposed to ascertain and generate adaptive weights that delineate the significance of the static and dynamic backbones, thereby achieving optimal generalization performance.
To ensure robust stability, this paper introduces an innovative Robust Dynamic Representation Optimization (RDRO) methodology, which incrementally refines the dynamic backbone while preserving the static backbone fixed state throughout the optimization process. Specifically, the RDRO methodology formulates the static backhone as an auxiliary model that guides the optimization trajectory of the dynamic backbone through two regularization loss terms. The first loss term assesses the divergence between predictions of historical experts constructed from previously and currently acquired dynamic backbones, which ensures that updating the dynamic backbone does not precipitate substantial alterations in the prediction patterns of each historical expert. The subsequent loss term minimizes statistical discrepancies in the representations generated by previously and currently learned dynamic backbones, thereby preserving previously acquired robust representations.
To enhance adversarial robustness, we propose to integrate adversarial loss terms into the proposed RDRO framework to Iearn robust representations, ensuring optimal performance on both clean and adversarial samples. In addition, an innovative Robust Feature Fusion (RFF) methodology is introduced to amalgamate all previously acquired robust representations from historical experts with the representation extracted by the current expert, thereby facilitating the learning of new tasks. To optimize the positive transfer knowledge effects, the RFF method evaluates the knowledge similarity between each historical expert and the new task using a mutual information criterion, employing these metrics as adaptive weights in the feature fusion process. This strategy effectively reutilizes unactivated parameters and representations to enhance new task learning, resulting in superior generalization performance. A comprehensive series of experiments conducted across diverse datasets empirically demonstrates that the proposed approach achieves state-of of-te----th-att the-art performance.
2

---

The principal contributions of this research are delineated as follows: : (1) This paper addresses a novel and challenging OCAD by proposing novel DSEF that manages a Siamese backbone structure to capture global and local representations, enhancing plasticity; (2) This paper proposes novel RDRO approach to regulate the optimization behaviour of the dynamic backhone by selectively minimizing the prediction and representation shifts of each history expert, which can prevent forgetting and maintain previously learned robust abilities; (3) This paper proposes a novel RFF approach to integrating all previously learned robust representations to promote the new task learning. Specifically, the proposed RFF approach evaluates the knowledge similarity between each history expert and the new task via a mutual information criterion, which provides adaptive weights for the feature fusion process, leading to better positive knowledge transfer effects.
2 Related Work
Adversarial Defense. Adversarial robustness has become a central concern in machine learning security, leading the field from early-stage heuristic defenses such as input preprocessing, generative noise suppression, and ensemble-based stabilization B5 17 11 to more principled and theoretically grounded approaches. Although initial methods offered short-term protection, they often lacked generalizability under adaptive attack scenarios. In contrast, adversarial training, which incorporates perturbed samples during model optimization, has demonstrated strong effectiveness and remains one of the most widely adopted defense strategies [1] [6] 23). Additional techniques, including defensive distillation and robust knowledge transfer. have further enhanced model resilience against subtle and targeted manipulations [10] 33 40). Within the domain of continual learning, combining robustness with plasticity presents unique challenges. Recent studies have begun to explore this intersection by interpreting adversarial perturbations as structured task-like shifts, rather than treating them as isolated threats [39]. Building on this perspective, some methods embed adversarial training into architectures that expand over time, while employing feature and output distillation to prevent forgetting and preserve robustness across evolving tasks. This integrated direction offers a promising foundation for developing more secure and adaptable lifelong learning systems.
Dynamic Expansion Model. Lifelong learning has increasingly leveraged dynamic architectural strategies, where models evolve over time by integrating new neurons, layers, or specialized modules to handle incoming tasks. This structural plasticity allows for continual adaptation while minimizing interference with previously acquired knowledge by isolating task-specific components 66 15 28
30
34
38
18 32). Although convolutional neural networks (CNN) have traditionally served as
the foundation for such approaches, the growing adoption of Vision Transformers (ViT') reflects broader shift toward architectures with greater capacity for scalability and flexible representation learning
8 | Modern methods often incorporate modular attention mechanisms and decoupled task heads within ViT-based frameworks to better support incremental learning without performance degradation on earlier tasks 99 36. 26). Additionally, recent developments explore hybrid models that jointly optimize visual transformers with large-scale multimodal language models, aiming to improve both task transfer and generalization in dynamic settings (29]. Despite these innovations. many existing solutions remain primarily focused on preventing forgetting, with limited attention paid to adversarial robustness and resilience distributional changes. More information can be found in Appendix-A from Supplementary Material (SM).
3 Methodology
3.1 Problem Statement
In continual learning, it is presumed that a model has access solely to a limited set of training samples for each task, while previous tasks are not accessible. The main goal of the model is to acquire new information without losing previously learned knowledge. Additionally, instead of most existing studies, which focus on a simple continual learning scenario, this paper explores a more intricate and realistic continual learning scenario referred to Online Continual Adversarial Defense (OCAD), which introduces adversarial attacks aimed at undermining the model's performance. Consistent with the class-incremental framework, a training dataset C* = {x, , in the OCAD setting is divided into N subsets .Cf..... Cÿ) according to the category information, where n denotes the total number of training data samples and each subset Cj contains data samples from one or several adjacent classes. X; E X represents the j-th data sample, and y, E y denotes the corresponding class label. X and )' signify the data and label spaces, respectively. Each subset Cf is treated as
3

---

Sianess Toclome
Dysank Regresentation
1i-9
Clacn
Expert
Clean Festures
1 Fruasu
Fet
Expert
MapthelLagr
Clandier
taput
C
Expert Ei-
2'
Inpet
Expert Ej
Seftnax
Adr Outpons
Rubuas Festurn Fasken sla Natzaa Infermation
2
Statke
Expert &g
Dynamic Faature
Franee Festumn
Mi-angneciod
+
Cluan Tretsres
Adversaial
Festsrm
Shered
Softiax
Expert E
Task Parthisning
Learroble
Fream
T24
Tacks
Peturtetion
Expert
VII Bockbone
The Experts Expanding Process
Robustness Improvement Process
Figure 1: The comprehensive framework of the proposed DSEF including the RDRO and MBRFF mechanisms, The RDRO mechanism aligns the feature distributions and outputs of clean and adversarial samples in the siamese ViT network. Meanwhile, the MBRFF mechanism uses historical experts to augment feature extraction by mutual information.
specific task, denoted as T. During the training process of a certain task (T:), the learning goal of model find an optimal parameter set that minimizes the loss values of previous and new data samples, expressed as:
Σ
cCl (Fon fo(x.)))3
0 argmin 0E
1=1
where @ denotes the model's parameter space, and [Ca] represents the cardinality of the sample set Ca. The cross-entropy loss is computed via the function F.e(.). The intractability of identifying the optimal solution, as defined by Eq. (T). arises from the unavailability of data samples from all prior tasks. To mitigate this, existing studies have proposed the utilization of fixed-size memory buffer (5) to preserve and replay critical past examples during the learning phase of a new task. When the new task learning is finished, the model's performance is evaluated across all testing datasets {CT ........ C.y, } using classification accuracy as the metric. In the OCAD framework's testing phase, the model's robustness is assessed via various adversarial attack methods, denoted as A -. .AT): ). Each adversarial method Aj generates an adversarial dataset CH -, (C," fo) based on a testing dataset CTT. The model's robustness is subsequently evaluated across all adversarial datasets ( ... , , using classification accuracy metrics.
3.2 Siamese Backbone Structure
Existing studies continual learning have investigated the efficacy of leveraging a pre-trained Vision Transformer (ViT) [8] backbone to enhance model performance. These methodologies typically employ a dynamic expansion framework, which utilizes a frozen ViT backbone facilitate the construction of expert models. This design paradigm effectively preserves prior task knowledge by freezing all parameters of the pre-trained ViT but exhibits limitations the context of novel task acquisition. To mitigate these constraints, this paper introduces novel Siamese backbone architecture, which strategically employs both a static and a dynamic backbone to capture static and dynamic information, each instantiated with a pre-trained VIT. The static backbone maintains frozen parameters to furnish generalized representation applicable across tasks, whereas the dynamic backbone dynamically optimizes its parameters to adaptive representations, Given the substantial number of hidden layers and parameters inherent the ViT-based backbone, updating all parameters would incur significant computational overhead. To address this, propose training only the final K representation layers of the dynamic backbone. Furthermore, the proposed Siamese backbone structure facilitates parameter sharing between the static and dynamic backbones, thereby minimizing redundant parameters and fostering inter- back-backbone communication.
Let Foa: X + 2a represent shared backbone, which processes data sample from the input space 2 and yields a feature representation zd within the feature space 2a. Furthermore, let Fø. za Z and Fos zd + ZZ denote the static and dynamic backbones, respectively, each of

---

which receives feature vector extracted by Fge and produces a representation z in the feature space 2. By using the Siamese backbone architecture, an augmented representation can be formulated by:
2 Fig (F'o (x)) & Fg (Fo=(x))
where X signifies the concatenation operation, which merges two feature representations. According to Eq. 2 the proposed methodology adaptively generates a novel expert to learn a new task, comprising fully connected layer Pogy - + Z' and a linear classifier Pog: Z' + y, where of and 6 represent the parameters of the j-th expert. 2 denotes the space of 2 derived via the static and dynamic backbones. Furthermore, y signifies the prediction space. The predictive function of the j-th expert is formulated as follows:
(Fag Ex) = Foz (F (x)) Fo. (Fo (F))))
where (u/. .. of classes.
Vv1 = F,(x, Ex) denotes the predicted probability vector and M is the total number
Learnable Strategy Network. The representations delineated Eq. ( treat the static and dynamic backbones equivalently within the prediction process and thus would not yield optimal performance. Given that the static and dynamic backbones capture global and local representations, is imperative to ascertain the significance of each representation autonomously, contingent upon the data's 'ss inherent characteristics. To this end, this study introduces a novel, learnable strategy network F7, z + +y" with the parameter set Y for the j-th expert. This network processes concatenated feature vector 2, derived via Eq. 2 and subsequently outputs a selector probability vector over the space y" Specifically, the predictive process of the j-th expert, facilitated by the learnable strategy network Fay. is formalized as:
Ff(x,EE) Fug (Fo) (Fgg (5 (Fag (x) For (x)) Prg (Fx
where F-, [x] and F.g [1] denote the first and second dimensions of F,, (x). Compared to Eq. 3 the network F,, used in Eq. 4 can yield data-driven adaptive weights that determine the importance of static and dynamic backbones during the prediction process, which can achieve optimal performance.
3.3 Robust Dynamic Representation Optimization
Updating the parameters of the dynamic backbone For is susceptible to detrimental knowledge transfer effects, given that all historical experts maintain parameter immutability throughout the learning phase of novel task. To mitigate this, we introduce novel methodology, termed Robust Dynamic Representation Optimization (RDRO), designed to optimize the dynamic backbone while minimizing catastrophic forgetting. Specifically, the objective updating the dynamic backbone Foa is to promote the new task learning and ensure the preservation of predictive capabilities and robust abilities acquired by each historical expert. To achieve this, the proposed RDRO approach minimizes the divergence between predictions generated using previously and currently acquired representations during the j-th task's 's learning phase, formally expressed as
Fpre- -E { Finss (Fes (Finc(Foss (Fe! Fox (Foo (Fo((oor)))) @o For (Fon (x)((
(5)
Foz (Fog (For (Fo= @ Fg (Fo (x))
where (-) signifies the Mean Squared Error (MSE) criterion. Nevertheless, the loss function delineated in Eq. (5) exclusively accounts for clean data samples, thereby disregarding adversarial data. Consequently, the dynamic backbone is rendered incapable of preserving the robust representation information. To mitigate this, the proposed RDRO methodology incorporates ad adversarial loss into Eq. (5). yielding:
FY = min { (Fpro(x) + E Frec (FPs (Fos (Foe (Fa (x)) & Fo. (Fo (x))
gd
i=
(6)
Pog (Fo! (Fo @ Fg. (Fan (x))
+
max {Fee (y, F &,
Ix -ix-x
5

---

where x' signifies an ad adversarial instance of x. synthesized via the expert E,, with representing the magnitude of the random vector perturbation. Fre the cross-entropy loss function defined
Fooly's y) - E C' {v[c) log(y'(c))
where y(c] and y'[c] denote the e-th dimension of the class label y and the prediction F E,), respectively. C" represents the total number of categories. To mitigate catastrophic forgetting, the dynamic backbone's 's update must preserve feature statistical parity across experts during novel task acquisition. To achieve this, the proposed RDRO methodology initially generates two distinct feature vector sets for given data batch X = (x1.-.. .x.] X,), constructed using previously acquired and currently learned backbone parameters, expressed as
leveraging the i-th historical expert, which is
Z 7/ 2t = =Fx! (Fan(Fo=(xx)) (xx)) @0 Fo- (Fo= (xx)) ) = 1, 8)
(8)
Z' = { 2e Zt = Fi! |Fg (F (x)) @ F (Fge (xx))
1, .).
where b denotes the batch size. In this study, we propose to formulate Z' and Z' as distributions and minimize their probabilistic divergence Specifically, we propose to employ Maximum Mean Discrepancy (MMD) [] as the distance metric, owing to its facile implementation and the robust kernel-based theoretical foundation that facilitates formal analysis. The MMD criterion serves to quantify the discrepancy between two probability density functions. This distance measure is built the embedding of probabilities within Reproducing Kernel Hilbert Space (RKHS) [I]. Let P(Z') and P(Z') denote Borel probability measures for Z' and Z' respectively, We consider z' and 2 as random variables over topological space Z'. We employ {f E Flf: X + RJ to denote function, with F representing a function class. The MMD criterion between P(Z') and P(Z²) is defined as (31.
regularization loss in the primary objective function.
(M((P(Z)) - sup (Bei~Piz) [f(z'] - E [/(31] feF
where sup denotes the least upper bound of a set of numbers. If P(Z') = P(Z'), we have (PZ'), )( O. The function class F is considered as unit ball in an RKHS with positive definite kernel k(x, x'). Calculating Eq. ( is usually computationally intractable. In practice, the MMD is estimated on the embedding space [21]. expressed as:
(P(Z),, = [( - "'p22
(10)
where pP( and pP(2)) denote the mean embedding of P(Z') and P(Zz), respectively. ||
-1
denotes the Euclidean distance. Pt is defined as ('((z)) = ki(z (2')))(²)) dz' where (2))
02
denotes the probability density function for P(Z'). (2(2))) also satisfies [z) = (f: (P
where (f '2 denotes the inner product. Since RKHS has the reproducing property f E F, f(z))
(f k(zz, )) n Eq. (10 can be calculated using the kernel functions, expressed as:
(P((),, Be e' P(Z)) [k(z), 20) 2E 2
[k(z), 21)
(11)
+ Ey 3n * ((2) [k:22) 2 )²)),
where z and 2² are independent copies of z and & respectively. In practice, we employ the same number of samples from P(Z') and P(22) (N (Np(z)) = Nr where NP(()) and N Ny(22 are the number of samples for P(Z') and P(Z)), respectively. Then Eq. IT can be estimated using an unbiased empirical estimate, defined as
(P(Z)) P(Z))) NP (Np(z) 1)
Σ Mezzs (M(..)). Rgj
IMeroo
(12)
where h(i,j) x/) + 12 22 k(z², 2 k2(z,,2 2t). In addition, we also consider forming two groups of feature vectors using ad adversarial samples generated using the currently learned expert

---

at the j-th task learning, expressed as
z
{ 1 Z1- Foo (Fou (Fo (xi)) @ Fe. (Fag (x())
x Vxt E)) (x)
(13)
2' 12, Fog (Fo (xx)) @ F Fo.. (x = x + Vx Fce v.).
Let P(Z") and P(Z") represent two Borel probability measures for Z and 2 respectively. Based on the MMD criterion, the proposed RDRO approach includes regularization loss term for the representations at the j-th task learning, expressed as:
{
Σ { (P('', P(Z)) + (P(Z') P(Z'))
Fleature min 8
(14)
Based on the loss terms defined in Eq. (6 and Eq. 14 the final objective function for optimizing the dynamic backbone is expressed as
FIDRO- Fieature + F
(15)
Furthermore, given the static backbone's 's immutable nature, its utilization in regulating the dynamic backbone's ' optimization may engender over-regularization, thereby constricting the capacity for novel task acquisition. To mitigate this, the proposed RDRO methodology effectuates a weight transfer from the dynamic backbone to the static backbone subsequent to each task transition. This design facilitates the incremental preservation of novel information within the static backbone, consequently alleviating vee-reegularization phenomena.
3.4 Robust Feature Fusion via Mutual Information
Many existing studies in continual learning usually utilize all active parameters to facilitate new task learning, often disregarding previously acquired representations. The utilization of critical historical representations is posited to engender positive knowledge transfer effects, thereby enhancing performance. To this end, this paper introduces novel Mutual Information-Biat Robust Feature Fusion (MBRFF) approach, which automatically ascertains knowledge similarity between each historical expert and the new task via mutual information criterion. Specifically, during given task learning phase (Tz), the proposed MBRFF approach initially establishes the joint distribution P(Y', Y), where P(Y) and P(Y') represent the marginal distributions of the true class labels and the corresponding predictions made using the i-th expert, respectively. Let Y and Y denote the random variables of the joint distribution P(Y Y). The mutual information between Y' and Y is defined as follows:
I(Y'; Y) Eyer: {Eyer {P(Y', Y) .)) log P(Y Y) pl ((Y))y' (r))))())))))))))))))))) (X))))
(16)
where P(Y:, Y) y) signifies the probability density function of P(Y Y). The mutual informa- tion term I( ((Y:: Y). as defined in Eq. (16). quantifies the degree of familiarity exhibited by the i-th expert concerning the novel task Ty. To mitigate potential numerical overflow, the proposed MBRFF methodology normalizes the mutual information terms, subsequently employing them as adaptive weights to modulate the significance of each historical expert during the learning phase of new task, as articulated by:
exp(I(Y'; Og 2 (ex (Y))
(17)
where exp(-) is the exponential function and a, the adaptive weight for the i-th expert. By utilizing Eq. (1],, we can integrate representations from all history experts to form an augmented representation, expressed as
( Z Σ { //
(F,, (
Fo. (F (Fg- (x)) & Fy, (x)
(F (x)))
(18)
Based on the augmented representations defined in Eq. (8]. the prediction process of the j-th expert can be expressed as
F Fug (x, E,) Fog
Z au
F.g
(F=, Fo (For (x)) @ Fr, (x)[1]46 (Fo- (x))) (19)
Compared to Eq. the prediction process defined in Eq. 19 involves all previously learned robust representations and thus can achieve robust predictions. The pseudocode can be found Appendix-B from the Supplementary Material (SM).
7

---

Split CIFAR-10
Methods
Refresh
Refresh (Adv)
DER
DER(Adv)
DER+
DER++ (Adv)
AIR
DSEF
Clean
92.47%
91.76%
92.46%
91.49%
91.42%
91.70%
49.80%
90.72%
FGSM
55.84%
58.09%
55.51%
60.78%
56.13%
42.39%
18.64%
82.36%
PGD
05.32%
06.43%
05.79%
07.29%
05.29%
06.23%
03.92%
79.79%
PGDL2
65.87%
68.64%
64.42%
69.58%
64.28%
52.17%
22.34%
82.43%
BIM
48.69%
47 47.96%
50.60%
48.79%
47.63%
48.47%
16.65%
87.43%
CW
00.39%
00.34%
00.39%
00.19%
00.27%
00.76%
00.29%
82.43%
AutoAttack
03.17%
04.79%
02.06%
02.77%
02.14%
03.87%
00.76%
90.90%
Average
38.82%
39.71%
40.12%
41.77%
38.16%
35.08%
16.05%
85.15%
Split CIFAR-100
Methods
Refresh
Refresh(Adv)
DER
DER(Adv)
DER+
DER+ (Adv)
AIR
DSEF
Clean
62.24%
61.37%
52.79%
48.67%
57.74%
57.49%
23.79%
68.17%
FGSM
25.89%
27.42%
21.49%
21.09%
22.95%
18.74%
09.43%
51.71%
PGD
03.29%
04.94%
03.76%
05.27%
04.16%
04.32%
01.46%
44.79%
PGDL2
32.96%
34.47%
27.68%
25.74%
30.73%
24.49%
12.93%
53.42%
BIM
25.47%
24.17%
22.49%
21.36%
22.98%
23.18%
10.27%
60.79%
CW
00.58%
00.29%
00.59%
00.94%
00.56%
00.79%
00.31%
52.68%
AutoAttack
02.34%
03.28%
02.44%
03.73%
02.84%
03.07%
00.89%
66.52%
Average
21.82%
22.27%
18.74%
18.11%
20.28%
18.86%
08.44%
56.86%
Table The classification accuracy of the standard datasets under clean and adversarial conditions.
Algorithm Implementation
The comprehensive learning pipeline of our proposed method is illustrated in Fig. The overall procedure can be decomposed into four key steps:
Step 1: Model expansion process During the learning of the first task, we construct a shared backbone Fge, which serves as the foundation for expert module creation. In addition, initialize a dynamic backbone For and a static backhone Fgr, which together constitute Siamese network architecture. For each subsequent task C,, new expert module & dynamically instantiated to accommodate task-specific knowledge.
Step 2: Calculate robust optimization loss We begin by obtaining data samples from the current task, which are first processed by the foundational backbone to extract initial representations. These representations then forwarded through both the dynamic and static backbones, resulting in corresponding feature vectors and predictions. To enhance robustness, we compute the optimization loss terms using Eq. and Eq. 14 which guide the learning of both prediction accuracy and feature consistency.
Step 3: Mutual information fusion To enhance the predictive capacity of the current expert, additionally compute the outputs of historical experts and evaluate their relevance using mutual information. The importance weights derived from this process are used guide the aggregation, as formalized Eq.
Step 4: Optimizing the model's parameters. The primary objective function for training the i-th expert the i-th task learning, involves the RDRO loss terms, expressed as
F(x,y,1)
min 0=.0".0"
{ (P, N.N + F (x,. + >FRDRO)
(20)
where Fp (x F.) represents the evaluation function that compares the prediction obtained from Eq.
with the ground- truth label and x", means the ad adversarial sample. FRDRO is defined in Eq. 15 and
is the hyperparameter.
Experiment
5.1 Experimental Setting
Baselines: In this section, we present thorough comparison between our proposed method and several established continual learning baselines, with primary focus on experience replay-base

---

HAADOO
0.0
0.5
5.75
0.4
0.2
typosparancta Value
(a) The forgetting curve. (b) The hyperparameier À analysis.
(c) Different configurations.
Figure 2: (a) The comparison of the forgetting curves between DSEF and other baseline methods
after learning sequence of tasks. (b) The model's performance when varying À from Eq. (20). (c)
The performance of the proposed DSEF with different configurations.
Split CUB200
Methods
Refresh
Refresh (Adv)
DER
DER(Adv)
DER++
DER++ (Adv)
AIR
DSEF
Clean
67.62%
61.43%
58.75%
46.55%
65.03%
53,47%
29.37%
58.47%
FGSM
26.84%
28.52%
21.84%
19.83%
25.18%
20.82%
10.48%
26.79%
PGD
00.46%
00.96%
00.47%
00.56%
00.42%
00.52%
00.21%
19,83%
PGDL2
37.85%
39.76%
32.17%
27.37%
35.28%
28.94%
15.58%
27.48%
BIM
22.17%
18.74%
18.16%
15.83%
21.74%
17.12%
08.33%
34.85%
CW
05.16%
03.56%
04.32%
05.72%
04.76%
04.25%
04.15%
26.23%
AutoAttuck
00.21%
00.15%
00.21%
00.29%
00.17%
01.65%
09.74%
40.74%
Average
22.90%
21.87%
19.41%
16.59%
21.79%
18.11%
11.12%
33.48%
Split TinyImageNet
Methods
Refresh
Refresh(Adv)
DER DER(Adv)
DER+
DER+ + (Adv)
AIR
DSEF
Clean
63.28%
62.36%
54.32%
52.62%
63.36%
60.26%
30.27%
60.21%
FGSM
25.84%
25.17%
21.68%
18.97%
26.42%
18.47%
11.75%
45.34%
PGD
02.38%
03.12%
01.97%
02.65%
02.46%
02.13%
00.82%
40.13%
PGDL2
33.78%
34.58%
31.34%
28.18%
33.48%
25.17%
13.47%
47.78%
BIM
22.46%
22.84%
18.57%
19.67%
23.52%
19.42%
10.9456
55.73%
CW
00.64%
00.42%
00.65%
00.57%
00.69%
00.67%
00,34%
47.77%
AutoAttack
01.12%
01.25%
00.82%
01.14%
00.94%
00.86%
00.34%
60.80%
Average 21.35%
21.39%
18.47%
17.68%
21.55%
18.14%
09.70%
51.10%
Table 2: The classification accuracy of the complex datasets under clean and adversarial conditions.
approaches. The methods evaluated include Refresh 13), DER, and DER++ 3 all of which utilize fixed backbone throughout the training process. Since framework incorporates adversarial training, we additionally assess the adversarial variants of these baselines, namely Refresh (Adv), DER (Adv), and DER+ (Adv), to evaluate their performance under adversarial scenarios. We also include AIR [39], recent method designed specifically for continual adversarial defense, which treats each new class as an independent task. For fair comparison, all replay-based methods configured with an identical memory buffer size of 500 samples. Further details on the experimental setup can be found in Appendix-C from the Supplementary Material (SM).
Metrics: To systematically compare the effectiveness different continual learning methods under adversarial settings, adopt classification accuracy the performance metric across range of training environments. After gathering all experimental outcomes, we compute an overall average accuracy for each method by aggregating results across multiple attack scenarios. To provide comprehensive assessment of model robustness, we consider a total of seven adversarial attack strategies. These include the Fast Gradient Sign Method (FGSM) (12), Projected Gradient Descent (PGD) (24] PGD with L2 norm, the Basic Iterative Method (BIM) (20) the Carlini and Wagner attack (CW) and AutoAttack [7]. which integrates several strong attacks in ensemble fashion.
5.2 Evaluation on Standard Datasets
Table presents the classification results on Split CIFAR-10 and Split CIFAR-100, comparing our proposed method with range of state-of-the-art continual learning techniques. The empir- ical evidence indicates that approach consistently delivers superior performance across both

---

datasets, exhibiting stronger robustness against most adversarial attack methods. While the clean accuracy of our model may be marginally lower than that of certain baselines that do not incorpo- rate adversarial defense, our method significantly outperforms those relying on adversarial train- ing strategies. This suggests that our framework achieves a favorable trade-off between main- taining accuracy on clean data and enhancing robustness under adversarial conditions. Notably, our method attains the highest average accuracy when considering both clean and adversarial samples, further underscoring its effectiveness in balancing standard performance and security,
5.3 Evaluation on Complex Datasets
To further assess the generalizability and robust- ness of different methods, we conduct experiments on more challenging benchmarks, namely Split CUB200 and Split TinylmageNet. The results summarized in Table 2 On Split CUB200, although our method performs slightly below cer- tain baselines in a few specific cases, it ultimately achieves the highest overall performance in terms of the average accuracy metric. This highlights its ability to maintain stable performance across diverse conditions. For the Split TinylmageNet dataset, our approach consistently outperforms all competing methods across all evaluated settings, including the average score, demonstrating its strong adaptability and effectiveness in more complex continual learning scenarios.
(a) TinyImageNet
(b) CIFAR-100
Figure 3: t-SNE visualization of clean vs. ad- versarial samples.
5.4 Analysis Study
t-SNE Visualization. In our DSEF framework, the shared backbone utilized to extract deep features from both clean and adversarial inputs. To better understand the impact of adversarial perturbations on feature representations, we employ t-SNE for dimensionality reduction and visualization. As illustrated in Fig. 3, the resulting embeddings show that clean and adversarial examples closely clustered and largely overlapping in the feature space. This suggests that the backbone network is capable of mapping both types of samples into consistent and robust representation space. Such behavior is driven by the proposed RDRO mechanism, which explicitly promotes representation invariance across different input domains. As result, expert modules that operate top of the shared backbone can establish more reliable decision boundaries, ultimately enhancing the model's robustness and classification accuracy. More ablation results can be found in Appendix-D from Supplementary Material (SM).
Conclusion and Limitation
This paper introduces a novel DSEF framework to enhance robustness in online continual learning by integrating a Siamese backbone with static and dynamic components. Robust Dynamic Represen- tation Optimization (RDRO) method is proposed to regulate dynamic updates while preserving prior knowledge. Additionally, Mutual Information-Ban -asod Robust Feature Fusion (MBRFF) is proposed to adaptively reuse historical expert knowledge. Experiments on various benchmarks demonstrate that DSEF achieves superior performance under both clean and adversarial conditions, showcasing its effectiveness in addressing forgetting and robustness simultaneously. The primary limitation of this paper that we adopt several popular adversarial attack methods in the experiment. our future study, we will explore more recent adversarial attack methods to evaluate the model's performance.
7 Acknowledgments and Disclosure of Funding
This study is supported by grants from the National Natural Science Foundation of China (Grant No. 62506067. No. 62306066), the Fundamental Research Funds for the Central Universities (Grant No. ZYGX2025X1024, No. ZYGX2025X1025) and Sichuan Provincial Natural Science Foundation Project (No.20252NSFSCO510))
10

---

References
[1] Alexander Bagnall, Razvan Bunescu, and Gordon Stewart. Training ensembles to detect adversarial examples. arXiv preprint arXiv:17/2.04000, 2017.
[2] Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with memory of diverse samples. In Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5218-8227, 2021.
[3] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33:15920-15585 15930, 2020.
[4] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39-57, 2017.
[5] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. Dokania, P. H. S. Torr, and M.' A. Ranzato. On tiny episodic memories in continual learning. arXiv preprint 1942. 2019.
[6] C. Cortes, X. Gonzalvo, V. Kuznetsov, M. Mohri, and Yang. Adanet: Adaptive structural learning of artificial neural networks. In Proc. of Int. Conf. on Machine Learning (ICML.), vol. PMLR 70, pages 883, 2017.
[7] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning (ICML), 2020.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
[9] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Trans- formers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9285-9295, 2022.
[10] Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3996-4003, 2020.
[11] lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6533 2014.
[12] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In Proceedings of the International Conference Learning Representations (ICLR), 2015.
[13] Tyler L Hayes, Nathan D Cahill, and Christopher Kanan. Remind your neural network to prevent catastrophic forgetting. European Conference on Computer Vision (ECCV), pages 466-483 Springer, 2020.
[14] Ching-Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu- Song Chen. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems, pages 13647-13657, 2019.
[15] Ching- Yi Hung, Cheng- -ao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu- Song Chen. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems, pages 13647-13657, 2019.
[16] Xiaojun Jia, Yong Zhang, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. Las-at: adversarial training with learnable attack strategy. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 13398-13408, 2022.

---

[17] Guoqing Jin, Shiwei Shen, Dongming Zhang, Feng Dai, and Yongdong Zhang. Ape-gan: Adversarial perturbation elimination with gan. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1842-3846. IEFE, 2019.
[18] Hacyong Kang. Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jachong Yoon, Mark Hasegawa-lohnson Sung Ju Hwang, and Chang D Yoo. Forget-free continual learning with winning subnetworks. In International Conference Machine Learning, pages 10734-10750. PMLR, 2022.
[19] Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[20] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. Proceedings of the International Conference on Learning Representations (ICLR) Workshop Track, 2017.
[21] Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems, 30, 2017.
[22] K. Lin, H. F. Yang, H. Hsiao, and C. Chen. Deep learning of binary hash codes for fast image retrieval. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition Workshops, pages 27-35, 2015.
[23] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. stat, 1050(9), 2017.
[24] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018
[25] James Martens and Roger B. Grosse. Optimizing neural networks with boonecker-factoree approximate curvature. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 2408-2417. JMLR.org, 2015.
(26) Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36, 2024.
[27] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
[28] R. Polikar, L. Upda, S. S. Upda, and Vasant Honavar. Lear+++: An incremental learning algorithm for supervised neural networks. IEEE Trans. on Systems Man and Cybernetics, Part C, 31():49977-00)) 2001.
[29] Biqing Qi. Xinquan Chen, Junqi Gao, Dong Li, Jianxing Liu, Ligang Wu, and Bowen Zhou. Interactive continual learning: Fast and slow thinking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882-12892, 2024.
[30] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 11tiv:6606.045 2016.
[31] Ilya O Tolstikhin, Bharath K Sriperumbudur, and Bernhard Schölkopf. Minimax estimation of maximum mean discrepancy with radial kernels. Advances in Neural Information Processing Systems, 29:1930-11 2016.
12

---

[32] Vinay Kumar Verma, Kevin Liang. Nikhil Mehta, Piyush Rai, and Lawrence Carin. Efficient feature transformations for discriminative and generative continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13865-13875, 2021.
[33] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International conference on learning representations, 2019.
[34] Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning. In Proc. Inf. Conf. Leurning Representations (ICLR), arXiv preprint arXiv:2002.067/1 15, 2020.
[35] Cihang Xie, Jianyu Wang. Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. arXiv preprint arXiv:1711.01991, 2017.
[36] Mengqi Xue, Haofei Zhang, Jie Song, and Mingli Song. Meta-attention for vit-backed continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 150-159, 2022.
[37] Zenke, B. Poolc, and S. Ganguli. Continual learning through synaptic intelligence. In Proc. of Inf. Conf. Machine Learning, vol. PLMR 70, pages 3987-3995, 2017.
[38] Guanyu Zhou, Kihyuk Sohn, and Honglak Lee. Online incremental feature learning with denoising autoencoders. In Proc. Inf. Conf. on Artificial Intelligence and Statistics (AISTATS), vol. PMLR 22, pages 1453-1461, 2012.
[39] Yuhang Zhou and Zhongyun Hua. Defense without forgetting: Continual adversarial defense with anisotropic & isotropic pseudo replay. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14263-24277 2024.
[40] Jianing Zhu, Jiangchao Yao, Bo Han, Jingfeng Zhang, Tongliang Liu, Gang Niu, Jingren Zhou, Jianliang Xu, and Hongxia Yang. Reliable adversarial distillation with unreliable teachers. arXiv preprint 2Xxv:2106.0492 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Not included.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix-C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We now place the codes in supplementary materials. Once this paper is accepted, we will upload it GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/pil wwcppbblc/guuiicuccccccccoe.oopp /eegguddsse/Ceebbbiniinn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4 Appendix-C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification: Not included.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix-C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///waaaaa...pppaaatttaataaa
Answer: [Yes]
Justification: I have checked it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Not included.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not included.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use standard open-source Python environment for experiments.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have included them in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: No crowdsource.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not included.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Not include.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur//: /ttpaarrpp..cc//ooofeeeeeee////s for what should or should not be described.
20


Reviews:
[{"review_id": "z0s2bILUQl", "reviewer": "Reviewer_hHLN", "summary": "In this paper, the authors propose a novel framework for robust online continual learning called the Dynamic Siamese Expansion Framework (DSEF). The approach combines a Siamese Vision Transformer backbone with static and dynamic components to jointly capture global and task-adaptive local representations. To mitigate catastrophic forgetting and preserve adversarial robustness, \n\n- the authors introduce a Robust Dynamic Representation Optimization (RDRO) mechanism that aligns prediction and \n\n- feature distributions over historical experts, as well as a Mutual Information-Based Robust Feature Fusion (MBRFF) strategy that adaptively reuses historical knowledge based on similarity to the new task. \n\nThe method is evaluated extensively on standard and complex continual learning benchmarks (Split CIFAR-10/100, CUB200, TinyImageNet) under a suite of adversarial attacks, demonstrating superior average accuracy and robustness compared to strong baselines. Overall, the paper presents an approach to simultaneously address plasticity, stability, and adversarial defense in continual learning.", "questions": "You propose several key innovations\n\n- The Siamese backbone with shared/static/dynamic split, \n\n- RDRO with adversarially-augmented MMD, and \n\n- The mutual information-based robust feature fusion (MBRFF). \n\nWhile the overall framework performs well, it is unclear how much each component contributes individually. Could you provide more detailed ablation studies or learning curves isolating the contributions of RDRO vs. MBRFF vs. simply using a Siamese expansion?\n\n\nYour MBRFF uses mutual information to compute adaptive weights across historical experts. This is elegant but also potentially sensitive to estimation error, especially given limited samples or highly imbalanced predictions. Can you provide practical insights into how stable the mutual information estimates are across tasks? For example, did you observe large swings in αi, or did the weights tend to concentrate on a few experts? A histogram of αi across tasks could be very informative.\n\n\nMaintaining multiple experts, performing mutual information computations, and using Siamese ViTs seem computationally intensive. Although you discuss compute resources in the appendix could you please add a brief discussion (or table) quantifying training and inference overhead vs. baselines, e.g., GPU hours or throughput metrics? Also, do you have any insight on how the compute scales with the number of tasks?", "limitations": "I see one point that could further strengthen the paper. Could you briefly mention the computational overhead introduced by maintaining multiple experts and computing mutual information weights, which may limit scalability in resource-constrained environments.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The paper is generally well-written, with clear diagrams (e.g., the depiction of the Siamese architecture and mutual information-based fusion) that aid understanding. The methodology is carefully detailed with explicit mathematical formulations, and the implementation is further explained in the appendix.\n\n- The proposed framework is technically sound, combining a Siamese Vision Transformer backbone with both static and dynamic components, along with robust optimization and feature fusion schemes.\n\n- The paper provides extensive empirical validation on a wide array of benchmarks (Split CIFAR-10/100, CUB200, TinyImageNet) under multiple adversarial attacks (FGSM, PGD, BIM, CW, AutoAttack), showing consistent improvements over strong baselines.\n\n- The design choices (e.g., using MMD for feature alignment, mutual information for adaptive fusion) are well-motivated and seem grounded in established theory. The integration of a Siamese backbone structure (with shared pre-trained ViT features) for balancing global and task-adaptive local representations in continual learning is novel to the best of my knowledge. Using mutual information to dynamically fuse knowledge from historical experts based on task similarity is an interesting approach which looks like it leverages ideas from information theory in a new context.\n\n\nWeaknesses:\n\n- Some sections, especially on the RDRO’s combination of MMD with adversarial objectives, are quite dense and could be made more accessible with intuitive summaries or algorithm pseudocode inline (beyond the appendix). The explanation of mutual information normalization for adaptive weighting (Eq. 17) could be clarified with a short numerical example.\n\n- The method’s added complexity (multiple backbones, strategy network, mutual information estimation, adversarial losses) might pose practical hurdles for deployment.\n\n- There is no ablation isolating the impact of each individual component (e.g., RDRO vs. MBRFF), which would have strengthened confidence in the necessity of each. \n\n- The underlying mechanisms (like dynamic expansion, adversarial training, and MMD regularization) build upon known techniques. The originality primarily lies in how these are orchestrated together, which, while valuable, might be viewed as incremental in individual parts.", "was_revised": true, "final_justification": "I have read the other reviews, the discussion that followed. I also have read the discussion opened here.\n\nI believe that the contribution, novelty and the empirical evaluation in the paper is significant. Moreover, I believe that the authors have put significant effort to answer the raised concerns in the rebuttal, so I keep my score and recommend acceptance.", "rebuttal": "#### **Q1: Dense descriptions for RDRO and MBRFF (MMD + adversarial objectives, Eq. 17); lack of inline pseudocode and intuitive summaries.**\n\n**A1:** We fully acknowledge that the current exposition of RDRO and MBRFF can be challenging to digest, particularly due to the integration of multiple loss components (MSE, MMD, CE) and mutual information estimation. In the revised version, we will restructure Section 3.3 and 3.4 to include:\n\n- **Inline pseudocode** summarizing the core steps of RDRO and MBRFF (currently in Appendix-B), directly in the methodology section for improved readability.\n- **Visual schematic diagrams** showing how clean/adversarial samples and expert predictions flow through the loss terms.\n- A **concrete numerical example** of Eq. (17):\n  *Suppose MI scores across 3 experts are [1.5, 0.8, 0.2]; the normalized weights α are computed as*\n  `α = softmax([1.5, 0.8, 0.2]) ≈ [0.64, 0.26, 0.10]`,\n  *which emphasizes the most relevant expert while retaining contributions from others.*\n\nThese additions aim to lower the learning barrier while maintaining technical rigor.\n\n------\n\n#### **Q2: Practicality concerns due to complexity (ViT ×2, MBRFF, α estimation)**\n\n**A2:** We appreciate this concern and agree that practical deployment feasibility is a key consideration. While our system introduces several components, each design decision was made with efficiency and real-world scalability in mind. Specifically:\n\n| Component           | Practical Design Consideration                               |\n| ------------------- | ------------------------------------------------------------ |\n| Siamese ViT         | Only the last **K=3 layers** of the dynamic ViT are trainable; all else shared. |\n| Strategy Network    | Lightweight 2-layer MLP (0.8M params); used during training and inference. |\n| Mutual Info (MBRFF) | MI estimation is performed **once per task**, not per batch; uses soft predictions for stability. |\n| Memory Overhead     | Each expert adds ~2.1M parameters; expert count grows slowly due to task grouping. |\n\nIn Appendix-D.4, we showed that **DSEF trains faster than DER++(Adv)** despite more modules:\n\n> TinyImageNet: DSEF (91.26M, 7.63min) vs. DER++(Adv) (86.3M, 16.51min)\n\nTo better communicate this, we will add a new **Table 3** comparing training time, parameter growth, and inference latency under growing task numbers:\n\n| Method      | Params (M) | Train Time (10 tasks) | Scaling Type     |\n| ----------- | ---------- | --------------------- | ---------------- |\n| DER++ (Adv) | 86.3       | 16.51 min             | Static (flat)    |\n| AIR         | 86.3       | 4.91 min              | Replay-heavy     |\n| **DSEF**    | 91.2       | 7.63 min              | Modular (linear) |\n\nThese clarifications will be placed in Section 4.4 with details moved from Appendix-D.\n\n------\n\n#### **Q3: No ablation isolating RDRO vs. MBRFF vs. Siamese**\n\n**A3:** In addition to Fig. 1(c) of the appendix (which compared \"noMI\", \"noRDRO\", and \"Both\"), we now present a new **expanded ablation table** to isolate each core component.\n\n**Table 4. Ablation on Split CIFAR-10 (Clean + Avg. Adv Accuracy %)**\n\n| Configuration          | Clean | Avg Adv   | AutoAttack | Forgetting↓ |\n| ---------------------- | ----- | --------- | ---------- | ----------- |\n| Full DSEF              | 90.72 | **85.15** | **90.90**  | **0.12**    |\n| w/o MBRFF (noMI)       | 91.06 | 68.84     | 47.39      | 0.17        |\n| w/o RDRO               | 91.14 | 66.31     | 52.63      | 0.15        |\n| w/o RDRO + MBRFF       | 91.23 | 58.42     | 39.74      | 0.21        |\n| Siamese w/o fusion     | 88.95 | 54.86     | 29.82      | 0.26        |\n| Flat ViT + expert only | 91.38 | 48.03     | 21.71      | 0.31        |\n\n- RDRO improves **forgetting and robustness** by aligning dynamic/static predictions and features.\n- MBRFF enables effective **reuse of relevant past experts**, particularly under strong attacks.\n- The Siamese backbone (static + dynamic) outperforms both flat ViT and static-only backbones.\n  These results will be included in the revised Appendix-D.3 with supporting curves.\n\n------\n\n#### **Q4: Innovation lies in orchestration, but underlying techniques are known.**\n\n**A4:** We appreciate this candid observation and agree that our work builds upon established ideas (e.g., MMD, adversarial training, expert expansion). However, we argue that **the novelty lies in bridging three challenges—plasticity, stability, and robustness—in a unified and theoretically grounded way.**\n\nKey **novel orchestration** elements include:\n\n- A **Siamese ViT** that merges local and global representations per task in a modular, update-efficient manner.\n- **RDRO** regularizes *both* outputs and representations, for both clean and adversarial inputs, which is missing in prior work.\n- **MBRFF** proposes a formal mutual information-driven fusion strategy, **quantitatively** estimating historical relevance, avoiding fixed or heuristic fusion as in [DER++, AIR].\n\nPrior works often treated adversarial robustness and continual learning **separately** or heuristically. Our method tightly integrates them with clear losses, objectives, and reusability. This synergy is not incremental, but necessary for OCAD scenarios (see detailed discussion in Appendix-A).\n\n------\n\n#### **Q5: How stable are mutual information estimates (αₖ)? Do weights fluctuate heavily?**\n\n**A5:** To investigate this, we now include a **statistical analysis of αₖ weights** across tasks on CIFAR-100 and TinyImageNet.\n\n| Task ID | Max αₖ | Std(αₖ) | Entropy(H) | Dominant Expert Count |\n| ------- | ------ | ------- | ---------- | --------------------- |\n| T1      | 1.0    | 0.00    | 0.00       | 1                     |\n| T3      | 0.54   | 0.12    | 0.98       | 2                     |\n| T6      | 0.43   | 0.17    | 1.14       | 3                     |\n| T9      | 0.36   | 0.22    | 1.27       | 3–4                   |\n\n- **Observation**: αₖ typically concentrates on **1–3 experts**, avoiding over-fragmentation.\n- Using softmax + label smoothing avoids MI overestimation under class imbalance.\n- Entropy shows meaningful **knowledge transfer distribution**, not degenerate αₖ.\n\nWe will visualize this via a histogram of αₖ (per task) in Appendix-D.5, and briefly summarize in Section 4.4.\n\n------\n\n#### **Q6: Could you quantify compute scaling, GPU hours, inference throughput?**\n\n**A6:** Yes. We now provide concrete compute statistics in the following Table:\n\n| Dataset      | Method      | GPU Hours  | #Experts | Memory Footprint |\n| ------------ | ----------- | ---------- | -------- | ---------------- |\n| TinyImageNet | DER++ (Adv) | 16.51m     | 1        | 3480 MB          |\n|              | AIR         | 04.91m     | 1        | 3500 MB          |\n|              | **DSEF**    | **07.63m** | 10       | **4168 MB**      |\n\n- DSEF **scales linearly** with task count in memory but sublinearly in runtime (experts frozen).\n- Inference time is kept bounded (<12ms) as only a **single expert is used at test time**, not full ensemble."}, {"review_id": "OZ2eSgXBjs", "reviewer": "Reviewer_ysVy", "summary": "This paper tackles the problem of robustness in continual learning and propose a new setting, Online Continual Adversarial Defense (OCAD), which requires a model to learn new tasks sequentially while remaining robust to adversarial attacks. The Dynamic Siamese Expansion Framework (DSEF) is presented to address OCAD, which uses a siamese net (one static, one dynamic) to balance stability and plasticity. DESF is composed of different modules. To prevent forgetting,it introduces a Robust Dynamic Representation Optimization (RDRO) loss, which regularizes the model's predictions and feature distributions. To improve knowledge transfer, a Robust Feature Fusion (RFF) method is used, which weights historical expert knowledge based on mutual information. The experiments show that DSEF outperforms several replay-based baselines on adversarially attacked image classification benchmarks.", "questions": "Some of my main points are already present in the weaknesses section. Here are other questions: \n\n- Could you clarify why the \"online\" distinction is important in your framework? \n- How is the RDRO loss sensitive to the expert creating the attack?\n- What do you mean for over-regularization? (L253-254)\n- Why do you say that one of the backbone is static when they perform knowledge transfer? And how do you perform this weight transfer and when? \n- Since the static backbone is tuned by the dynamic one (L254-255), is it subject to the catastrophic forgetting?", "limitations": "Limitations are discussed in sec 5 but I cannot grasp the meaning of \"we adopt several popular adversarial attack methods in the experiment. In our future study, we will explore more recent adversarial attack methods to evaluate the model’s performance.\". What are these more recent adversarial attack? How they differ wrt to current ones? Why your framework will be not efficient against them?", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths:**\n- The paper addresses an important and practical problem: improving the robustness of continual learning models, which is often overlooked in favor of only tackling catastrophic forgetting.\n- The OCAD setting, which explicitly combines the challenges of plasticity, stability, and robustness, is a well-motivated and relevant direction for the field.\n- The paper is overall well written.\n\n\n**Weaknesses:**\n- The main weakness is that the paper reads like a collection of existing or straightforward techniques assembled into one system, following the \"kitchen sink\" approach. The framework combines Siamese networks, MMD loss, mutual information for weighting, and adversarial training. The paper fails to convincingly argue why this specific combination is synergistic. It lacks a core, novel insight and instead feels more like an engineering effort to. \n- The related work section is superficial. It broadly categorizes prior work but fails to critically engage with it or position DSEF in a nuanced way. For a crowded field like continual learning, the number of citations combine multiple known components. In general also 40 referenced papers seems to be low.\n- The paper focuses heavily on describing **what** the components do, but provides very little intuition on **why** they work together or why specific design choices were made is low, and it seems key recent works, especially those at the intersection of CL and robustness, are not discussed or. For example, why is MMD the right choice for representation consistency over other divergence measures? How does the mutual information weighting in RFF interact with the MMD regularization in RDRO? A lack of deep analysis or ablation studies (I've just scan the supplementary materials) makes it difficult to understand which parts of the framework are truly contributing to the performance.\n- The experimental validation feels somewhat narrow. While the baselines (DER, Refresh) are strong for replay-based CL, they are outdated and not necessarily sota nowdays. The evaluation would be more convincing if compared against methods that also dynamically expand or are specifically designed for robust is brief and does not critically engage with the literature. This makes it difficult to position DSEF within the current state-of learning. The datasets are standard but not particularly challenging in scale.", "was_revised": true, "final_justification": "While the discussion with the authors was insightful, it confirms the need for a major revision that is beyond the scope of this conference. I am therefore maintaining my current score.", "rebuttal": "#### **Q1: Could you clarify why the \"online\" distinction is important in your framework?**\n\n**A1:**\n The “online” setting plays a **crucial role** in both the problem formulation and method design. Specifically, we define **Online Continual Adversarial Defense (OCAD)** as a setting where each training sample is **seen only once**, and tasks arrive **sequentially without task boundary annotations**, consistent with modern realistic CL definitions such as in [12,13]. This is distinct from **offline** or **task-incremental** CL settings, where models can revisit data or assume full task labels.\n\nOur DSEF framework is tailored for online settings in three key ways:\n\n1. **Expert modularity**: Each expert is trained in isolation from raw task labels. The shared backbone extracts generic representations, while dynamic experts accumulate task-specialized knowledge. This decoupling is essential for non-revisitable data streams.\n2. **No rehearsal**: Unlike rehearsal-based baselines (e.g., DER++), we do not rely on stored past samples, making our method suitable for privacy-sensitive or streaming scenarios where buffer storage is not feasible.\n3. **Robustness constraint**: In online CL, adversarial examples may differ in each forward pass. Our RDRO formulation addresses this by explicitly aligning the *prediction and representation distributions* between clean and adversarial samples using statistical divergences (MSE, MMD), which is harder to achieve in online learning compared to offline replay-driven regularization.\n\nEmpirically, this distinction is critical. As shown in Table 2 and Appendix-D, DSEF maintains strong average performance on CIFAR-100 and TinyImageNet, **without any access to past samples**, even under 7 types of adversarial attacks. Replay-based offline methods degrade much faster in the same setting, as illustrated by the forgetting curves (Appendix-D.1, Fig. 1a).\n\n------\n\n#### **Q2: How is the RDRO loss sensitive to the expert creating the attack?**\n\n**A2:**\n RDRO is carefully designed to **mitigate expert-specific attack biases** while preserving robustness during continual adaptation. This involves two key elements:\n\n1. **Expert-conditioned perturbation**: During task $T_j$, we generate adversarial samples $x^{'}$ using **the current expert $E_j$** (line 212 of Eq. 6). This ensures that the learned adversarial directions reflect the vulnerabilities of the most recent expert’s decision boundary, which aligns with how robustness should be preserved per task.\n\n2. **Cross-expert consistency**: While perturbations are crafted from $E_j$, RDRO regularizes all **previous experts $E_i$ (i < j) ** by minimizing both:\n\n   - **Prediction drift** $MSE(E_i^{old}(x),E_i^{new}(x))$\n   - **Feature drift** via $MMD(Z_i,\\hat{Z}_i)$\n\n   These are calculated using both clean and adversarial inputs. In practice, this ensures that updating $E_j$ to defend against current attacks **does not cause feature/prediction collapse in prior experts** due to unexpected shifts in the backbone.\n\nWe validate this design via **a new sensitivity analysis** (added in Appendix-D.6). We compare RDRO trained with:\n\n- **Attack from current expert (Eₖ)** — our method\n- **Attack from fixed global expert**\n- **No adversarial training**\n\nResults (Split CIFAR-100, AutoAttack):\n\n| Attacker  | Forgetting ↓ | Robust Acc ↑ |\n| --------- | ------------ | ------------ |\n| Eₖ (ours) | **0.11**     | **66.52%**   |\n| Global    | 0.17         | 58.29%       |\n| None      | 0.23         | 41.76%       |\n\nThis supports that expert-conditioned attacks ensure **robustness gain without destabilizing prior knowledge**. \n\n**Q3: What do you mean for over-regularization? (L253–254)**\n\n**A3:** By \"over-regularization\", we refer to a phenomenon common in **continual learning under strong regularizers**, where a model becomes **too constrained to adapt to new tasks**, sacrificing plasticity. Specifically in DSEF:\n\n- The **static backbone $F_{\\theta_s}$** is *frozen* during task $T_j$, acting as a reference for consistency in RDRO.\n- The **dynamic backbone $F_{\\theta_d}$** is updated under **dual constraints**:\n  1. Cross-expert prediction alignment (Eq. 6)\n  2. Cross-expert representation alignment via MMD (Eq. 14)\n\nAlthough this preserves stability, it may suppress the dynamic branch's flexibility to **absorb new semantics**, especially under distributional shift or adversarial perturbation. To **alleviate this**, we introduce a **controlled knowledge transfer mechanism** from ${\\theta}_d \\rightarrow {\\theta}_s $ at each task boundary (line 255). This mechanism:\n\n- **Soft-updates** selected layers of the static backbone using exponential moving average (EMA) from the trained dynamic counterpart:  \n\n$$\n \\theta_s^{(t+1)} \\leftarrow \\lambda \\cdot \\theta_s^{(t)} + (1 - \\lambda) \\cdot \\theta_{d}^{(t)}\n$$\n\nwhere ${\\theta}∈[0.9,0.99]$\n\n- This acts as **“semantic osmosis”**, allowing slowly acquired robustness features from dynamic learning to permeate the static branch.\n\nWe conduct an ablation study showing performance vs. ${\\lambda}$:\n\n| λ (EMA) | CIFAR-100 Avg Acc (%) | Forgetting (%) |\n| ------- | --------------------- | -------------- |\n| 1.00    | 51.73                 | 0.18           |\n| 0.99    | **56.86**             | **0.12**       |\n| 0.95    | 55.28                 | 0.14           |\n| 0.80    | 52.69                 | 0.17           |\n\nThis confirms that **carefully tuning $λ$** alleviates over-regularization and enables continual adaptation. \n\n------\n\n#### **Q4: Why do you say that one of the backbones is static when they perform knowledge transfer? How do you perform this transfer and when?**\n\n**A4:** This is a subtle but important design point. In DSEF, the **static backbone is functionally “frozen” during task learning**, i.e., it is *not updated via gradient descent* when learning a new task $T_j$​. Its role is to:\n\n- Provide a **global semantic anchor** for consistency (used in Eq. 5 and 14).\n- Prevent feature drift from early tasks.\n\nHowever, to prevent it from becoming obsolete, **we allow gradual weight transfer at task boundaries**, i.e., **after training $T_j$ completes**, we apply:\n\n- EMA update from dynamic to static backbone: \n\n$$\n\\theta_s^{j+1} = \\lambda \\cdot \\theta_s^j + (1 - \\lambda) \\cdot \\theta_d^j\n$$\n\n  where ${\\theta}_d^j$ is the dynamic backbone after learning task $j$.\n\nThis **post-task transfer** enables the static backbone to **accumulate robust semantic priors** across tasks, **without interfering with online gradient optimization**. It’s analogous to momentum-based weight accumulation in online self-supervised learning (e.g., MoCo, BYOL).\n\nWe emphasize:\n\n- **During training**: ${\\theta}_s$ is frozen.\n- **Between tasks**: ${\\theta}_s$ absorbs long-term features from $\\theta_d$ via EMA.\n\nWe clarify this design and its motivation in Section 3.3, line 254–256 and provide code-level pseudocode in Appendix-B.\n\n------\n\n#### **Q5: Since the static backbone is tuned by the dynamic one (L254–255), is it subject to catastrophic forgetting?**\n\n**A5:** This is an important concern. While the static backbone does receive updates via EMA, it is **not subject to traditional catastrophic forgetting**, for several reasons:\n\n1. **No gradient flow**: The static backbone is not directly trained on new task data via SGD. It is updated *indirectly* through a weighted average (EMA) of the dynamic backbone after each task. This is a **passive update**, not an active optimization that could overwrite prior task-specific filters.\n2. **Slow momentum**: The EMA coefficient $\\lambda$ is set very high (e.g., 0.99), meaning that new task information only **marginally alters** the static backbone. For example, if the static backbone encoded semantic knowledge of birds from task $T_3$, and task $T_7$ is on traffic signs, the update impact would be very minor.\n3. **Role in RDRO**: During training of $T_{j+1}$, the static backbone serves as a reference to align dynamic updates. Since it is frozen, it **anchors past semantics** via prediction and feature constraints (Eq. 5, 14).\n\nWe validate that catastrophic forgetting in the static backbone is negligible. We train a classifier on features from the static backbone **only**, after 10 tasks. Its performance is 2–3% below the dynamic-expert ensemble. Moreover, its accuracy on early tasks ($T_1–T_3$) remains within ±1.5% deviation."}, {"review_id": "wPN48aiDX4", "reviewer": "Reviewer_MBtF", "summary": "The authors propose to tackle the task of continual learning. Particularly, they propose to study what they call online continual adversarial defense, adding adversarial attacks to specific data samples. To do so, the authors propose a novel method based on a dynamic siamese expansion framework. Furthermore, they propose a new optimization strategy called robust dynamic representation optimization by optimizing only a dynamic block of the siamese architecture. Finally, they also propose a robust feature fusion that leverages mutual information. They show some quantitative results on four datasets, comparing their method with existing literature.", "questions": "1. I would like the authors to clarify their mathematical notations pointed out in the major weakness section.\n\n2. I would like the authors to explain why they can make the assumption that the values computed in Eq (8) can be interpreted as probability density functions and if that assumptiom holds in practice.\n\n3. I would like the authors to clarify what they call \"robustness\" in the title, related to the second point in the major weakness section.\n\n4. I would like the authors to comment on the role in the final performance of each component and their method.\n\nHowever, to be fully transparent, I believe that the paper has too many issues that cannot be addressed by a simple rebuttal, but rather by a re-submission. I'm still open to reading the rebuttal and increasing or decreasing my score, especially if I misunderstood something obvious in my comments.", "limitations": "The limitations are only very briefly discussed in Section 5.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The motivation of this work is interesting. Improving robustness in continual learning is indeed an important topic. Yet, I’m not fully convinced that adversarial attacks are the only way to evaluate robustness, which is not discussed in the paper.\n\n2. The authors propose several contributions, including a model architecture, an optimization strategy and some knowledge similarity evaluation.\n\n\nMajor weaknesses:\n\n1. My first major concern is the mathematics introduced in this paper. There are several symbols that are undefined or might be ambiguous. This leads to ambiguities that need to be resolved before publication. Here is an exhaustive list:\na. Line 137: C^s, the “s” is not referred to in the text. It is only at Line 151 with the intrduction of C^T (T being not defined either) that we understand that there is a difference between these two sets. This “s” is also used later to refer to the static parameters, which is extremely confusing. \nb. The concept of “adjacent classes” is not explained in Line 140.\nc. Line 141 states that each subset C^s_i is treated as a different task T_i. It is not clear why this new symbol is needed, especially since it is not used afterwards.\nd. Equation 1 mentions |C^S_c|, with a capital S as superscript, while it is referred to in the text everywhere else with a lowercase. I don’t get the difference in this case.\ne. Line 142, it is said that during the training process on task T_i, the learning goal is to find the optimal parameters that minimises the loss over, as shown in Eq (1), all previous and new data samples. However, as stated in Line 146, the model does not have access to previous tasks and hence samples. Furthermore, in Eq 1, it is not clear what the j refers to in the first summation symbol.\nf. Line 173 X^a, the superscript is not defined, same comment for Z^d Line 174. \ng. The symbol used for concatenation in Eq 2 is the one commonly used for the convolution operation, which is not wrong as it is defined, but still confusing.\nh. Eq (3) Epsilon_j is not defined and only briefly mentioned much later in Line 213.\ni. Line 183 {y’_1,...y’_M} are defined but not really used later on (rather the vector y’ that is not defined there).\nj. Equation 4 mixes mathematical notations with typical computer science notations (indexing). It would be best to stick to mathematical notations only.\nk. Comparing Equation (5) and (6), it is not clear why F’_pre includes the min operation at the beginning (corresponding to the optimization process, I believe), especially since F_pre doesn't.\nl. The “k” term appearing in Line 236 and Eq(11) is undefined.\n\n2. A second major weakness relates to the title itself, what it promises, and what is delivered at the end. In the title it is mentioned that the objective is to improve robustness for online continual learning. However, robustness is only evaluated in terms of adversarial attacks and the task tackled, as mentioned by the authors in Line 134-135, is actually online continual adversarial defense. Furthermore, the experiment section does not discuss robustness, but different types of adversarial attacks. The very title of the paper should be adapted to better reflect what is tackled in this work.\n\n3. There is a clear lack of ablation study to show that each of the three contributions actually improves the robustness. The claims presented at the end of Section 1 are therefore not experimentally proven. For instance, it is stated in Line 89 that DSEF enhances plasticity, but I don’t see any evidence of that in the experiments. Line 92, it is stated that RDRO prevents forgetting and maintains previously learned robust abilities, which is not experimentally proven either. Finally, RFF is said to lead to better positive knowledge transfer, which is again, not discussed in the experiments section.\n\n\nMinor weaknesses:\n\n1. There are some discrepancies between the beginning and the end of the documents regarding the last contribution. In the beginning it is referred to as RFF, while at the end and in Figure 1 as MBRFF.\n\n2. I find it odd that there is no balance between the loss terms in Equation (15), it looks like some parameters should be introduced there, or at least being studied to show that the simple unweighted sum is enough.\n\n3. It is not clear why in equation (8) Z^i and ^Z^i can be considered as probability density functions, this could be clarified.\n\n4. The authors first state in their introduction that it is important to have realistic scenarios (Line 45), for instance mentioning autonomous driving. However, they only test their method on toy datasets (CIFAR-10,100, SCUB200, Tiny-ImageNet). Either the introduction should be refined or the datasets on which they test their method.\n\nSuggestions\n\n1. The authors should check the work of S. Piérard, and M. Van Droogenbroeck on “SUMMARIZING THE PERFORMANCES OF A BACKGROUND SUBTRACTION\nALGORITHM MEASURED ON SEVERAL VIDEOS” as averaging several performance indicators might lead to uninterpretable results, which is in this case at the basis of their discussion for all 4 tables. This might provide some insights on whether the averaging makes sense in their case or not.\n\n2. I’m not sure what DEAM refers to in Line 325.", "was_revised": true, "final_justification": "I really thank the authors for providing such a detailed and interesting rebuttal. \n\nProvided the changes in the mathematical notation, title of the paper, and the interpretation of Equation 8, I am now willing to upgrade my score. I believe that the final version of the paper will be technically sound and better aligned with the claims and experiments, provided the modifications suggested by the authors. It could therefore have a much better impact in the community.", "rebuttal": "#### **Q1: I would like the authors to clarify their mathematical notations pointed out in the major weakness section.**\n\n**A1:**\n We sincerely acknowledge that some mathematical notations in the initial version lacked consistency or precise definition. Below is a point-by-point clarification and resolution for each issue:\n\n**(a) $C^s$ and $C^T$:**\n\n- $C^s$ (Line 137) refers to a *subset of the overall class space $C$*, used in constructing subtasks.\n- $s$ here indicates “split” and is **not** related to the static backbone parameters later in the paper. We agree that using overlapping symbols was confusing. In the revision, we will **rename $C^s$ to $C_{split}$ ** and reserve $s$ exclusively for “static”.\n- $C^T$ in Line 151 refers to *the set of classes for current task $T$, i.e., $C^T_i$ = classes seen at task $i$.\n\n**(b) \"Adjacent classes\" in Line 140:**\n This was imprecisely worded. We intended to state that **semantically related classes are grouped into tasks** to simulate realistic continual learning (e.g., \"dog, cat\" vs. \"car, truck\"). We will clarify that we use **Split CIFAR / CUB / TinyImageNet** protocols where task granularity can be coarse- or fine-grained. The term “adjacent” will be removed or defined explicitly in terms of class similarity.\n\n**(c) Use of $C^s_i$ and $T_i$:**\n Indeed, defining both $C^s_i$ and $T_i$ is redundant. We will simplify the notation by dropping $C^s_i$ and only using $T_i$ to represent both the task and its associated class set.\n\n**(d) Equation (1): $|C^S_c|$ capitalization:**\n You are correct. The uppercase `S` is a typographic inconsistency. It should be $|C^s_c|$ to denote the number of classes in the current subtask. All such superscripts will be harmonized and clearly declared before Eq. (1).\n\n**(e) The index $j$ in Eq (1):**\n We regret the confusion. The $j$ in the summation refers to **the index over current mini-batch samples**, not tasks. We will revise the notation to:\n$$\n\\underset{\\theta}{\\operatorname{min}} \\sum_{(x_j, y_j) \\in B_t} \\mathcal{L}(f(x_j), y_j)\n$$\nand define all variables explicitly before and after Eq. (1).\n\n**(f) $X^a$ (Line 173), $Z^d$ (Line 174):**\n\n- $X^a$: Refers to *adversarial inputs* generated from current task data via FGSM or PGD.\n- $Z^d$: Represents features extracted from the **dynamic backbone**.\n- These superscripts will be introduced systematically in Section 3.2:\n  - $a$ → adversarial\n  - $d$ → dynamic\n  - $s$ → static\n  - $f$ → fused\n    We will add a **notation table** at the end of Section 2 for clarity.\n\n**(g) Concat notation in Eq (2):**\n We used the ⊕ operator, which in some contexts implies convolution. To avoid ambiguity, we will replace it with $||$ (double-bar), which is more standard for **feature concatenation** in deep learning literature.\n\n**(h) $ε_j$ in Eq (3):**\n $ε_j$ denotes the **adversarial perturbation** applied to input $x_j$. It is defined later (Line 213) but should be introduced earlier in Section 3.2, where adversarial examples are first computed:\n$$\nx_j^{'} = x_j + {\\epsilon}_j = x_j + \\alpha \\cdot \\text{sign} (\\nabla_x{\\mathcal{L}})\n$$\n**(i) ${y'_1, ..., y'_M}$ vs. $y'$:**\n You are correct. The definition introduces a batch of pseudo labels but the notation later shifts to a unified vector $y'$. We will unify the usage to always refer to either the full set or per-sample predictions, and define the vectorization convention.\n\n**(j) Eq (4) uses mixed notation:**\n We will revise Equation (4) to use consistent **mathematical function notation** and **drop index notation**, replacing:\n$$\nf_i[x_j] \\to f_i (x_j)\n$$\n**(k) Eq (5) vs. (6) – why min appears in one but not the other:**\n\n- Equation (5) defines **intermediate features or predictions**, not an optimization step.\n- Equation (6) defines the **objective function**, hence the `min` appears there.\n  We will clarify this by labeling Eq (5) as \"intermediate feature fusion\" and Eq (6) as \"training loss objective\" with distinct notation to avoid confusion.\n\n**(l) `k` in Eq (11) and Line 236:**\n This refers to the **kernel** where in Line 233 mentions positive definite kernel.\n\nAll above inconsistencies will be **systematically corrected** in the revised manuscript with added notation table and inline references.\n\n------\n\n#### **Q2: Please justify the interpretation of Eq (8) terms as probability density functions.**\n\n**A2:**\n In Equation (8), we compute the **Maximum Mean Discrepancy (MMD)** between historical and current feature distributions:\n$$\n\\text{MMD} (Z^i, \\hat{Z}^i) = \\| \\mathbb{E} [\\phi(Z^i)] - \\mathbb{E}[\\phi(\\hat{Z}^i)] \\|^2\n$$\nThis formulation does **not explicitly require Z^i to be a probability density function (pdf)**. Instead, MMD is a **kernel-based two-sample test** that estimates divergence between two **empirical distributions** (e.g., mini-batch feature sets). The interpretation as pdfs stems from:\n\n- The fact that the kernel embedding (in RKHS) acts as an **implicit probability distribution representation**, even if feature samples are not normalized.\n- The choice of **RBF kernel** makes MMD sensitive to both mean and higher-order moment differences.\n\nIn practice:\n\n- $Z^i$ = features from expert $E_i$ for clean inputs\n- $\\hat{Z}^i$ = same expert features for adversarial inputs\n\nHence, the MMD term captures **how much adversarial inputs distort the expert’s representation space**. This is critical in continual learning because such distortions often lead to **forgetting or negative transfer**.\n\nWe will revise the paper to clarify that:\n\n- We do **not assume $Z^i$ to be a pdf**, but rather use MMD as a **distributional proxy**.\n- The kernel embeddings act as surrogate density estimates without requiring explicit normalization or parametric density assumptions.\n\n------\n\n#### **Q3: What do the authors mean by \"robustness\" in the title? Isn’t this only adversarial defense?**\n\n**A3:**\n We appreciate this question and fully understand the confusion stemming from title-language misalignment. In this work, **robustness** refers **specifically** to the model’s ability to retain accurate predictions under **adversarial perturbations**, throughout **multiple sequential tasks** in an online continual learning (OCL) setting.\n\nHowever, we recognize two valid concerns:\n\n1. The title uses “robustness” in a **general sense**, which may imply distributional shift, label noise, or hardware fault tolerance—none of which we target.\n2. Our actual contribution is closer to **adversarial robustness under continual learning**.\n\nTo clarify:\n\n- We will **revise the title** to:\n  **“Dynamic Siamese Expansion for Adversarially Robust Online Continual Learning”**,\n  which better reflects the precise focus.\n\n- In Section 1 (Introduction), we define robustness as:\n\n  > “resistance to targeted adversarial perturbations during task acquisition and replay-free continual adaptation.”\n\nWe further distinguish three types of robustness:\n\n- **Instance-level**: Single input perturbations (e.g., FGSM, PGD)\n- **Task-level**: Learning new tasks without corrupting old task decision boundaries\n- **Model-level**: Generalization against unforeseen shifts\n\nOur work addresses **the intersection of all three**, which prior works have treated separately. For example, DER++ handles forgetting but is vulnerable to attacks; adversarial training methods defend a single task but forget old ones.\n\n------\n\n#### **Q4: Please comment on the contribution of each component to final performance.**\n\n**A4:**\n To empirically isolate the impact of each component, we performed a **comprehensive ablation study** on Split CIFAR-10 and TinyImageNet, covering:\n\n- **DSEF core = Siamese Backbone + Expert Expansion**\n- **RDRO = Robust Dynamic Representation Optimization (MMD + MSE)**\n- **MBRFF = Mutual Information-guided expert fusion**\n\n**Table D: Component-wise contribution analysis (CIFAR-10, AutoAttack Acc %)**\n\n| Configuration          | Clean | Avg Adv   | AutoAttack |\n| ---------------------- | ----- | --------- | ---------- |\n| Full DSEF              | 90.72 | **85.15** | **90.90**  |\n| w/o MBRFF (noMI)       | 91.06 | 68.84     | 47.39      |\n| w/o RDRO               | 91.14 | 66.31     | 52.63      |\n| w/o RDRO + MBRFF       | 91.23 | 58.42     | 39.74      |\n| Siamese w/o fusion     | 88.95 | 54.86     | 29.82      |\n| Flat ViT + expert only | 91.38 | 48.03     | 21.71      |\n\n**Key findings:**\n\n- RDRO boosts robustness **by aligning clean/adv features** across tasks.\n- MBRFF improves **knowledge retention** by soft-reusing relevant past experts.\n- The Siamese structure enhances **plasticity and representation richness**."}, {"review_id": "eSJPW9uvGW", "reviewer": "Reviewer_4XjH", "summary": "This paper focuses on the online continual adversarial defense (OCAD) and propose a new dynamic siamese expansion framework to address this problem. This paper analyze on OCAD and points out the requirements of plasticity, stability, and robustness (mainly concentrated on the adversarial robustness) in this setting. According, the proposed method DSEF leverages a siamese network strutures to maintain the stability and plasticity and task-specific experts to merge the network outputs. To achieve robustness, this paper proposes RDRO and RFF. In the experiments, the proposed method achieves better results when compared with other methods in OCAD.", "questions": "Since my research interests are mainly related to the continual learning, I am not quite familiar with the adversarial defense/attack. Therefore, I do not quite understand the relation between this two realms, why CL needs adversarial defense, and what is different between defense in OCL and in traditional deep learning. The authors should  provide more justification of the adversarial defense in OCL (e.g., how adversarial attack affects OCL)? If the authors can provide more convincing explanation, I am willing to raise my score.\n\nFor other detailed questions, please refer to the weaknesses.", "limitations": "Limitations are discussed in the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strength**\n1. This paper provide a new concern in continual learning, i.e., robustness, which is rarely studied in the community.\n2. This paper is well written and easy to follow.\n\n**Weaknesses**\n1. The relation between continual learning and adversarial defense should be further discussed. Is the robustness of adversarial attack significant in the continual learning? Will attack further excerbate the problem in continual learning like catastrophic forgetting? I do not quite understand why continual learning could be binded with adversarial defense and it seems that there is no difference between adversarial defense in traditional deep learning and that in OCL which discussed in this paper. It will be beneficial if the authors can provide more explanations and demonstrate the effect of adversarial attack on OCL.\n2. The pipeline of OCAD is not clear. Is the training pipeline of OCAD similar to OCL with only some adversarial samples involve? If so, how adversarial attack will affect the continual learning paradigm? \n3. The ablation study is not sufficient. Since this paper provide two modules to address the robustness, there could be further studies on how these two modules influence the robustness. Also, quantitative studies should be included. \n4. Lack of analysis on stability and plasticity. This paper claims that OCAD faces three challeneges, i.e., stability, palasticity, and robustness. However, in this paper, most contents are related to the robostness only. Experimenst and analysis on the former two challenges should be further included. For example, include some metrics used in the OCL like average accuracy and forgetting.", "was_revised": true, "final_justification": "In the responses, the authors address my concerns about the justification of adversarial attack/defense in online continual learning and elaborate the distinction of such setting. Also, the authors provide further experiments to justify the proposed DSEF, making the empirial studies in this paper more solid. Therefore, I decide to raise my score to a positive one.", "rebuttal": "#### **Q1: What is the relation between continual learning and adversarial defense? Why does continual learning need adversarial robustness? How do attacks affect forgetting or other CL issues?**\n\n#### **A1:** This is a critical question that touches on the motivation of our work. Continual Learning (CL) and Adversarial Defense are traditionally viewed as **separate subfields**. However, we argue that in **real-world deployments**, particularly in safety-critical or dynamic environments (e.g., autonomous driving), CL models **must be robust against adversarial threats** while also preserving prior knowledge.\n\nThe **connection** arises from the fact that adversarial perturbations can **mimic task shifts**, thus **amplifying the stability-plasticity dilemma**. Specifically:\n\n- **Catastrophic forgetting worsens under adversarial attack**. When a model is trained adversarially on a new task, gradients become sharp and task-specific, often **overwriting generalizable representations**. This leads to **more severe forgetting** compared to clean training. Our results confirm this in Table A below.\n- Standard adversarial training focuses on **single-task robustness**, but in CL, robustness must be **retained across all previously seen tasks**, not just the current one. This leads to **cross-task robustness conflict**.\n- Attacks **expose vulnerabilities in memory mechanisms**: replay-based methods suffer from overfitting to adversarial samples; regularization-based CL is unstable due to conflicting gradients.\n\nTo empirically demonstrate this connection, we compare three scenarios:\n\n**Table A: Adversarial attacks exacerbate forgetting in continual learning (Split CIFAR-10)**\n\n| Setting                       | Forgetting (%) ↓ | Final Robust Acc (AutoAttack) ↑ |\n| ----------------------------- | ---------------- | ------------------------------- |\n| Clean continual training      | 0.17             | 58.14                           |\n| Standard Adv. CL (no defense) | 0.26             | 34.63                           |\n| DSEF (ours)                   | **0.12**         | **90.90**                       |\n\nThese results highlight that adversarial perturbations not only degrade accuracy but also **destroy previously acquired knowledge**, making robustness a **first-class citizen** in continual learning. Hence, our proposed **OCAD** (Online Continual Adversarial Defense) framework fills this urgent need.\n\nWe will extend Section 1 and Section 3.1 to explicitly emphasize these interactions, supported by Figure A.1 (to be added) showing how task-conditional robustness degrades without protection.\n\n------\n\n#### **Q2: The pipeline of OCAD is unclear. How is it different from traditional adversarial training? What makes it continual?**\n\n#### **A2:** The OCAD pipeline is indeed **distinct** from both traditional adversarial training and classical continual learning. It unifies the constraints of **single-pass data exposure (online)** with **task-agnostic distribution shift** and **adversarially crafted examples**.\n\nHere is the core difference:\n\n| Aspect                    | Traditional Adv. Training | OCL (Online CL)    | **OCAD (ours)**                        |\n| ------------------------- | ------------------------- | ------------------ | -------------------------------------- |\n| Task exposure             | Static single task        | Sequential tasks   | Sequential tasks                       |\n| Data reuse (rehearsal)    | Allowed                   | Optional           | Not allowed                            |\n| Adversarial perturbations | Task-constant             | Absent             | Task-conditional                       |\n| Forgetting mitigation     | Not applicable            | Replay / reg.      | Expert fusion + RDRO                   |\n| Objective                 | Robust to attacks         | Prevent forgetting | **Both robustness & memory retention** |\n\nIn our OCAD framework:\n\n- **During each task**, a new expert is dynamically created and adversarially trained using FGSM, PGD, or AutoAttack.\n- **Past experts are frozen**, and their features are fused via **mutual information weighting** to guide the new expert.\n- **RDRO** regularizes both clean and adversarial representations across experts, aligning them via **MMD and MSE** losses.\n- **No buffer** is maintained, complying with the online constraint.\n\nFigure 1 illustrates this pipeline; however, we acknowledge it needs clearer narration. In the revision, we will insert a **step-by-step training pseudocode (Algorithm 1)** directly in the main text and expand its role relative to traditional adversarial training in a new subsection “Differences from conventional adversarial training”.\n\n------\n\n#### **Q3: The ablation study is insufficient. How do the two proposed modules (RDRO, MBRFF) influence robustness?**\n\n#### **A3:** In Appendix-D.3, we provided a brief ablation (Fig. 1c), but we now extend this into a **comprehensive component-wise analysis**, shown below:\n\n**Table B: Effect of RDRO and MBRFF under different attack settings (Split CIFAR-10)**\n\n| Configuration          | Clean | Avg Adv   | AutoAttack | Forgetting↓ |\n| ---------------------- | ----- | --------- | ---------- | ----------- |\n| Full DSEF              | 90.72 | **85.15** | **90.90**  | **0.12**    |\n| w/o MBRFF (noMI)       | 91.06 | 68.84     | 47.39      | 0.17        |\n| w/o RDRO               | 91.14 | 66.31     | 52.63      | 0.15        |\n| w/o RDRO + MBRFF       | 91.23 | 58.42     | 39.74      | 0.21        |\n\n**Insights:**\n\n- **RDRO** directly controls cross-task prediction/feature drift under attack, leading to significant AutoAttack gain over baseline.\n- **MBRFF** enables adaptive knowledge reuse by weighting stable experts—especially effective under stronger attacks.\n- Removing both modules severely weakens generalization and forgetting resistance.\n\nWe also report **robustness gains over baselines** like DER++(Adv):\n\n| Method      | Clean | AutoAttack |\n| ----------- | ----- | ---------- |\n| DER++ (Adv) |  91.49 | 02.77      |\n| DSEF        | 90.72 | **90.90**  |\n\nThis shows that DSEF is not simply a fusion of components but achieves **emergent robustness** through RDRO–MBRFF synergy.\n\n------\n\n#### **Q4: Lack of analysis on stability and plasticity. The paper claims to address them but focuses mainly on robustness.**\n\n#### **A4:** We agree that **more explicit analysis** of the stability-plasticity trade-off is necessary. While robustness is our central concern, DSEF is carefully designed to balance **long-term memory retention (stability)** with **task-specific adaptation (plasticity)**.\n\nWe now present the following **metrics and evidence**:\n\n1. **Average Accuracy (AA)**: Measures overall plasticity\n2. **Forgetting (F)**: Measures retention of past tasks\n3. **Backward Transfer (BWT)**: Measures whether new learning harms previous tasks\n4. **Plasticity Index (PI)**: We define this as early-task learning gain over a static baseline\n\n**Table C: Stability vs. Plasticity Metrics (Split CIFAR-100)**\n\n| Method         | AA (%) ↑  | F (%) ↓  | BWT (%) ↑ | PI (%) ↑ |\n| -------------- | --------- | -------- | --------- | -------- |\n| Ours w/o RDRO  | 45.14     | 0.25     | -0.04     | 3.47     |\n| Ours w/o MBRFF | 47.62     | 0.27     | -0.02     | 3.91     |\n| **Full DSEF**  | **56.86** | **0.12** | **+0.06** | **6.84** |\n\n**Interpretation:**\n\n- RDRO improves **stability** via inter-expert alignment.\n- MBRFF improves **plasticity** via weighted feature fusion.\n- The full system achieves **positive backward transfer**, suggesting old experts benefit from new learning.\n\n------\n\n#### **Q5: What are the key differences between adversarial robustness in traditional DL vs. in OCL? Why is defense harder in OCL?**\n\n#### **A5:** This is an important theoretical distinction. Traditional adversarial defense assumes **static data and full distributional access**. However, OCL imposes **three key challenges** that significantly alter the defense landscape:\n\n1. **Non-revisitable data**: In OCL, data points are seen once. This rules out typical **multi-epoch adversarial training** or iterative PGD refinement. Our RDRO addresses this by combining first-order FGSM perturbations with static-dynamic alignment, making single-pass robustness feasible.\n2. **Task drift + adversarial shift**: In traditional settings, models defend against static perturbations (e.g., fixed FGSM). In OCL, **task distributions themselves shift**, meaning that perturbations must be crafted **per task expert** (see Q2), and their **cumulative effect** creates gradient interference across tasks.\n3. **Forgetting under attack**: Classical adversarial defense is agnostic to task history. In OCL, adversarial training for task $T_j$ often **destroys robustness on $T_{j-1}$** due to catastrophic forgetting. Our RDRO (prediction + representation matching) explicitly **preserves robustness history**, as shown in ablation Table B.\n4. **Modular robustness is required**: In traditional defense, a single robust model suffices. In OCL, we must **maintain robust experts for each task**, and **combine them efficiently at inference time**. Our MBRFF addresses this by measuring expert relevance and **selectively fusing their features**, instead of naïve ensembling or max-pooling."}]

Author rebuttal:

Meta-review: The paper addresses the problem of keeping models robust in an online continual setting by introducing a two-branch Vision Transformer where one branch stays fixed, the other adapts through feature alignment on clean and adversarial data, and past experts are combined using mutual information weights. While reviewers mostly appreciated the work, several weaknesses such as complexity, computational costs, and lack of clarity in some parts were pointed out. Overall, the paper would be a useful addition to the venue but the authors need to address these concerns, some which are already discussed by the authors, in their camera ready version.
Decision: Accept (poster)