The Lie of the Average: How Class Incremental Learning Evaluation Deceives You?
Authors: Guannan Lai
Da-Wei Zhou
Xin Yang
Han-Jia Ye
Venue: ICLR 2026 Poster (2026)
Keywords: Class Incremental Learning
Continual Learning
Evaluation Protocol
Extreme Class Sequences
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Class Incremental Learning (CIL) requires models to continuously learn new classes without forgetting previously learned ones, while maintaining stable performance across all possible class sequences. In real-world settings, the order in which classes arrive is diverse and unpredictable, and model performance can vary substantially across different sequences. Yet mainstream evaluation protocols calculate mean and variance from only a small set of randomly sampled sequences. Our theoretical analysis and empirical results demonstrate that this sampling strategy fails to capture the full performance range, resulting in biased mean estimates and a severe underestimation of the true variance in the performance distribution. We therefore contend that a robust CIL evaluation protocol should accurately characterize and estimate the entire performance distribution. To this end, we introduce the concept of extreme sequences and provide theoretical justification for their crucial role in the reliable evaluation of CIL. Moreover, we observe a consistent positive correlation between inter-task similarity and model performance, a relation that can be leveraged to guide the search for extreme sequences. Building on these insights, we propose **EDGE** (Extreme case–based Distribution \& Generalization Evaluation), an evaluation protocol that adaptively identifies and samples extreme class sequences using inter-task similarity, offering a closer approximation of the ground-truth performance distribution. Extensive experiments demonstrate that EDGE effectively captures performance extremes and yields more accurate estimates of distributional boundaries, providing actionable insights for model selection and robustness checking. Our code is available at https://github.com/AIGNLAI/EDGE.

Published as a conference paper at ICLR 2026
THE LIE OF THE AVERAGE: How CLASS INCREMENTAL LEARNING EVALUATION DECEIVES YOU?
Guannan Lai Da-Wei Zhou Xin Yang² Han-Jia Yel
School of Artificial Intelligence, Nanjing University
National Key Laboratory for Novel Software Technology, Nanjing University
3 School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics
Email: [laign, zhoudw, yehj| @ lamda. edu. en, yangxin @ swife.edu.cn
ABSTRACT
Class Incremental Learning (CIL) requires models to continuously learn new classes without forgetting previously learned ones, while maintaining stable per- formance across all possible class sequences. In real-world settings, the order in which classes arrive diverse and unpredictable, and model performance can vary substantially across different sequences. Yet mainstream evaluation proto- cols calculate mean and variance from only a small set of randomly sampled sequences. Our theoretical analysis and empirical results demonstrate that this sampling strategy fails to capture the full performance range, resulting in biased mean estimates and severe underestimation of the true variance in the perfor- mance distribution. We therefore contend that robust CIL evaluation protocol should accurately characterize and estimate the entire performance distribution. To this end, we introduce the concept of extreme sequences and provide theoretical justification for their crucial role the reliable evaluation of CII. Moreover, we observe a consistent positive correlation between inter-task similarity and model performance, relation that can be leveraged to guide the search for extreme sequences. Building on these insights, we propose EDGE (Extreme case-based Distribution & Generalization Evaluation), an evaluation protocol that adaptively identifies and samples extreme class sequences using inter-task similarity, offering a closer approximation of the ground-trutt performance distribution. Extensive experiments demonstrate that EDGE effectively captures performance extremes and yields more accurate estimates of distributional boundaries, providing action- able insights for model selection and robustness checking. Our code available https:: /tte:://gttttto
github. com COw.AAIGNLA//EDG.....
INTRODUCTION
Class Incremental Learning (CIL) seeks to equip a model with the ability to incorporate new class knowledge over time while preserving accurate recall of previously learned classes (Zhou al.
2024a by Li et al., 2025). While much of the literature has centered on advancing architectures and algorithms, the equally crucial question of how we evaluate CIL has received far less attention. Recent studies reveal that final performance in CIL is highly sensitive to the sequence in which new classes arrive (Bell Lawrence| [022 Lin et al., 2023) [Shan et al.) 2024, Wu et al. (02T). Such sensitivity to class order is particularly problematic in realistic settings (e.g., autonomous driving), where the order of class emergence is inherently uncontrollable. Compounding this challenge, the number of possible sequences grows factorially with the number of classes (O(N!)), rendering exhaustive evaluation impractical. Consequently, CIL evaluation must rely on sampling only a subset of class sequences to assess and compare model performance.
Existing CIL evaluation protocols (Wang et al., 2024b] Zhou et al. 2024b) typically compute model capability by sampling only 3-5 random class sequences and reporting the sample mean and standard deviation an approach we call the Random Sampling (RS) protocol. Because RS relies on only handful of sequences, it yields only point estimates and provides no characterization of the full
"Coreesooningg author.

---

Published as a conference paper at ICLR 2026
Incrementul Learning Process
Incremental Learning Process
An Auto-Drive System
Possible Sequence (Hordest Sequence)
Posstble Sequence (Easiest Sequence)
Desitty
Probabiity Denity
0.20
0.15
Probability Density
0.20
0.15
0.10
0.15
0.10
0.05
0.10
0.05
70
80 Accumacy
90
0.05
0.00
70
80 Accuracy
90
Real Performance Distribution
0.00
70
Accuracy
Your model is safe with an overage accuncy 85%6
Your model may be flawed, with accuracy
The CIL evaluation protocol should consider the model performance distribution (upper lower bounds)
Existing Protocol
Ours EDGE Protocol
Existing CIL evaluations may be misleading! They merely compute the average accuracy without perceiving the performance distribution, failing to anticipate the impact of potential extreme sequences on the model.
Figure I:
performance distribution. To examine whether RS can reliably approximate the true performance distribution, we conduct a controlled study with 6 classes organized into 3 sequential tasks, resulting in 90 possible aarivalrrival orders. We exhaustively evaluate each sequence to obtain the ground truth distribution of test accuracies. Figure [ illustrates this experimental setting. Consider a realistic incremental-traine -triinnntrraiingggg scenario, such as autonomous driving system, where numerous intrinsically different class-arrival sequences may occur in practice. Evaluating all sequences in our controlled study produces the ground-truth performance distribution. Analysis of this distribution reveals two key observations: first, the model performance approximately follows a Gaussian shape; second, there substantial variation in extreme cases, with the gap between the easiest and hardest sequences reaching up to 20 in our example (Hide-Promp) (Wang et al. 2023b on CIFAR- 100 (Krizhessky) ).09
Following the RS protocol, we emulate typical practice by randomly sampling three sequences and fitting a Gaussian N(u, o²) using their sample mean and variance. As highlighted in the blue box of Figure [ comparing this RS RS-essimated Gaussian against the ground-truth distribution reveals systematic bias: RS tends to overestimate the mean, dramatically underestimate the variance, and fails capture the true upper and lower performance bounds. Consequently, selecting models based solely on the reported average is risky - a model with an inflated mean but a poor lower bound may cause severe failures in real-wort deployment. These observations demonstrate that RS is inadequate for faithfully capturing CIL performance; a reliable evaluation protocol must either characterize distributional extremes otherwise provide substantially better approximation of the full performance distribution.
Motivated by the RS protocol's neglect of extreme sequences and supported by theoretical analysis of these cases, we adopt extreme case sampling to more comprehensively characterize model capability and achieve more accurate performance estimates. Through both theoretical and empirical analysis of inter-task similarity and model performance, we identify inter-task similarity as key factor influencing CIL performance. Building on this insight, we propose EDGE (Extreme case-brr case-based Distribution & Generalization Evalnation), novel evaluation framework for CIL. EDGE encodes class-level textual descriptions using a pre-trained CLIP model to construct a class similarity matrix. It then generates three representative class sequences: one that maximizes inter-task similarity to simulate an easy scenario, one that minimizes to represent difficult scenario, and one randomly sampled to serve and their results are aggregated by computing the mean and standard deviation, providing a more comprehensive approximation of the model's 's performance distribution. As highlighted in the red box of Figure I] EDGE produces a substantially closer approximation to the ground-trunt -trth distribution than the RS protocol, capturing both the central tendency and the distributional extremes.
medium case. Model performance is evaluated
these three sequences,
2

---

Published as a conference paper at ICLR 2026
The main contributions of this work are summarized as follows:
We conduct a systematic study of evaluation protocols in CIL, emphasizing that evaluation should aim to capture the full performance distribution of a model. Through both theoretical analysis and empirical investigation, we show that the widely adopted RS protocol produces biased estimates and fails to reflect the realistic behavior of CIL models.
We propose EDGE (Extreme case-based Distribution & Generalization Evaluation), a novel evaluation framework that adaptively identifies and samples both easy and challenging class sequences based on inter-task similarity, thereby providing a more faithful approximation of the ground-truth performance distribution.
Extensive experiments validate the effectiveness of EDGE in sampling extreme sequences and estimating model performance accurately. Our analysis also uncovers notable phenomena, such as different methods exhibiting comparable lower-bound performance in specific scenarios, offering critical insights for the design of future CIL models.
2 RELATED WORK
Class Incremental Learning (CIL): Existing CIL approaches can be broadly categorized into non- pre-trained and pre-trained based methods (Cao al. methods typically fall into three categories: (1) Regularization-base methods, which introduce explicit regularization terms into the loss function to balance the learning dynamics between old and
2023 Dohare et al. 2024). Non-pre-trained
new tasks (Aljundi al. 2018: Kirkpatrick et al. 2017 Li & Hoiem 2017, Wang et al., (022c);
(2) Reelay-based methods, which alleviate catastrophic forgetting by replaying data from past tasks, either through stored exemplars (Cha et al., 2021 Lapez-Paz & Ranzato 2017 Riemer al. 2018
Wang et al. 2022a) or via generative samples synthesized by GANs (Cong et al.] 2020 Liu et al. 2020: Shin et al. 2017 Zhu et al. 2022); and (3) Dynamic network methods, which modify the network architecture- such as by expanding layers or neurons to accommodate new knowledge while preserving prior information (Aljundi et al., 2017: Cao et al.) 2025 Ostapenko et al., 2021 Wang et al. 2023c 2022b). In contrast, PTM-base methods leverage the representational power of pre-trained backbones and mitigate forgetting through three main strategies: (1) Prompt-based methods, which apply lightweight updates via prompt tuning while freezing the backbone to maintain generalization (Jia et al. [2022] Li et al. [024] (Smith al.) [023] Wang et al.) [223a e))2d]e);
(2) Model mixture-based methods, which store intermediate checkpoints and integrate them using ensemble or model-merging techniques (Gao et al., 2023 Wang et al. 2024a 2023d Zheng et al.
2023 Zhou et al. [024c] 2025); and (3) Prototype- trasetype-based based methods, which classify examples using
nearest- paarest-class-eaure strategies grounded in PTM embeddings (Lai et al. 2025 McDonnell
al. 2024 Panos et al.) 2023 Zhou al. 2024a).
Evaluation Protocols of CIL; Evaluation protocols in CIL have received comparatively limited attention. Prior studies such as Farquhar & Gal (218) Hsu al. (2018); Mundt al. (202T) propose
multi-dimensional assessment criteria and benchmarks, while Chenn et al. (2025) investigates dynamic task allocation probe lower-bound performance. In contrast, our work adopts a distribution- oriented perspective: rather than relying on a few random trials, we aim to estimate the underlying performance distribution via a small set of informative, extreme-aware samples. This approach enables more reliable assessment under atypical or adversarial class sequences and delivers more actionable guidance for model selection and design.
PRELIMINARIES
3.1 PROBLEM DEFINITION
Class Incremental Learning (CIL). Given an ordered sequence of tasks (1....,t.... f....., }. each task i associated with a training set D = (Xi, Y'), where X denotes the input samples and y the corresponding labels. Let CLS denote the class set for task i, with cardinality |CLS A crucial constraint enforces strict separation between tasks: Vi + j E (1..... .n]....n].. CLS n CLS' = 0. and no inter-task data accessibility is allowed during training. The goal of CIL is to learn unified embedding function v D' + Rd that maps inputs to a shared embedding space, along with a classifier f(() capable of maintaining discriminative performance across all encountered tasks.
3

---

Published as a conference paper at ICLR 2026
In the CIL scenario, given a sequence of learning classes O comprising T tasks, we define Ag, r as the classification accuracy of the model on the test set of the t'-th task after training on the first t tasks. Based on this, the overall evaluation metric for sequence O can be formally defined as:
Σ Ar.-- A(O) T
1
(I)
Objective of CIL Evaluation Protocol Design. Let Z denote the space of all possible class sequences under a given CIL setting. By sampling L sequences (Oi,...,OO, Or.)
C S and computing their final ( we construct an empirical performance distribution Pemp with
accuracies A(Or) using Equation
mean μad and standard deviation
As shown in Figure D and the appendix, the realistic distribution Piree is approximately Gaussian. We therefore use N (,L,, 2 ) to approximate it, and define the goal of protocol design as minimizing the distributional distance between Pire and its estimate, measured by metrics such as Jensen- -hannnn divergence (Lamberti et al., (007) or Wasserstein distance (Villani 2009).
Random Sampling (RS) Evaluation Protocol. For a given CIL model M. the conventional
evaluation protocol uses three fixed random seeds (Lai et al. 2025 Li & Zhou 2025 McDonnell
al. 2024 Wang et al. (022e) to generate class sequences (RS, j-r- The performance of the model then estimated by computing the mean and standard deviation of final accuracies: μ.A -i=1 A(Si). o2 >i (A(R))) p.)2.. However, prior work only uses these statistics to summarize performance, without evaluating how well the estimated distribution matches the true one. This leads overconfidence in the evaluation results and may result in misleading conclusions.
3.2 LIMITATIONS OF RS EVALUATION PROTOCOL
Despite the substantial advances in CIL, to our knowledge, no prior work has critically examined the validity of prevailing evaluation protocols. In this section, we undertake theoretical investigations to address this oversight. All theoretical results and proofs in this section are provided in the Appendix.
First, we demonstrate through Lemma that CIL evaluation cannot be reliably accomplished using existing RS protocols, due to the combinatorial explosion in the number of possible class sequences. Lemma 1. Let N be the total number of classes, partitioned into K tasks of equal size M N/K. Then the number of distinct class sequences is [S2] (M Moreover, under linear scaling K = =(N), the quantity [S] grows factorially, satisfying [ = (((N/e)M), which asymptotically dwarfs any polynomial-scaee sampling capacity as N + co.
N
For = 100 classes divided into K = 10 tasks, the number of possible sequences is approximately 10 vastly exceeding practical enumeration. The RS protocols typically sample only 3 class sequences, covering less than 10-90% of the space and thus suffering from severe under-sampling bias. Building on Lemma [ we now ask: How Many random sequence samples are required to approximate the true accuracy distribution over the full sequence space within given tolerance?
Theorem
Let Q be the set of all possible class sequences with [Ω] elements, and fix tolerance s > and failure probability d E (0, 1). Suppose we draw L sequences {RS:) without replacement uniformly from Q and Ar. 1. Et= A(RSi) be the empirical mean, respectively. Then for any > 0, if
[0] (2(212218)
[ L
L
(2)
2₂a
then with probability least 1 8, [ [A[W]]]]] I
Remark 1. Substituting [Ω] NI/(MI)* (N/e) from Lemma 1 the condition equation becomes
N! MMI
L
In(2/5) + IN(NI/(MM))))))
L
Nin(N/e) + In
NI
2
2=2
(MIJ
For large (92] and L < 10)) the poatte-poplttion correction
[8 12
1, so one recovers the same
sample complexity scale D N In
5² ) in the with-replacement case. Even for moderate N (e.g.
4

---

Published as a conference paper at ICLR 2026
100
90
40
Accuracy (%)
Accurncy (%)
85
30
Accuracy
CBl BeLL17
80
80
Alt
20
75
70
Min
2.2
10
70
1.2P
CODA
RanPA RanPA
EASE
MOS
EWC
DER
ICaRI.
FOSTER
NEMO
TagFox
0.00
0.25
0.50
0.75
1.00
Similarity Score
(a) Under 100
(b) Under ImageNet-R
(c) Algorithm design motivation
Figure 2: Figures 2a and 2b show model performance under fully enumerable scenarios (green: maximum, red: minimum), along with estimates from the random sampling (RS) protocol (blue error bars). Figure 2c illlstrrats the correlation between inter-task similarity scores and model performance, where denotes the Pearson correlation coefficient.
N = 100) and a coarse E = 0.1, achieving high confidence (say = 0.05) still requires on the order of L 2 2 10a samples, so purely random sampling remains fundamentally impractical.
Noting that in Equation we have L < |92]. let E. { E E [A(w)) - [A(w)]] >
t [[(a)] The probability that none of the I, sampled sequences falls into E, is approxi- mately (xp(-((E,)/(1)) L). Hence, uniform random sampling almost surely fails to capture model performance in the most extreme cases. This observation motivates the idea deliberately construct- ing such extreme class sequences to directly evaluate easy and hard case performance; Theorem provides an initial theoretical analysis of this approach:
Theorem 2. Let Ω be the set of all class sequences, and define μ
Eunni [Aw))
V Varw~n [A(w]] as the realistic mean and standard deviation of the accuracy function A. Suppose we know two extreme sequences W4, w- satisfying A(w+) - 1 2 and μ A(w-) 2 a. Draw L sequences (RS without replacement uniformly from 12 [-+,w-}, and define AL+2 L+2 A(w-) + A(w+) + EIZIA(RS)) A(RSi)
Then for any 0 and 8 (0.1). if
[ - 2 L L
In(2(19)
2)/6) (R(r))²
[2 3
2
where R(o) Alw+) ) A(w ), then with probability at least 1 - 8, |A442 Bwsn (awa S 8. Remark Theorem 2 demonstrates that, under the conditions outlined in Remark incorporating extreme class sequences reduces the required sample size to a value proportional to (Hl0))² For instance. when R(a) 0.1 (which is common in practical scenarios), the lower bound the sample size L drops to around 50. This represents a significant reduction compared to uniform random sampling, underscoring the practical benefit of setrem--sequence-asssiee evaluation in CII..
4 EDGE: EXTREME CASE-BASED DISTRIBUTION & GENERALIZATION EVALUATION
4.1 MOTIVATION
we conduct an exhaustive evaluation under 6-class, 3-task setting. As illustrated in Figures 2a and 2b the RS protocol often fails to accurately estimate the true performance distribution, frequently leading to either underestimation or overes- timation of certain models, which compromises fairness in comparison. Meanwhile, the findings from Theorem importance of incorporating extreme class sequences to improve evaluation quality. Nevertheless, a key challenge remains in how effectively leverage dataset-specific structures and characteristics to generate extreme sequences that are both robust and generalizable, thereby enabling more reliable and informative evaluation protocols.
Building on the theoretical analyses in Section 3.2
2 together with the near-Gaussian nature of the true distribution, highlight the
In the CIL setting, it is intuitively understood that when adjacent tasks exhibit low similarity. model parameters undergo significant changes during task transitions, which increases the risk of forgetting.
5

---

Published as a conference paper at ICLR 2026
Empirical Distribusion 5(0) 0.72
CIL Label Space
EDGE Protocol Estimation
Apple Pear Orange Bear
Beaver Dog
Text Fnoaeer
0.06
0.04
S(0)
8(0)
0.84
4.02
0.00
75
K5
Sequence Pool
Hard
Sequence
Easy
Sequeeno
Figure 3: Illustration of the EDGE evaluation protocol. The sequence with a green background represents hard case, where similar classes (e.g., apples and pears) appear within the same task, resulting in low inter-task similarity. The sequence with an orange background represents an easy case, where similar classes distributed across different tasks, leading high inter-task similarity.
To investigate this phenomenon further, examine the relationship between inter-task similarity and model generalization error.
Theorem Consider a CHL system consisting of tasks, where each task Ty is associated with data distribution Di and class set Cs. The generalization error is defined Eq
K EKK, E Exz,y)D,, [L(((z).)))))))))))))(()) (, y)}. where L(h(z).)) 3) denotes the loss between the model prediction f((x) and the true label y. Given task order O (T,.T2....,Tk, TTk]. the similarity score S(O) is defined as:
S(O) (K (K- ISiSK CEC, e'EC.11
K
ΣΣΣ Σ Sim(e,
(5)
where Sim(c. c') denotes the semantic similarity in the representation space between classes and c' belonging to tasks T, and T,, respectively. Let On and Og denote the sequences with the minimum and maximum similarity scores S(O), respectively, and let O, represent randomly generated sequence. Then, the following conditions hold:
The similarity score satisfies S(O,) S S(O,) S S(O.).
The generalization error satisfy c,(On) 2 (O,) 2 (O.).)
Theorem S theoretically demonstrates that task similarity decreases, the upper bound of the illustrras the trend between inter-task similarity scores and corresponding model performance for all possible class sequences. The majority of methods show a positive correlation, empirically supporting this result by showing consistent decline in model accuracy task similarity decreases. Motivated by these observations, take advantage of inter-task similarity to construct extreme class sequences, which facilitates a more thorough and representative evaluation of CIL.
generalization eтror increases significantly.
4.2 EXTREME SEQUENCE GENERATION ALGORITHM AND PROPOSED PROTOCOL
Figure illustrates the proposed EDGE evaluation protocol. Given a dataset, since direct access to image instances is unavailable, we leverage the text encoder from pre-trained CLIP model to embed class labels. Specifically, each class label is encoded into d-dimensional semantic feature vector via the CLIP text encoder 4 forming a label feature set = [LI..... LN J. where L, = 2 E Rd By computing cosine similarities between these label features. we construct symmetric similarity L, quantifies the semantic similarity between matrix D N where each entry dy [[. |
classes i and j. Based on the similarity matrix D. we generate candidate class sequences by hierar- chically clustering (Nielsen 2016) semantically similar classes and strategically selecting subsequent tasks to minimize or maximize inter-task similarity. Two optimal permutations are identified using Equation 5 O₁ arg min S(o) for the hardest sequence and O. arg max.en S(o) for the

---

Published as a conference paper at ICLR 2026
Table 1:
Experimental results of pre-trained-basee methods on two datasets. The gray region indicates the ground-truth values, and the best results are highlighted in bold black.
CIFAR-100
ImageNet-R
CODA- Prompt
Hide- Prompt
CODA- Prompt
Hide- Prompt
Metric
L2P
EASE
MOS
RanPAC
L2P
EASE
MOS
RanPAC
minAv
71.83
64.67
72.50
95.33
92.17 17
98.50
57.75
18.72
65.90
88.24
85.56
90.91
RS
83.83
76.83
79.67
95.50
95.33
98.67
68.98
39.04
78.34
88.77
87.17
93.05
EDGE
72.83
73.00
73.00
95.33
93.83
98.67
66.31
21.93
71.89
88.24
86.10
93.05
maxAN
92.50
87.00
91.17
95.83
97.67
98.83
83,42
57.75
82.95
88.77
91.99 91.98
96.26
RS
87.33
81.83
90.67
95.50
95.83
98.83
75.40
43.85
78.34
88.77
87.70
95.19
EDGE
92.00
84.17
90.75
95.67
96,67
98.83
77.54
45.45
76.04
88.77
91.44
95.19
JSDRS
0.44
0.38
0.34
0.00
0.15
0.00
0.23
0.65
0.41
0.00
0.37
0.57
JSDEDGE
0.30
0.28
0.22
0.00
0.00
0.21
0.20
0.18
0.00
0.17
0.36
W Rs
2.81
2.92
3.89
0.00
0.48
0.00
1.59
9.85
2.44
0.00
1.18
2.25
W WEDGE
2,00
2.03
1.42
0.00
0.22
0.00
1.74
2.37
1.11
0.00
0.77
1.07
casiest sequence. To ensure the total number of sampled sequences remains unchanged, we randomly select one sequence as the Median Sequence, which theoretically guaranteed lie near the center of the distribution Theorem [ By evaluating models on this triplet of task sequences, we approximate the true performance distribution and enable a more comprehensive assessment model capability.
To generate hard sequences, we first cluster classes based on semantic similarity using hierarchical clustering (Nielsen) 20T6). To encourage semantically similar classes to be grouped into the same task, we preserve large clusters intact and selectively split smaller ones needed, ensuring all classes are assigned to K tasks while minimizing global inter-task similarity.
After constructing task partitions, compute the inter-task similarity matrix ITS R and initialize the sequence with the task exhibiting the lowest global similarity. Subsequent tasks are iteratively selected based on minimal similarity the current task, forming candidate sequences.
By varying the clustering granularity, we generate multiple candidates and select the one with the lowest overall similarity score S(o). The easy sequence is constructed analogously, except that similar classes are intentionally assigned to different tasks, and each next task is selected based on maximal similarity to the previous one. Pseudo-code and analysis provided in the Appendix.
EXPERIMENT
Due to the exponential growth in the number of possible class sequences in CIL scenarios (as shown in Lemma obtaining the true performance distribution under standard experimental settings is infeasible. We therefore divide our experimental evaluation into two parts. First, conduct fully enumerable experiments on subsets of standard datasets, enabling quantitative analysis to validate the effectiveness of the proposed EDGE protocol. Second, we perform analytical experiments under standard benchmark settings, visually demonstrating EDGE's strong capability in capturing extreme performance cases.
5.1 ENUMERABLE EXPERIMENTS
5.1.1 EXPERIMENTAL SETUP
Dataset and Metrics. We conduct experiments on the CIFAR- 100 and ImageNet-R (Krizhevsky 2009) datasets. For each dataset, we select the first six classes and partition them into three tasks, generating 90 possible task permutations, which we consider the true distribution (D (rruc)) Next, we apply the RS evaluation protocol (using random seeds 0, 42, and 1993 (Lai et al., 2025 Li &
Zhou| 2025 McDonnell et al. 2024 Wang et al. 2022e)) to generate class sequences for evaluation, obtaining the estimated distribution DRS. Simultancously, we employ the EDGE protocol to perform th e evaluation, yielding the estimated distribution D DEDGE. To quantitatively assess the effectiveness of different evaluation strategies,
the JSD divergence and Wasserstein distance (JSDa (Lamberti et al. (007) and W,, (Villani 2009)) to measure the differences between the estimated and true distributions.
7

---

Published as a conference paper at ICLR 2026
Table 2: Experimental results of non-pre-trainedd-eee barseed methods on two datasets. Details are consistent with those in Table
CIFAR-100
ImageNet-R FOSTER
Metric
EWC
DER
iCaRL
FOSTER
MEMO
EWC
DER
iCaRL
MEMO
TagFex
min
12.50
16.83
36.33
16.00
21.83
10.06
11.73
10.61
11.17
12.85
11.73
RS
26.17
24.17
43.00
20.67
36.50
16.76
11.73
14.53
15.08
15.05
18.99
EDGE
12.50
26.35
38.50
16.67
35.67
10.61
11.97
14.53
11.73
15.44
14.53
Max
39.00
45.50
53.33
38.33
56.67
27.93
26.26
43.58
26.82
30.73
21.23
RS
27.50
34.17
43.00
23.50
51.17
21.23
18.99
22.91
24.02
21.23
20.11
EDGE
28.17
41.33
43.33
30.17
56.67
24.58
21.23
26.82
23.95
28.49
20.67
JSDRY
0.51
0.29
0.36
0.58
0.29
0.36
0.32
0.30
0.22
0.38
0.44
JSDEDGE
0.29
0.31
0.32
0.40
0.23
0.26
0.20
0.21
0.20
0.16
0.15
W ns
4.74
4.62
2.37
6.25
2,00
2,40
2.14
3.14
2.11
2.99
3.14
WEDGE
3,44
3.22
2.03
3.91
1.82
2.03
1.07
2.71
1.03
1.66
0.88
Baseline. To ensure a fair comparison, we benchmark our method under both non-pre-trainee and pre-trained settings against classic and state-of-the ofte---f-hhharrr art approaches: in the non-pre- pre-trainedd setting, we compare with EWC (Kirkpatrick et al.] 2017). DER (Yan et al., (202T), iCaRL. (Rebufh et al.) (017)) FOSTER (Wang et al. 2022a). MEMO (Zhou et al., 2023), and TagFex (Zheng et al. 2025 ) the
pre-trained setting, following Sun et al. (Sun al. 2025a), we evaluate against L2P (aang et al.
2022e CODA-Prompy (Smith et al. 2023), HidePrompt (Wang et al. 2023b). EASE
(Zhou et al.
2024c). and MOS (Sun et al. 2025b).
Implementation Details. Our framework is implemented in PyTorch, and the code is provided in the supplementary materials. Complete experimental details can be found in the appendix.
5.1.2 EXPERIMENT RESULTS
Tables and 2 present the experimental results of two types of methods on the CIFAR- -100 and ImageNet-R datasets. The results are organized into four sections: the first section shows the true lower performance bound (highlighted in gray) along with the lower bounds estimated by RS and EDGE; the second section similarly compares the upper performance bounds. The third and fourth sections display the JSD Divergence and the Wasserstein Distance, respectively, between the estimated distributions from both methods and the true distribution. Based on these experimental results, draw the following conclusions:
RS leads to inaccurate and unfair comparisons. RS produces biased estimates of performance boundaries, which may lead to unfair comparisons among models. For example, when evaluating non-pre-trained methods on the CIFAR-100 dataset, the lower bound estimated by RS for EWC (26.17%) is significantly higher than its true lower bound (12.50%). Notably, although the true lower bound of DER (16.83%) is actually better than that of EWC, the RS estimate suggests a worse erroneous conclusion. In contrast, EDGE provides
lower bound for DER (24.17%) leading more accurate estimations of these boundaries, thereby avoiding such incorrect comparisons.
EDGE captures extremes and supports more comprehensive evaluation. In the vast majority of experimental cases, the performance bounds estimated by EDGE are significantly closer the ground-truth bounds (gray area) than those estimated by RS. Furthermore, EDGE demonstrates stronger capability approximating the true performance distribution, as reflected in its consistently lower equal JSD Divergence and Wasserstein Distance values compared to RS in most scenarios.
Multiple methods may converge to similar worst-case performance under hard sequences. On the challenging ImageNet-R dataset, the true lower-bound performance (i.e., worst-case accuracy) of multiple non-pre-trained methods clusters within narrow range of 10.06% 12.85%. This consistency suggests that task difficulty itself, rather than architectural differences, constitutes the primary bottleneck in this setting. EDGE helps model developers recognize this phenomenon, highlighting that variations in model design have limited impact under such conditions.
The accuracy of boundary estimation is correlated with model performance stability. A model's performance stability directly affects how accurately its bounds can be estimated. Models with stable performance and low variance (e.g., EASE, MOS, and RanPAC in Table I enable both RS and EDGE to estimate bounds accurately, yielding near-Zee zero JSD and Wasserstein Distance. In contrast, models with high performance fluctuation (e.g., non-pre-trained trained methods in Table 2 pose
8

---

Published as a conference paper at ICLR 2026
Realistie
0.10
KS Entmmate
Probahilty Density
EDGE with ResNed0
Probablity Density
0.08
0.15
EDGE will Ro
with RoNNeto0)
EDGE with 16
0,06
with VIT--12
BDDGE with VIFLIA
0.04
0.05
0.02
70
75
80
25
30
Accuracy (%)
Accuracy (%)
(a) Hide-Prompt
(b) TagFex
Figure Effect of task sequences generated with CLIP text encoders varying scales on the estimation of performance distributions under the EDGE protocol. The black curve denotes the ground-truth distribution, and the blue curve indicates the estimation obtained via the RS protocol.
greater challenges for bound estimation. It is in these cases that EDGE shows a clearer advantage over RS, producing closer bound estimates and lower distribution distances.
EDGE is robust across different configurations. Figure demonstrates that the EDGE protocol maintains high estimation accuracy under various settings, including different model backbones (e.g., ResNet vs. VIT) and different sizes of the CLIP text encoder. In all cases, EDGE consistently outperforms the RS protocol by providing estimates that more closely align with the ground-trute performance distribution. This highlights the reliability and generalizability of EDGE across diverse model architectures and embedding capacities. Additional results and detailed analyses for other experimental settings are provided in the appendix.
5.2 EXPERIMENTS ON CLASSIC CIL SETTINGS
Following the classic CIL setup, we conduct experiments using three datasets: CIFAR- 100 (Krizhessky) 2009), CUB-200 (Wah al., (01T). ImageNet-R (Krizhevsky 2009). Each dataset is partitioned into multiple tasks of equal size. Figure 5 visualizes the maximum and minimum accuracy values (max and min the sampled sequences under each protocol, highlighting their ability to capture the extremes of the performance distribution. The results demonstrate that EDGE consistently achieves both a lower estimated lower bound and a higher upper bound across nearly all scenarios, including highly stable methods such EASE (Zhou al.) (024c) and RanPAC (McDonnell et al.) 2024). This allows it to identify rare but critical performance extremes, providing a more reliable and practical assessment of performance for real-world deployments. For more detailed analysis, please refer to the Appendix.
90
90
KS list
RS list
Ascuncy (Ng)
Accuracy (%)
Apcurnoy
86
24
70
LDGB Est.
60
CoDA-
(a) Under CIFAR-100
(b) Under CUB-200
(c) Under ImageNet-R
Figure 5: Visualization of the estimated lower and upper performance bounds across three datasets under the classic CIL setting: (a) CIFAR-100, (b) CÚB-200, and (c) ImageNet-R. The slashed bars (/) denote the proposed EDGE, while the dotted bars (.) correspond the existing RS protocol.

---

Published as a conference paper at ICLR 2026
6 CONCLUSIONS
Class incremental learning (CIL) is inherently sensitive to the arrival order, making evaluation a distributional problem rather than a single-point estimate. In this paper, revisit the mainstream random sampling (RS) protocol and show, both theoretically and empirically, that estimating perfor- mance from only few randomly drawn sequences yield biased means, severely underestimated variance, and misleading conclusions about robustness under rare but critical class orders.
To address this issue, propose EDGE, extreme case-aware evaluation protocol that leverages inter-task similarity to construct representative easy, medium, and hard sequences, thereby providing more faithful approximation of the underlying performance distribution and its boundaries. Our analysis establishes the key role of extreme sequences in sample-efficient distribution estimation, and motivates similarity-guidee
sequence construction. Extensive experiments on both fully enumerable settings and classic CIL benchmarks demonstrate that EDGE more reliably captures performance extremes and yields distribution estimates closer to the ground truth than RS, offering more actionable evidence for model comparison, selection, and robustness checking in realistic deployments.
ACKNOWLEDGMENTS
This work is partially supported by NSFC 18, 62506160, 62476228), the Basic Research Program of Jiangsu (BK 20251251), and JSTJ-2025-177
ETHICS STATEMENT
This work does not involve human subjects, personally identifiable information, sensitive data. All experiments were conducted on publicly available benchmark datasets under their respective licenses, and no private or restricted data were used. The proposed evaluation protocol, EDGE, is designed to provide more reliable assessments of continual learning models and does not directly introduce risks of harm. The authors affirm that this research complies with the ICLR Code of Ethics and with standard practices of research integrity and transparency.
REPRODUCIBILITY STATEMENT
We are committed ensuring the reproducibility of our work. To this end, we provide the following: (I) Code availability: Our code is publicly available at
com/ C.om/AIGNLA AIGNLAI/
Detailed instructions for reproducing our experiments are provided Section E (2) Theo- retical results: All assumptions are clearly stated, and complete proofs of our main theorems are
EDGE
presented in Sections C.I. (3) Experimental details and additional results: Further dataset
descriptions, additional experiments, and analyses are provided in Sections B.2 and D.1.2
REFERENCES
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In CVPR, 2017.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
Samuel Bell and Neil Lawrence. The effect of task ordering in continual learning. arXiv preprint 2:Xi::2205./3323. 2022.
Boxi Cao, Qiaoyu Tang, Hongyu Lin, Xianpei Han, Jiawei Chen, Tianshu Wang, and Le Sun. Retentive forgetful? diving into the knowledge memorizing mechanism of language models. arXiv preprint arXiv:2305.09) 09144, 2023.
Xuemei Cao, Hanlin Gu, Xin Yang, Bingjun Wei, Haoyang Liang, Xiangkun Wang, and Tianrui Li. Erroreraser: Unlearning data bias for improved continual learning. In SIGKDD, 2025.
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co21: Contrastive continual learning. In CVPR, 2021.
Shengzhuang Chen, Yikai Liao, Xiaoxiao Sun, Kede Ma, and Ying Wei. Cldyb: Towards dynamic benchmarking for continual learning with pre-trained models. ICI.R, 2025.
10

---

Published as a conference paper at ICLR 2026
Yulai Cong, Miaoyun Zhao, Jianqiao Li, Sijia Wang, and Lawrence Carin. Gan memory with forgetting. NeurIPS, 2020.
Shibhansh Dohare, Fernando Bermandez-Garcia. Qingfeng Lan, Parash Rahman, A Rupam Mah- mood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 2024.
Scbastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint 17X.::1805.09733, 2018.
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang, A unified continual learning framework with general parameter-efficien tuning. In ICCV, 2023.
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. arXiv preprint arXiv: /8/0. 12488, 2015 8.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A et al. Overcoming
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka catastrophic forgetting in neural networks. National Academy of Sciences, 2017.
A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Guamnan Lai, Yujie Li. Xiangkun Wang, Junbo Zhang, Tianrui Li, and Xin Yang. Order-robust class incremental learning: Graph-driven dynamic similarity grouping. In CVPR, 2025.
Pedro W Lamberti, Ana P Majtey, Marcos Madrid, and María E Pereyra. Jensen-shannon divergence: A multipurpose distance for statistical and quantum mechanics. In AIP Conference Proceedings, 2007.
Qiwei Li and Jiahuan Zhou. Caprompt: Cyclic prompt aggregation for pre-trained model based class incremental learning. AAAI, 2025.
Yujie Li, Xin Yang, Hao Wang, Xiangkun Wang, and Tianrui Li. Learning to prompt knowledge transfer for open-world continual learning. In AAAI, 2024.
Yujie Li, Guamnan Lai, Xin Yang. Yonghao Li, Marcello Bonsangue, and Tianrii Li. Explor- ing worldworld continual learning with knowns-unknen knowledge transfer. arXiv preprint arXiv:2502.20124. 2025.
Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, 2017.
Sen Lin, Peizhong Ju, Yingbin Liang, and Ness Shroff. Theory on forgetting and generalization of continual learning. In ICML, 2023.
Xialci Liu, Chenshen Wu, Mikel Menta, Luis Herranz, Bogdan Raducanu, Andrew D Bagdanov, Shangling Jui, and Joost van de Weijer. Generative feature replay for class-incremental learning. In CVPR Workshops, 2020.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. NeurIPS, 2017.
Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. NeurlPS, 2024.
Martin Mundt, Steven Lang, Quentin Delfosse, and Kristian Kersting. Cleva-compass: continual learning evaluation assessment compass to promote research transparency and comparability. arXiv preprint arXiv:21 2021.
Frank Nielsen. Hierarchical clustering. Introduction to HPC with MPI for Data Science, 2016.
Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. NeurlPS, 2021.

---

Published as a conference paper at ICLR 2026
Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: A strong replay-fres baseline for class- inssemnncremttlll learning. In ICCV, 2023.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv: 11910, 2018.
Haozhe Shan, Qianyi Li, and Haim Sompolinsky. Order parameters and phase transitions of continual learning in deep neural networks. arXiv preprint arXiv:2407.10315. 2024.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. NeurIPS, 2017.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal free continual learning. In CVPR, 2023.
Hai-Long Sun, Da-Wei Zhou, De-Chuan Zhan, and Han-Jia Ye. Pilot: pre-trained model-based continual learning toolbox, 2025a.
Hai-Long Sun, Wei Zhou, Hanbin Zhao, Le Gan, De-Chuan Zhan, and Han-Jia Ye. Mos: Model surgery for pre-trained model-based class-incremental learning. In AAAI, 2025b.
Cédric Villani. The wasserstein distances. Optimal transport: old and new, 2009.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD 200ds-200-2011 Dataset. Technical Report CNS-TR-200 1-001, California Institute of Technology, 2011.
Fu- Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. ECCV, 2022a.
Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Coscl: Cooperation of small continual learners is stronger than big one. In ECCV, 2022b.
Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. In NeurlPS, 2023a.
Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. NeurIPS, 2023b.
Liyuan Wang, Xingxing Zhang, Qian Li, Mingtian Zhang, Hang Su, Jun Zhu, and Yi Zhong. Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. Nature Machine Intelligence, 2023c.
Liyuan Wang. Jingyi Xie. Xingxing Zhang. Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. NeurIPS, 2024a.
Liyunn Wang, Xingxing Zhang, Hang Su, and Jun Zhu. comprehensive survey of continual learning: Theory, method and application. TPAMI, 2024b.
Yabin Wang, Zhiheng Ma, Zhiwu Huang, Yaowei Wang, Zhou Su, and Xiaopeng Hong. Isolation and impartial aggregation: A paradigm incremental learning without interference. In AAAI, 2023d.
Zhen Wang, Liu Liu, Yiqun Duan, and Dacheng Tao. Continual learning through retrieval and imagination. In AAAI, 2022c.
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, 2022d.
12

---

Published as a conference paper at ICLR 2026
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, 2022e.
Tongtong Wu, Xuckai Li, Yuan-Fang Li, Gholamreza Haffari, Guilin Qi. Yujin Zhu, and Guoqiang Xu. Curriculum-meta learning for order-robust continual relation extraction. In AAAI, 2021.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In CVPR, 2021.
Bowen Zheng, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Task-agnostic guided feature expansion for class-incremental learning. CVPR, 2025.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In ICCV, 2023.
Da- Da-Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De- De-Chuan Zhan. A model or 603 exemplars: Towards memory-efficient class insremmactlttlll learning. In ICLR, 2023.
Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. IJCV, 2024a.
De-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: survey. In IJCAI, 2024b.
Da- Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model- based dass-incrrmmnntl learning. In CVPR, 2024c.
Da-Wei Zhou, Yuanhan Zhang, Yan Wang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for vision-language models. TPAMI, 2025.
Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zueng-Jun Zha. Self-sustaining representation expansion for non-exemplar class- daas-inncrmmentl learning. In CVPR, 2022.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
The appendix is organized follows:
Section A the notations and mathematical symbols used throughout the paper, providing a clear reference for theoretical and algorithmic components.
Section B presents detailed analysis of the existing RS (Random Sampling) protocol, including formal proof's of limitations and additional empirical results that support our claims.
Section C provides a comprehensive discussion of the proposed EDGE protocol. This includes pseudo-code, step-by-step explanations of the sequence generation algorithm, and theoretical justification for its effectiveness.
Section D reports extended experimental results and offers in in-depth analysis of the observed patterns. These findings provide new insights into the design and selection of CII. algorithms under varying sequence difficulties.
Section provides a practical guide to using our open-source EDGE repository, including installa- tion, running commands for the random vs. EDGE protocols, and notes the currently supported datasets and how to extend EDGE to new datasets.
A
NOTATION
Table Al: Notations and their explanations used throughout this paper.
Notation
Explanation
N
Total number of classes.
N(u, o²) Di
Gaussian (normal) distribution with mean μ and variance o².
Training dataset for task i, consisting of n; input-label pairs. Set of classes associated with task i. Sample space of all possible class sequences. A specific ordered class sequence of length K. True (but unknown) distribution of the model's performance.
CLS
Ω O Ω
Puree
A(O)
Final accuracy achieved by the model on sequence O.
K
Total number of tasks (i.e., the length of each sequence).
L
Number of sampled sequences used for estimation. Allowed failure probability (so confidence - 8).
5
Tolerance for estimation enror.
S(O)
Inter-task similarity score for sequence O.
Cg
Theoretical upper bound the generalization error.
CLIP text encoder mapping text tokens to d-dimensional embeddings.
L
Matrix of text embeddings for the C' class labels.
D
Similarity matrix between label embeddings (e.g. cosine similarity).
ITS
task similarity matrix aggregated from D.
Table AT provides a detailed description of the notations used throughout the paper, facilitating a clearer understanding of the mathematical formulations and algorithmic procedures.
B DETAILED ANALYSIS OF RS PROTOCOL
B. THEORETICAL ANALYSIS AND PROOF
Lemma 1. Let N be the total number of classes, partitioned into K tasks of equal size M N/K. Then the number of distinct class sequences is [
(MIJK Moreover, under linear scaling
N!
14

---

Published as a conference paper at ICLR 2026
K = (NN), the quantity [S2] grows factorially, satisfying [ dwarfs any polynomial-seaae sampling capacity as N - cc.
()((N/))*) which asymptotically
Proof. We begin by noting that the total number of distinct permutations N classes is N!. When partitioning these classes into K tasks of equal size M N/K, each individual task contains M unordered classes. Since the order of classes within task is irrelevant but the order tasks themselves is preserved, must quotient out the intra-task permutations.
For each task, there are M! ways to permute its classes internally. Since there are K such tasks, the total number of intra-task permutations (M!)* Consequently, the total number of distinct class N!
sequences that respect this task-based structure is given by [S] (Mrj* To analyze the growth rate of [.], assume a linear scaling regime where = ((N). Then M = N/K @(1), implying that
M! is constant with respect to N. Therefore, (A1)) K ( for some constant >> 0.
By Stirling's s approximation, we have:
N N! ~ 2
(6)
Thus,
N (M!) = Ω
((N//)MM)
)
N cc
[
=2
(7)
cN
which shows that [S] grows at least fast (Nfe') for some constant e' > i... [|] Ω ((N
Finally, note that any polynomial function in N is dominated by (N/e)* as N + co. Hence, the number of possible class sequences [S2] asymptotically exceeds any polynomial-scale sampling budget, concluding the proof.
Theorem 1. Let Ω be the set of all possible class sequences with [ elements, and fxx tolerance 0
and failure probability d E (0, 1). Suppose we draw sequences {RSi) without replacement uniformly from Ω and let Ar. 1 Σ A(RSi) be the empirical mean, respectively. Then for any 0, if
1n(2119188)
L
(8)
! -
2
then with probability least 1 8, [ [A(W[)]]] S E.
Proof. Let = (wi, w2, wai denote the finite set of all possible class sequences. Define the true mean accuracy
a
== == ==Ew-o(A(w)) = |
Σ A(wi).
(9)
Let (RS1, RSS...., RSL) be a sample of size I drawn uniformly at random without replacement from Q2 and define the empirical mean
Zr L A(RS.).
(10)
Our goal is to bound the deviation probability P( μa 2 E), under the assumption that A(-)
[0, for all w E Ω. Let us define the Doob martingale sequence
Zo = E[A,), Z, = A RS RS.,, 1,...,L L.
(11)
Then (Z forms martingale with respect the filtration Fi (RS ... RSS), and have:
1.
Zo = p. Z1 = Ar. and As - = Z1 - Zn = Σ (Z, - Zz-1):
(12)
i=1
15

---

Published as a conference paper at ICLR 2026
Since the sampling is without replacement from bounded set A(w) E [0, 1/, we can bound each martingale difference:
2 - 1 L
for all L....,,.
(13)
This bound can be obtained via an extension of McDiarmid's inequality for sampling without replacement, or directly computed via sensitivity analysis of the sample mean with respect one replacement in the sequence. Thus, the variance proxy is bounded as:
L 2-1
(1-
[ 1.
-1-1--11
[ [2 - L
1
SL
(14)
L
Ω
1=1
Using the standard Azuma Hoeffding inequality for martingales with bounded increments, obtain:
( 1 2 - 2 exp
e²
I 2ex -22 L-
Ω -
[ -I
(15)
2 N (Z+ Z:-1)² 12
To ensure that the deviation probability at most 5/152| for each of the [S2] possible values (for use in a union bound), it suffices that:
Ω L
exp
--2
(16)
[ -
Solving this inequality, we take logarithms on both sides:
[ L
2
- ln(0/25),
(17)
|
which is equivalent to:
L [0 L Ω
ln(2/92/6)
(18)
2s
L
Theorem 2. Let Q be the set of all class sequences, and define
Eunne[AW))] o =
V /Varwal [[w)] as the realistic mean and standard deviation of the accuracy functionn A. Suppose
we know two extreme sequences W4, w_ satisfying A(w+) p 2 and μ A(w-) a. Draw sequences { H without replacement uniformly from 52 [W+, and define ALt2 = Alw-) + A(w+) + Et. A(RS:)
Then for any e > 0 and 8 E (0, 1). if
L22
| 2 L
In(2 (0) 2)/8) (H(O))²
(19)
[ - 3
2
where R(0) Alw+) - A(w ), then with probability at least 1 - 8, |urt - Euoss [A(w)] s 8.
Proof. Let = (Wi,.... wlti } and write
[0]
= [2 Σ A(wi), 0² (ar=~a((Aw))
(20)
By assumption there exist sequences W 14,,0 w_ satisfying
A(w-) 1 2 G. 1 - A(w-)
(21)
Define the total range
R(6) = A(w+) A(w-) _ 2 20.
(22)
Draw samples { without replacement from S2 =Ω \ fw4, w- and set
I.
Ā = L+2 2 [A(w-) + A(w+) + Σ A(RS,))
(23)
16

---

Published as a conference paper at ICLR 2026
Consider the Doob martingale
Zo B[A4+2],
= =AA422 ........, RSS).. i= l=,....,
(24)
Then
Zo = 14 Zr = A AL+2 - p= 12, (i-1)-
(25)
1=1
Since each (SS) lies between the two extremes, replacing one sample can change the sum by at most R( Moreover, because we sample without replacement from a set of size [S2] 2, the sensi-
tivity of the average AL+2 to single replacement is further scaled by ( 2 T. 1.)/( 3). Altogether one obtains
12 - Zi-1l S L
R(o)
Ω
2
L
i
I.....L
(26)
[92| - 3
Hence, the sum of squared increments bounded by
I.
Rl0)
2
[0 - L
(2 - -14))=113 2;-1) L Σ
(27)
[2] -
By Azuma-Hoeffding.
Pr(lAL+2-wl 2 s 2exp
8²
< 22exp -2e²
(exp(-222
L
[ - (1)
2 E (Z,
2z-1)²
[9] -3
(28)
Requiring this probability to be at most 6/([S2) 2) and solving for gives
12 2
L In [n(2 2)/0) (R()))²
L
(29)
-
2
B.2 EMPIRICAL ANALYSIS
To empirically validate the distributional characteristics of performance metrics across different continual learning methods, we conducted experiments on CIFAR- 100 and ImageNet-R. shown in Table A2 the number of possible class sequences grows rapidly even with small number of classes. To balance feasibility and distributional richness, we chose configuration with 6 classes divided into 3 tasks of classes each, yielding 90 possible class sequences. This setting is large enough to exhibit meaningful variation in performance, yet still allows complete enumeration of the sequence space. For each dataset, randomly selected 6 classes and partitioned them accordingly. We then evaluated two groups of methods: Table A2: Number of possible class sequences [ under different partitions
N
K
M
N/K
Formula [2
N! (Md))
2
2
2
6
20
2
(4)) 2
10
2
5
101
252
(
3
(2
3
3
g!
1680
4
2520
(2t)
Non-pretrained CIL methods: EWC Kirkpatrick et al. (017). DER Yan et al. (2021) iCaRL
Rebuffi al. (2017), FOSTER Wang et al. (2022a), MEMO Zhou et al. (2023), and TagFex Zheng
al. (2025)
17

---

Published as a conference paper at ICLR 2026
Pre-trained CIL methods: LLP Wang et al. (2022e), CODA-Prompi Smith et al. (2023). Hide-
Prompt Wang et al. (2023b), EASE Zhou et al. (2024c), and MOS Sun et al. (2025b)
Observation The capacity distribution of the model is near-Gaussian
For each method-daten dataset pair, collected the final task accuracies over the 90 sequences and applied a Box-Cox power transformation. The optimal parameter À was chosen by maximizing the log-likelihood under the normality assumption. We then performed three normality tests on the transformed accuracies: the Shapiro- Wilk test, D' D'Aostinn''s 's K test, and the one-sample Kolmogorov- Smirnov (KS) Each yields a p-value indicating the probability of observing the data under Gaussian null hypothesis.
Table A3: Normality test results, demonstrating that the model capacity distribution approximates a Gaussian
Method
Dataset
Shapiro- Soopiro-Wilkyg p D'Agostino's p
KS p
CODA- PODA-PPrompt
CIFAR-100
4.5971
0,4753
0.4357
0.8365
CODA-Prompi
ImageNet-R
2.3957
0.9321
0.7483
0.8811
DER
CIFAR-100
2.2577
0.3582
0.3030
0.3824
DER
ImageNet-R
0.0321
0.2755
0.5488
0.6937
EWC
CIFAR-100
1.5076
0.1926
0.1913
0.4064
EWC
ImageNet-R
0.7893
0.2854
0.5735
0.3519
FOSTER
CIFAR-100
2.8988
0.2215
0,9340
0.4010
FOSTER
ImageNet-R
0.1042
0.2216
0.1903
0.2880
Hide-Proe iide-Prompt
CIFAR-100
4.1164
0.5802
0.4904
0.4783
Hide-Prompt
ImageNet-R
1.7462
0.3538
0.8186
0.6373
iCaRL
CIFAR-100
7.2381
0.1369
0,0326
0,4413
iCaRL
ImageNet-R
1.1917
0.9577
0,9403
0.9870
L2P
CIFAR-100
11.8644
0.0554
0,3215
0.2628
L2P
ImageNet-R
4.4415
0.1850
0.1541
0.6715
MEMO
CIFAR-100
0.7041
0.1720
0.0524
0.7084
MEMO
ImageNet-R
0.6267
0.8781
0.8697
0.7357
MOS
CIFAR-100
-1299763
0.4968
0.7681
0.5230
MOS
ImageNet-R 3.0909
0.1546
0.7973
0.2497
The results in Table A3 indicate that, after Box-Cot Cox transformation, most method com- binations exhibit p-values above the conventional 0.05 threshold in at least of the three tests, suggesting an adequate approximation to normality.
In addition, methods not listed in Table A3 such as RanPAC and EASE, produce limited number of possible task sequences due to their architectural design, resulting in insufficient sample sizes for reliable normality testing; hence, their results reported as n/a.
Observation 2: The sampling of RS protocol cannot reflect the realistic ability of the model well
Figure A and Figure A2 illustrate the performance distribution and the sampling locations obtained using the RS protocol (random seeds 0, 42, and 1993). Random sampling often fails to capture the true characteristies center of the distribution, making them ineffective in reflecting the model's behavior under extreme conditions. Second, the randomness of the sampling process introduces significant uncertainty across different data types and datasets, as the sampling locations vary considerably. This variability leads to unstable evaluations, where some methods are overestimated while others are underestimated. Third, two major issues arise when using these randomly selected points estimate the true distribution: the mean inaccurately estimated, and the variance severely underestimated. These problems together compromise the reliability of model evaluation under the RS protocol.
the distribution. First, most sampled points cluster around the
Observation 3: The performance of the model positively correlated with inter-task similarity
Building upon the prior theoretical analysis and the first two observations, recognize the necessity of incorporating extreme sequences for auxiliary evaluation. However, key challenge lies in adaptively identifying such extreme sequences and determining principled basis for algorithmic design. In the CIL setting, is intuitively understood that when adjacent tasks exhibit low similarity, the model parameters undergo substantial changes during task transitions, increasing the risk of
18

---

Published as conference paper at ICLR 2026
DIR CHEARIO)
IrageNn-i
EPWCCo CHPARIO
(DOiCeo IraggNNeitt
FIONTTRR
POSTER
0:30
3.66
0.01
6:94
0.64
0.5
0.02
BBC
0.00
JC3BL
Imageher-dl
M1360 CHPAROS)
NIEOO bragelNes.
Tlughox CHFAILTO
Tags tragehesot
8.10
0.00
0,10
9.10
00030
0.45
0.00
3.00
0.50
05
76
70
90
70
75
70
50
20
10
Figure Al: True performance distribution (black) and sampling positions under the RS protocol (blue) for non-pretrained CIL methods. The figure illustrates that RS fails to adequately capture the true distribution, leading to biased estimation.
ISIDAPP CIPARIO)
RrapeNNe-
BASSE Inugth
CHAMEEO
Iialo Imagehhesll
0015
6.00
8:04
6.1
0:05
6.02
0000
6:00-
0.8
00-
0.86
0:00
50
$4
66
55
93
36
1.2 CIRAR130
(M38 CIFARO00
1008 ImsggeNcsd
RantSC CHAARIOO
BaziMCC ImageNe-0
0.6
0.20
0.00
0.54
8/44
0.18
0.90
0.23
0.00
Figure A2: True data distribution and the sampling positions under the RS protocol (pre-trained CIL methods).
forgetting. Motivated by this intuition, investigate the relationship between inter-task similarity and model performance. As shown in Figure 2c, a strong positive correlation observed across most methods. This insight suggests that inter-task similarity be foundation for designing strategies to sample challenging sequences and evaluate model robustness.
C DETAILED ANALYSIS OF EDGE
C.1 THEORETICAL ANALYSIS AND PROOF
Theorem 3. Consider a CIL system consisting of tasks, where each task Ty. is associated with data distribution DD and class Cp. The generalization error is defined Cg
1 Ex= E(z,y),, [L(h(z).))) (,()))) where L(h(z), y) denotes the loss between the model prediction h(x)) and the true label y. Given a task order (Ti:T2....,Tk)) the similarity score S(O) is defined as:
K
K
S(0)) = (K 1)N
Σ E
Σ Sim(c,
(30)
ISiCK DEC; e'EC.11
where Sim(c, denotes the semantic similarity in the representation space between classes c and d' belonging tasks T; and T, respectively. Let O₁ and O. denote the sequences with the minimum and maximum similarity scores S(O), respectively, and let O, represent a randomly generated sequence. Then, the following conditions hold:
The similarity score satisfies S(O,)) < S(O,) S S(O.).
The generalization error satisfy (g(0n) 2 cg (O,) (O.)
19

---

Published as a conference paper at ICLR 2026
Lemma 2.
Lin et al. (2023)
When
must have:
7-1
T
T
Ele
W
1|
+
r
w
T
T
im1
im
1=1
pa²
-1² )²)
(31)
p
where the overparameterization ratio = - in this context quantifies the degree of overpa- rameterization in model, where represents the sample size, and p denotes the number of model parumeters. The coefficients (i = (1 - -r)(rr)-iii - +3 with Sisis < I T, correspond the indices of tasks, and denotes coefficient representing the model's noise level.
Proof. First, consider the total cross-task similarity:
Σ
Σ
Σ
Σ Sim(c,
(32)
ISiSK ISiSK e'Ec,
and the intra-task similarity:
Σ ISiSK CCEEC
Σ
Sim(c,d).
(33)
These satisfy the conservation relationship:
Σ ΣΣΣ
Sim(e, +
Σ ISiSK c,
Σ
Sim(c, c)= = C1.
(34)
GiSKISiSH CECIsKKKKCssssseo c'Ec,
For optimal parameters between tasks and j, we have: ||0 w/l 1|2
U
1l
mEC,
nEE
(Umss vnd)
(ums Unn -2 Σ
Σ
(Ums End
mEC; NEC,
mEC, NEC,
mEC, nEC,
Σ
Simlc, Σ Sim(e,d)
Σ OEC, C'EC,
Σ
Sim(c,e'), (35)
EC:
c,e'EC
where a is proportionality constant.
Substituting into the key term from Lemma
K
W
w
K
K
K
Σ a
Σ Sim(c,c) +
Σ Sim(c, - Σ
Σ
Sim((,d)
i=1
k=l
C
OEC. e e'EcK
K
K-ia
(K-1) Σ Sim(c,e))
Σ
Σ Sim(e,d)
1=1
c.e'EC.
GEC, d'eCk
1 K
KK- 11)
C1 Σ
Σ
Sim(e, c')
j= Fijitic d'ccy
K
K
K K--a
0
Σ
Sim(c,d)
K
k=1 cEC, e'Eck
K
-1-pk
JaC1
-
Σ
Σ
Sim(c, c')
K
(=1
k=1.kf
DEC, c'ECs
20

---

Published as a conference paper at ICLR 2026
= =2 -2
a
Σ
Σ
Sim(c, d):
(36)
K
ISISK DEC, e'Ec,
where C2 contains terms independent of the task ordering.
To establish the probabilistic bound for random sequences, let 52 denote the set all possible task permutations and O, Uni.(S2. Define the random variable Xij -Enec, Deec, Sim(e, The similarity score can be rewritten as:
K
K-1 Σ 1=1
S(O)
xx(0)=(++1)
(37)
(K
I)N
where ı is the permutation function. By symmetry, the probability that any two distinct tasks T; and T3 adjacent in random permutation is K[K-1]" Thus, the expected similarity score is:
E[S(O,)) (K-1)D 1)N
K
2
Σ ISKjSK
Xis
K(K -
X1..
(38)
1K2
iff
Applying McDiarmid's inequality to the function f(x) S(O), observe that swapping tasks in changes (xx by at most 40 N. where U is the upper bound of Sim(c, d.). This yields:
(( - [[/]| 2 8) S 2xxp (- 28
(-K(UU)
(39)
Letting ((0))))) E(f)), we obtain the concentration bound:
K0²
(S((O)) 5 S S(O.)) 2 2exp
(40)
SU2
This reveals inverse relationship between the similarity score S(O) and the generalization error Cy: the coefficient before the similarity summation term is negative, meaning higher similarity scores correspond to lower generalization error. Therefore, the ordering with maximum similarity O minimizes Eg. while the minimum similarity ordering On maximizes eg. with random ordering O falling between them.
C.2 PSEUDO CODE AND ANALYSIS
Algorithm Analysis. The proposed algorithm generates hard task sequences by systematically minimizing inter-task similarities, which aligns with Theorem conclusion that lower similarity scores correspond to higher generalization error. Key design rationales are analyzed as follows:
Step 2-3 (Dissimilarity Computation): First, we convert the class similarity matrix G E N into a dissimilarity matrix by computing D = 1 - and setting the diagonal to zero. This transformation ensures compatibility with clustering algorithms, where larger values indicate greater dissimilarity. Next, we apply hierarchical clustering with complete linkage on the condensed form of D to obtain K clusters. These clusters then sorted by size in descending order. For clusters whose size exceeds the base task size M = N/K, we iteratively assign subsets of the cluster to the currently smallest task preserve internal semantic similarity while maintaining balance. For smaller clusters, we assign all classes directly to the current shortest task. After this initial allocation, we perform a final adjustment step ensure all tasks are of equal size: any task exceeding the base size has its excess classes redistributed to tasks with fewer classes. This process results in K balanced tasks, each composed of semantically coherent classes, effectively minimizing global inter-task similarity and supporting the construction of hard class sequences.
Step 4-5 Multi-Granularity Clustering): Varying granularity levels G enable exploration different class grouping resolutions. This multi-scale approach increases the probability of discovering optimal task boundaries that minimize cross-task similarities.
21

---

Published as a conference paper at ICLR 2026
Clustering and Calegory A Allocation
Cluster
Task
Gramularty level gi
Task Orlen [4. 3, S(0) =.67
Class Space C = (G,, Cz. .., Cg}
Inter Task Similarity Matrix
Select Order with minimized S(O)
Now divide these
into tasks.
Cluster
Task
2
3
Task Orden 4, 5(0) 0.59
Granularity level Bgg = 2
Inter InterTask Similarity Malrix
Figure A3: Details of generating difficult category sequences (Algorithm)
Algorithm Hard Task Sequence Generation Algorithm
Require: Similarity matrix D, classes number N, tasks number K, candidate granularities set
Ensure: Task sequence O
1: Initialize similarity graph G <- D; set Gū f- 0 for all
2: Compute dissimilarity matrix M + G
3: for Granularity level g E G
4: Perform hierarchical clustering M into clusters
5: Merge clusters into K tasks (C1....,Cr, }. minimizing cross-cluster task assignment
6: Initialize inter-task similarity matrix ITS Rk*k
7:
for i.j E [K], i do
8: ITS R.C., Enrec, EoSEC, Gorez
9: end for
10: Select the first task: O₂ larg minke[K] Eje(K)s 1TS;j]
Initialize remaining task set: [K]] '0,
12:
while R # ( do
Select the next task: '-' + arg minKeR Eieo, ITS Append k" to the task sequence: O f- O, ok* Update remaining task set: R + R {
14:
15:
16: end while
Compute sequence score S(O,) according to Equation (30)
18: end for
19:
Select the sequence: O arg mino, S(O,)
20:
return O
The normalized average similarity ITS.; accurately reflects task relationships defined in Equation (30). This ensures algorithmic objectives align with theoretical similarity metrics.
Step 6-9 (Inter-Tast Similarity Matrix):
Step 10-16 (Greedy Sequence Construction): The initialization strategy selects the most isolated task as the starting point, preventing early error propagation. The iterative selection of least similar subsequent tasks implements a locally optimal strategy that approximates global minimization of S(O).
Step 17 (Multi-Granularity Optimization): Evaluating multiple granularities leverages Equa- tion (30 where better local minima are more likely to be found through diversified grouping strategies.
Figure A3 provides an illustrative example of the proposed procedure. Suppose we are given 8 classes to be partitioned into 4 tasks. Under finer clustering granularity, the classes are grouped into
22

---

Published as a conference paper at ICLR 2026
clusters, where the first cluster contains 4 classes, the second contains 3, and the remaining two contain class each. To maintain high intra-task similarity, the two extra classes in the first cluster redistributed to clusters 3 and 4, resulting in balanced 4-task partition.
Under a coarser granularity, the same 8 classes might be clustered into only groups: the first cluster with 5 classes and the second with 3. In this case, two of the most semantically similar classes from the larger cluster are assigned to form a new task, while the remaining two form another task, resulting in tasks overall.
After generating task partitions, we compute the Inter-Task Similarity (ITS) matrix and select an initial task with the lowest global similarity. We then construct candidate sequences by greedily adding tasks with the smallest pairwise similarity to the most recently added task. For example, from this procedure, we may derive two sequences: 4-33+2-+1 and 2-+1+4-33 with corresponding similarity scores of 0.67 and 0.59, respectively. This process is repeated across clustering granularities, and the sequence with the lowest overall similarity score S(O) is ultimately selected as the hard sequence.
Theorem 4 (Greedy Strategy Optimality Bound). Let O, be a uniformly random permutation of K tasks and define the average inter-task similarity
Σ N
$
Sim(c, d'),
(41)
2 IsiSisK [C,
C'C,c'EC,
and assume 0 S Sim(c, d' < U. Let Og be the sequence produced by the greedy Algorithm
7 Then
with probability at least 1 - e K/2
S(O,)) s B[S(O,)] A,
(42)
where
N2 (K 5,
N (K -1
5 2N (In +1)
B[S(O,)]
A
U.
(43)
2K
2K
K-1
In particular, i 4k2 K+1) N(K-1}²
U. then
> 0 and hence S(Og) < B[S(O,))
Proof.
Define, for i < j.
==,=
Σ
Sim(c. c').
(44)
aEC, d'Ec,
Since each pair (i.j) appears adjacent with probability
2 -(K-
B(S(O,)) = Σ
2
Xij = K(K -1) icj
2
Xi.j
(45)
K(K
Kjj
Noting |C,| = N/K, one finds
K
N2
N2(K 1)
Σ
X
S
B[S(O.)]
S.
(46)
K2
2K²
At step t of Algorithm I there are t(K t) candidate edges, each bounded by N2U. By standard
order-statistic arguments,
N2U
E < t(K - t) + +1'
(47)
and a union bound shows that with probability 2 1 - e K/2 each A, is at most twice its mean.
Summing over / = 1,...,K K 1 gives
K
K-1
N
2 (In K 1)
B[S(O,)] S [K - 1)N
U. K-1
(48)
t(K 1) +1
With probability least 1 - KK22
4N (In K + +1)
S(O,) < 2E(8(0,)
U.
(49)
K -1
23

---

Published as a conference paper at ICLR 2026
Therefore
N2 -1
4N + +1) K-1 U =
E -[S(O))) 2
(50)
2K²
Completing the proof.
Theorem tells that the greedy strategy of always choosing the similar remaining pair of tasks takes advantage of strong inter-task affinities to produce an ordering whose total similarity remains tightly controlled and, when the average similarity is high enough, is lower than that of random arrangement. The theorem shows that optimal local choices based only on current similarity scores accumulate into reliable global solution even in noise.
Complexity Analysis. With time complexity of O (Nº + +))),, the algorithm remains tractable for practical CIL scenarios where K < N. The cubic terms stem primarily from hierarchical clustering (Step 4) and inter-task similarity computations (Step 6-9). In practice, these steps can be further accelerated using approximate nearest neighbor techniques. For instance, when partitioning 100 classes into 10 tasks, the algorithm completes in approximately 0.5 seconds; for 200 classes into 10 tasks, it takes around 0.9 seconds on a standard CPU, demonstrating its efficiency for common CIL settings.
Similarly, Algorithm presents the pscudocode for constructing a simple task sequence by iteratively selecting and appending each task according to the prescribed rule.
Algorithm 2 Easy Task Sequence Generation Algorithm
Require: Similarity matrix D, classes number N, tasks number K, candidate granularities set Task sequence O
Ensure:
1: Initialize similarity graph G t I D; set G. + for
2: Compute dissimilarity matrix M
G
3: for Granularity level g E G do
4: Perform hierarchical clustering M into clusters
5: Merge clusters into K tasks [CI,.... Ck), minimizing cross-cluster task assignment
6: Initialize inter-task similarity matrix ITS k
7: for i.j E [K], i 7 do
8: ITS., Ic,, Emrec. ExSec, Gaez
9: end for
Select the first task: Og |arg maxxe[k] Eje(K]AAR ITSs
Initialize remaining task set: 5 [K] /O,
12: while R * 0 do
13:
Select the next task: 1- + arg maxteR Eieo, ITS
14:
Append Ax to the task sequence: O +- O k
Update remaining task set: R R (k*)
16: end while
Compute sequence score S(O,) according to Equation (30)
18:
end for
Select the sequence: +- arg maxo, S(O,)
20: return O
D DETAILED ANALYSIS OF EXPERIMENT
D.1 ENUMERABLE EXPERIMENTS
D.1.1 EXPERIMENTAL SETUP
Dataset and Metrics. We conduct experiments two standard benchmarks: CIFAR- Krizhevsky (209)) and ImageNet-B Krizhevsky (2009). For each, we select the first six semantic classes and group them into three sequential learning tasks of two classes each, yielding total of 3! possible task orders per dataset; by considering all class-to-tase assignments, we obtain 90 distinct sequences, which treat as the ground-truth distribution D To estimate this distribution in practice,
24

---

Published as a conference paper at ICLR 2026
Random Seed (RS) protocol: draw task sequences by shuffling class-labels under random seeds
[0, 42, ]993] Lai et al. 2025): Li & Zhou (2025); McDonnell et al. (2024); Wang et al. (2022e),
forming the empinical distribution Dns-
EDGE protocol: apply our edge-selection strategy on the same seeds to produce DEDGE.
We compare each estimated distribution to Derue using two complementary divergence metrics:
Gensen-Shanmng divergence J:D Given discrete distributions P and Q over the same support, the Sensen-Shanner divergence is defined as
JS | Q) = DKL M) + (QI(
-P +Q)
where DKL is the Kullback-Leibhe divergence. Unlike DKL. the JSD is symmetric and bounded in [0, 2n. which makes it well suited for measuring similarity between empirical distributions with potentially non-overlapping support Lamberti et al. (2007).
Wasserstein distance Wa: Also known as the Earth Mover's Distance, the first-order Wasserstein distance between P and Q on a metric space (X,
Wi (P Q) rET(P.()
inf Blaw)) [d(x, v)],
where I'(P, Q) denotes the set of all joint distributions with marginals and Q. In the discrete case, this reduces to the minimum cost of transporting "mass" from P to Q. providing meaningful measure distributional distance that accounts for the geometry of the task-permutation space Villani 2009).
Implementation Details. All methods are implemented in PyTorch with the following shared
hyperparameters:
Memory: total size 2000, up to 20 samples per class, non-fixed allocation. Backbone: ResNet-18, trained from scratch in the non-pre-trained treited setting and with ImageNet pre-training otherwise.
Optimizer & Scheduler: SGD with step-LR; initial learning rate 0.1, weight decay x 10-4, LR decay factor 0.1 at epochs {60, 120, 170} (non-pre-trained) or {80, 120, 150} (pre-trained).
Training: 170 epochs, batch size 128.
D.1.2 ADDITIONAL EXPERIMENT RESULTS
Table A4: Fitting performance of EDGE on CIFAR-100 using HidePrompt with various backbones. EDGE consistently outperforms RS across nearly all backbones, demonstrating its effectiveness and robustness to different model architectures.
Model
JSD
Wa
RS Estimate
0.2694
2.8688
EDGE with ResNet50
0.0863
1.5677
EDGE with ResNet50x 64
0.1986
3.2553
EDGE with ResNet101
0.1386
1.7020
EDGE with ViT-B/16
0.1236
1.5196
EDGE with ViT-B/32
0.1237
2.3599
EDGE with ViT-L/14
0.0846
1.0642
Table A4 summarizes the fitting performance the EDGE method on the CIFAR-100 dataset under the HidePrompt setting, using various backbone architectures. The performance is evaluated in terms of Jensen-Shannon Divergence (JSDa) and the 2-Wasserstein distance (Wa). Among all configurations, EDGE with VIT-L/14 best fits the reference distribution, yielding the lowest JSDa (0.0846 bits) and the smallest Wa distance (1.0642).
Figure A4 and Figure AS visualize the ground-truth performance distributions (black), along with the estimates produced by the RS protocol (blue) and our proposed EDGE protocol (red), for non- pre-trained and pre- trained CIL methods, respectively. These results demonstrate EDGE's superior ability to approximate the true distribution, capturing both the central tendency and the spread more accurately than the conventional RS protocol.
25

---

Published as a conference paper at ICLR 2026
D.1.3 DISCUSSION ON EDGE FOR MODEL SELECTION
In addition to providing more reliable evaluation, EDGE offers new insights for model selection. To demonstrate this, we compare continual learning method rankings under EDGE and RS across three dimensions: performance upper bound, performance lower bound, and stability, and quantify the consistency between the two evaluation protocols using ranking distance.
Table A3 presents the rankings under the fixed-class setting described in Section 5.1 We observe that EDGE rankings are overall closer to the reference ordering across all three dimensions, while RS exhibits larger deviations, resulting in higher total ranking errors. Specifically, on CIFAR-100, EDGE's ranking error is compared to 12 for RS, and on ImageNet-R, 2 versus 10.
Table A5: Model rankings derived from Table I
The reference (true) ranking is highlighted in gray.
CIFAR-100
ImageNet-R
Rank
Method
Lower
Upper
Stability
Lower Bound
Upper Bound
Stubility
Real
EDGE
RS Real
EDGE RS
Real
EDGE
RS
Real
EDGE
RS Real
EDGE
RS
Real
EDGE
RS
L2P
CODA-Prompt
5
5
4
4
5
Prompt
EASE
2
MUS
4
3
RanPAC
2
To further test the robustness of these findings, we repeated the experiment using randomly sampled set of six classes (seed 42), as reported in Table EDGE again outperforms RS: on CIFAR-I00. EDGE achieves ranking error of 0 versus 12 for RS; on ImageNet-R, 2 versus 12. These results confirm that EDGE consistently produces rankings that are closer to the reference ordering, providing more reliable basis for model selection.
Table A6: Model rankings obtained using a randomly selected set of classes (seed 42). The reference (true) ranking is highlighted in gray.
Rank
CIFAR-100
ImaggeNet-R
Method
Lower Bound
Upper Bound
Stability
Lower Bound
Upper Bound
Stability
Real
EDGE
RS Real
EDGE
Real
EDGE
RS
Real
EDGE
Real
EDGE
Real
EDGE
RS
L2P
5
5
3
6
6
5
5
5
5
5
4
Prompp
4
4
Hide-Prompt
EASE
MOS
22
3
RanPAC
2
5
Conclusion. Across both fixed-class and random-class settings, EDGE demonstrates superior fidelity in reflecting the true performance ordering of continual learning methods. It more accurately captures worst-case robustness, best-case potential, and stability properties deployment in practical scenarios. Overall, these findings highlight the value of distribution-aware evaluation and demonstrate that EDGE provides more informative guidance for continual learning model selection than RS.
stability-poopertiies critical for dependable
D.2 ANALYSIS OF LARGE-SCALE EXPERIMENT
Table A7 and Table A8 present the evaluation results of existing CIL methods under both RS and EDGE protocols. Notably, we observe conclusions consistent with those discussed in Section From the perspective of EDGE, these results offer insights into CIL model design and selection:
The realistic performance range of CIL models can be substantially wider than what is captured by RS protocols. EDGE effectively identifies both easy and challenging class sequences in most cases, and demonstrates broad applicability across pre-trained and non-pre-trained models. For example, on the CUB dataset, the performance range of L2P expands from 2.38 to 16.25, while that of TagFex increases from 1.06 to 7.27, enabling more accurate and nuanced understanding of model behavior. These findings highlight the importance considering extreme task sequences during model design to ensure robustness under diverse deployment scenarios.
26

---

Published as a conference paper at ICLR 2026
Table A7: Performance of pre-trained model-based CIL methods under two evaluation protocols. White background denotes the RS protocol, while gray background denotes the EDGE protocol. Reported are the sampled minimum and maximum accuracies, along with the estimated mean and standard deviation of the ground truth performance distribution (unit: %).
CIFAR 100
CUB
ImageNet-R
Method
minAy
max A wH
m
xf A A
AN
min,A
A
A
82.93
84.48
83,4640.72
66.18
68.56
67.25
71.10
71.43
71.29
L.2
81.67
84.62
83.08
59.75
76.00
67.31
71.02
72.37
71.61
85.17
83.56
85.30+0.18
85.30
70.32
71.56
70. 70.74410.59
73.03
73.54
73.2740.21
CODA-Prompt
84.82
85.65
85. 85.22
67.83
76.42
71.94
70.80
74.50
72.95 1.57
85.06
86.20
85.45
81.69
82.77
82.0640.50
71.76
73.16
72.58
Hide-Prompt
84.25
87.36
85,56
80.49
85.44
82.90+2.02
70.75
72.83
71.79
90.32
90.87
90.68
89.34
89.75
89.4940.19
77.27
77.32
77.30+0.02
RanPAC
90.25
90.87
90.65 0.
89.31
89.90
89.66+0.25
75.97
77.65
76.97
87.24
87.53
87.35
81.09
83.06
82.21
75.89
76.12
76.00
EASE
85.77
88.41
87. 37.1541.00
81.56
83.33
82,45
75.46
75.97
75.79
90.69
91.22
91.0340.24
88.87
89.39
89.08
76.90
77.33
77.15
MOS
90.79
91.22
91.01 +0.18
87.69
90. 16
89.08 1.03
76.48
77.93
77.21 +0.59
DEK CIPARU00
DER Nea
EWC CIRARIEO
FOSTER CIEARI00
POSTER Iengn Nic.
0.2
0.2
0053
0035
882
8:45
9.65
ALI
SCaRL im
M120 CHRAROS)
NIEO IeapelNes
TagPex CHARLOO
Tagfioo Inugehes-
8.10
KLLS
0.00
0.10-
0:05
6:25
0.00
0:01
70
20
Figure A4: The ground-truth distribution (black), along with estimates from the EDGE protocol (red) and the RS protocol (blue), for non-pre-trained CIL methods. EDGE provides a more faithful approximation to the true performance distribution.
Model rankings may change under extreme task sequences. For example, the CUB dataset, MOS and RanPAC exhibit comparable performance under the RS protocol, yet diverge significantly when evaluated with EDGE: MOS attains higher upper bound (up to 90.16) but experiences a notable drop in its lower bound. This indicates that algorithm selection should be informed by specific deployment priorities, whether emphasizing worst-case robustness maximizing best-case accuracy. EDGE offers valuable empirical evidence to support such scenario-aware decision-making.
Some model designs exhibit inherent limitations. For instance, on the ImageNet-R dataset, the lower bounds of three prompt-based based methods all approach 70%, indicating that certain difficult sequences can drastically undermine their effectiveness. This observation suggests that analyzing which types of sequences consistently degrade performance in different methods. Such insights can inform targeted improvements in model robustness, guide the development of sequence-awan training strategies, and support the selection of appropriate models for deployment in challenging real-world scenarios.
help identify structural weaknesses
D.3 ANALYSIS OF CLIP ENCODING
When designing the EDGE protocol, our objective is to construct representative class sequences of varying difficulty without accessing actual image instances. To this end, employ the CLIP text encoder, a vision-lann Aasionlanguuae model trained with contrastive learning that exhibits strong visual-text alignment and zero-shot generalization capabilities.
27

---

Published as a conference paper at ICLR 2026
Table A8: Performance non-pre-trained CIL methods under two evaluation protocols. Other notations follow those in Table A7
CIFARTOO
CUB
ImageNet-R
Method
minAA
A
N
millA
A
WHAN
millA
A
AANiOAN
13.83
14.94
10.44
10.31
10.90
10.60
7.38
7.77
7.61
EWC
13.79
17.22
15.08+1.52
8.18
10.31
9.46++.92
5.47
7.79
+1.09
57.59
59.73
58 48..9
46.65
48.05
47.50+0.61
29.37
32.92
31.57
DER
56.42
60.20
58. 58.25+1.54
45.25
49.62
47.17+1.82
29.28
34.92
32.37 +2.33
36.60
41.54
38.85+2.03
32.10
32.57
32.36+0.19
15.43
16.40
15.83+0.41
iCaRL
34.16
40.56
37.11
30.40
35.28
32.5942.00
13.55
16.78
15.58+1.44
48.43
51.47
19.87
42.66
43.75
43.07
18.70
20.52
19.47
FOSTER
49.21
51.23
49.62+1.17
38.25
45.25
42.42+3.01
17.03
21.52
19.25
55.16
58.49
56.80+1.36
39.31
41.52
40.23+0.u4
20.05
21.70
21.08
MEMO
54.96
58.96
56.36
1
39.31
41.31
40.19
00.83
19.50
21.70
20.87
TagFex
62.23
62.69
62.42
10.19
46.06
47.12
46.
10.46
34.05
34.27
34.16
60.78
68.80
63.94
13.44
13.48
42.62
49.89
46. 26.29
33.38
34.72
34.05
RS
CO CIFAR000
COOX en R
EASE en CIFARU0O)
EASE un NaRR
HSle CIFARIOO
HSlsP Inng
0.2
10
0.15
0.10
0.
8.02
0.90
1,2 CIFARIO
1,3 en IeageNN4
M4S es CEEARIOO
RORPPC CIEARIOOO
0.28
1.0
6.1
8:02
0.66
0.6
Figure A5: The ground-trutt distribution (black), and the corresponding estimates from EDGE (red) and RS (blue) protocols, for pre-trained CIL methods. The results highlight the improved accuracy of EDGE in capturing both the central tendency and variance.
D.3.1 DISCUSSION ON NATURAL IMAGES.
natural concern whether the semantic similarity measured by CLIP text embeddings faithfully reflects the actual visual difficulty among classes. To examine this, we compare the similarity matrix obtained from the CLIP text encoder with the similarity matrix derived from real image features. We evaluate the results using two metries:
Mean Absolute Error (MAE): quantifies the average deviation between the similarity matrices computed from CLIP embeddings and image features. Lower values indicate closer alignment. Consistency of Sequence Generation (CSG): measures the proportion of classes that remain assigned to the same task after generating sequences using the two similarity matrices. Higher values indicate more stable sequence generation.
Table A9: Comparison between CLIP text-based similarity and image-based similarity across datasets.
Dataset
CIFAR100
CUB
ImageNet-R
MAE (L)
0.14
0.08
0.17
CSG (1)
0.79
0.83
0.77
The results indicate that although moderate deviation exists between semantic and image similarities (MAE around 0.15), CLIP embeddings capture the underlying relational structure effectively. In particular, CSG remains consistently above 0.75, showing that the generated extreme sequences robust and align well with those derived from image-based similarities. This confirms the feasibility of using CL.P to measure similarity for generating extreme sequences within EDGE.
28

---

Published as a conference paper at ICLR 2026
D.3.2 DISCUSSION ON NON-MATURAL/PROFESSDON IMAGES
potential issue whether CLIP text embeddings derived from class names faithfully capture visual relationships in specialized, non-natural domains. This is nontrivial problem for several reasons:
Many domain-speciffe class labels are terse, technical, stage- based (e.g., "moderate" vs "severe") and therefore omit visual descriptors such as color, texture, or morphology that are essential for visual discrimination.
Within-class heterogeneity and between-class subtlety common: distinct clinical labels can correspond to overlapping or gradual visual features (e.g., small hemorrhages vs. microaneurysms)) making semantic labels a poor proxy for perceptual distance.
Dataset issues such as class imbalance, labeling protocol differences and inter-observer variability further weaken the simple mapping from short class name to image-space distribution.
Together, these factors explain why directly using bare class names with general-purpose text encoder can fail to reflect true image-space similarity in professional domains.
To assess this empirically, examined two representative medical-imen -imiceel--mage benchmarks. EyePACS large-scale retinal fundus dataset for diabetic retinopathy grading, containing color fundus photographs acquired with diverse cameras and imaging conditions. Visual cues range from microa- neurysms and small hemorrhages to hard exudates and neovascularization. Labels correspond to five DR severity levels (no, mild, moderate, severe, proliferative), which encode stage progression rather than detailed appearance descriptors. HAM10000 is dermatoscopic image dataset of pigmented skin lesions collected from multiple clinical sources. Images exhibit substantial variability in morphology, color, and acquisition artifacts, and several diagnostic categories are visually similar. The seven classes used here are Actinic keratosis, Basal cell carcinoma, Benign keratosis, Dermatofibroma, Melanoma, Nevi, and Vascular lesion.
For each dataset we constructed (a) an inter-class similarity matrix from CLIP text embeddings of class names and (b) an inter-class similarity matrix from image prototypes. We measured agreement between (a) and (b) using Spearman's i rank correlation. Using class names directly yields only modest alignment with image-derived similarities; EyePACS shows Spearman's p 0.588 (p 0.07), and HAM10000 shows p 0.279 (p = 0.22), consistent with the intuition above that short, technical labels do not reliably encode visual detail in these domains.
To increase the visual content of the textual representations, we expanded each class name into a concise, visually informative caption using large language model (GPT-5 in this experiment). The prompt we used is shown below inside a boxed, wwo-end-justifie block:
Prompt Template
Generate one concise, visually descriptive caption (8-20 words) that highlights the typical visual appearance, color, texture, and anatomical context of a (class_name) lesion in medical images.
We encoded the generated captions with CLIP and recomputed the class similarity matrices. This simple augmentation substantially increased agreement: EyePACS improved to Spearman's 's p = 0.863 (p 0.01), and HAM10000 improved to p 0.653 0.02). These results indicate that short, visually focused textual expansions recover much of the image-space relational structure that bare class names miss.
D.4 DISCUSSION OF OTHER POTENTIAL BASELINES
Although most CIL evaluations use the RS protocol, comparing only to RS risks underestimat- ing EDGE's ability to find challenging task sequences. We therefore include several additional, conceptually distinct baselines to evaluate both effectiveness and efficiency:
LLM--geerrted sequences. LLM-1: single-round generation procedure in which a large language model directly produces a candidate sequence based prompt describing "easy"
29

---

Published as a conference paper at ICLR 2026
"hard" sequences. This baseline tests whether semantic difficulty be inferred directly from class names without any iterative refinement; =130 per sequence. LLM-5: A five-round iterative refinement procedure. Each round, the LLM receives feedback regarding the previously generated sequence and attempts to correct or adjust its output in the next iteration. This baseline evaluates whether multi-step reasoning helps the L.LM better capture difficulty; =x600 s per sequence.
Adversarial Sampling (AS). A greedy, similarity-base selects the class that is maximally dissimilar from all currently selected classes, thereby increasing sequence difficulty by pushing the sequence toward the tail of the similarity distribution; 200.9 s per sequence.
adversarial strategy. At each step. AS
Max-cover Sampling (MS). A randomized search-based approach. We first sample pool of candidate sequences (we use 200), compute for each sequence a coverage or srrtees-distance score relative previously selected sets, and finally choose the top-ranked sequences; =8 per sequence.
Table A10: Comparison of EDGE against alternative baselines (Hard Easy correspond sequences intended to be difficult/ / easy for the evaluated methods).
EDGE
RS
AS
MS
LLM-1
LLM-5
Method
Hard
Easy
Hard
basy
Hard
Easy
Hard
Easy
Hard
Easy
Hard
Easy
L.2p
58.35
72.13
62.58
66.21
61.75
69.48
62.60
63.70
64.97
66.92
67.01
67.68
CODA-Prompt
65.65
69.42
67.42
68.12
65.96
67.33
66.13
68.18
67.01
68.58
65.78
69.59
Hide-Prompt
80.52
83.21
81.45
82.35
81.02
82.39
80.96
82.63
82.35
82.36
80.89
81.45
RanPAC
88.68
89.40
88.72
89.15
88.99
89.21
88.72
89.21
88.72
89.25
88.72
88.68
EASE
84.29
85.37
84.60
84.96
84.69
85.07
84.39
85.33
84.56
85.0T
84.82
84.78
MOS
87.69
89.56
88.49
88,98
88.76
88.93
88.38
88.30
89.13
88.56
88.23
89.26
Key observations. Two main conclusions follow from these comparisons:
1. LLM-based generation effective but costly and unstable. Single-shot LLM outputs (LLM-1) 1) highly variable and frequently fail to produce consistently hard sequences. Iterative prompting (LL.M-5) significantly improves stability and often surpasses RS, but remains substantially less effective than EDGE in most cases. Crucially, multi-turn LLM workflows are orders of magnitude slower (a single interactive round typically takes 2-3 minutes in our setup; five rounds commonly exceed 10 minutes), making them impractical for large- scale or low-latency evaluation.
Sampling- caneling-based methods find some hard eases but lack transferability. AS and MS are computationally efficient and can locate sequences that are challenging for particular algorithms. However, the difficult sequences they discover frequently exploit idiosyncrasies of a single target method and do not generalize across the range of CIL approaches evaluate. By contrast, EDGE identifies extreme sequences that consistently increase difficulty across many methods, achieving better balance of effectiveness and generality.
Overall, these results show that while alternative strategies can occasionally produce challenging se- quences, EDGE provides the most reliable combination of performance, generality, and computational efficiency for discovering extreme task orders.
D.5 DISCUSSION ON THE NUMBER OF SAMPLES
In previous sections we compared EDGE with the standard CIL evaluation protocol that uses RS by drawing three random task orders. In this subsection simulate scenarios with an increased number of RS samples in order to investigate how RS-based evaluation behaves as the sample budget grows, and to further demonstrate the practical advantages of EDGE.
Distribution estimation in the enumerable setting The setup is described in Section 5.1 Figure A6 visualizes how the evaluation outcomes change for RS and EDGE the number of sampled sequences increases. From these experiments we draw two main observations:
1. If only RS sampling is increased, RS reaches the distribution estimation quality produced by EDGE with three EDGE samples after roughly five to six RS samples. Note that the total sequence space in this experiment is only 90 sequences. In this limited space RS therefore requires about twice many random sequences to match the estimation quality of EDGE.
30

---

Published as a conference paper at ICLR 2026
2. If increase the number of samples for both RS and EDGE, EDGE remains superior throughout. As the sample counts grow, the two procedures tend to converge, and this convergence typically occurs when the number of samples is on the order of ten to twenty sequences.
Di5e-Pomp--IDG
0.8
15
0.5
0.4
10
4
0.4
0.2
0.2
0.1
10 20
samples
Number samples InageNet-R-1SD
Number samples (d) ImageNet-R-WW
(a) CIFARI00-ISD
(b) CIFARICO-W
Figure A6: Evolution of RS and EDGE performance the number of sampled sequences increases on CIFAR-100 and ImageNet-R. Circles denote RS data points and triangles denote EDGE data points. Different colored curves correspond different pretrained continual learning methods.
Extreme-sequence capture in classic CIL settings The setup for these experiments is described in Section 5.2 In classic class incremental learning settings the class space is typically very large, which prevents us from directly estimating the full performance distribution. Accordingly, we evaluate an evaluation protocol by its ability to capture extreme sequences. We performed focused empirical study on the CUB dataset using two representative methods: L2P, which exhibits a wide performance range sequences, and EASE, which exhibits relatively narrow performance range.
Figure A7 presents the empirical distributions obtained by repeated RS sampling together with the follows:
single-shot EDGE positions. The specific observations
For L2P, under our setup EDGE found hard-case accuracy of 58.5 and an easy-case accuracy of 72.3. We ran 600 RS samplings. Only 6 of those RS samples produced lower accuracies, with the minimum RS accuracy equal to 57.55. None of the RS samples attained accuracy higher than EDGE.
For RASE, under setup EDGE found hard-case accur acy of 84.29 and easy-case accuracy of 85.37. After more than 400 random evaluations, only RS samples produced lower accuracies, with minimum RS accuracy equal 84.09, Only one RS sample achieved a higher accuracy, equal to 85,41.
These results confirm the intuitive fact that increasing RS sample count improves RS. However, such improvement typically entails large evaluation cost. Compared with EDGE, discovering extreme sequences using only RS is both time consuming and inefficient. The empirical evidence therefore supports the practical value of EDGE as a more sample efficient and reliable procedure for identifying extreme task orders.
Remarks All experimental details and plotting scripts are provided in Appendix E.5. The plots in AigureAAGan and Figure/A7]super support the conclusion that EDGE achieves comparable or better distribution estimation with far fewer samples than RS, which reduces evaluation cost and improves the reliability of worst-case and best-case assessments.
E How TO USE THE EDGE REPOSITORY
We release official implementation of EDGE for reproducing the proposed evaluation protocol in Unlike earlier versions that only provided pre-generated class orders, the current repository integrates EDGE directly into two widely used CIL toolboxes, enabling users to switch between the standard random-order evaluation and the EDGE protocol via a unified command- interface.
httpp::/ thw..///ttttuttoc/ com IGNIAI I /DGG
31

---

Published as a conference paper at ICLR 2026
58.5
73.3
84229
85157
50
60-
40
Count
40-
Count
30
20-
20-
10
57.5
60.0
67.5
70.0
72.5
84.25
84.50
84.75
85.25
Accumac
Accumcy
(a) Distribution of accuracies for 600 random sequences under The vertical black line marks accuracy position obtained by single EDGE sample.
(b) Distribution of accuracies for 400 random sequences under EASE. The vertical black line marks the accuracy position obtained by a single EDGE sample.
Figure A7: Random sampling distributions and EDGE single-shot positions the CUB dataset. Left panel corresponds to L2P and right panel corresponds to EASE.
E.1
REPOSITORY LAYOUT
The repository contains two integrated codebases:
PILOT//: an EDGE-integrated fork based the PILOT toolbox.
PyCIL/: an EDGE-integrated fork based the PyCIL toolbox.
In both integrations, EDGE is implemented as modular evaluation component and can be invoked without changing the training recipe or model definition. This design keeps EDGE largely decoupled from the training pipeline, making it straightforward to apply to different methods supported by the toolboxes.
E.2 INSTALLATION
We recommend creating a dedicated environment and installing the dependencies required by the corresponding toolbox. addition to common CIL dependencies (PyTorch, torchvision, numpy, tqdm, etc.), EDGE requires the CLIP package for computing the class- name similarity used by the protocol.
git clone betps://githieg cOm/AIGNIAT/EDGE od EDGE
conda create edge python=3.10 -y conda activate edge
pip install torch torchvision
pip install git+https://githaett com hwtp:a/.cco/pppe//cici. git
pip install scipy
pip install timm==0, 6. pip install tqdm
E.3 RUNNING: RANDOM PROTOCOL vs. EDGE PROTOCOL.
EDGE is exposed via single command flag (denoted as -eval in the repository usage guide) to select the evaluation protocol:
eval random: the standard random class-order protocol (randomly sampled class order- s/seeds).
32

---

Published as a conference paper at ICLR 2026
-evall edge: the proposed EDGE protocol (adaptive sampling with extreme sequences to better approximate the performance distribution boundary).
typical invocation is:
Run with EDGE protocol
python main.py py --config //epp// [MODEL_ json --eval edge
Run with standard random-orde protocol python main. py --config /exps/ (MODEL [AMM)] json --eval random
Notes. (1) Config files. The -confi g argument points to the experiment configuration provided by the toolbox (e.g., model, dataset, task size, memory budget, optimizer, and schedule). EDGE does not require modifying these training settings. (2) Fair comparison. For fair comparison between protocols, keep the training configuration fixed and only switch -eval. you use multiple seeds/orders under the random protocol, we recommend using the same overall evaluation budget when running EDGE. (3) Outputs. The scripts report the standard CIL metrics supported by the toolbox (e.g., average incremental accuracy). Under EDGE, the evaluation additionally emphasizes boundary-aware statistics by considering extreme sequences, which helps reveal the variance and worst/best-case behaviors that can be under-estimated by random sampling.
E.4 PRACTICAL RECOMMENDATIONS
We recommend reporting results under both protocols:
Random protocol for compatibility with prior work and standard leaderboard settings;
EDGE protocol to provide boundary-aware evaluation and more informative characteriza- tion of performance variability across class orders.
This combined reporting provides clearer picture of method's robustness under realistic order distributions and mitigates the risk of drawing conclusions from a small number of random orders.
Dataset support. The current release provides built-in support for CIFAR-100, CUB-200, ImageNet-R, and ImageNet-A. To extend EDGE to additional datasets, users need to regis- ter the corresponding class label names (used compute CLIP text-embedding similarities) in utils / /edge. py. Concretely, add the dataset-specific list/dictionary of class names in the same format as the existing entries, and ensure dataset identifier your configuration matches the key used in utils 1s/i///ee /edgee
33


Reviews:
[{"review_id": "YDDQTcsBHu", "reviewer": "Reviewer_Htje", "summary": "This paper discusses the evaluation protocol of the class incremental learning and shows that the widely used random sampling protocol fails to capture the full distribution of model performance. To solve this problem, the authors propose an evaluation framework, EDGE, that finds extreme class sequences based on inter-task similarity computed via CLIP text embeddings. Both theoretical analysis and empirical experiments demonstrate that EDGE offers a more accurate estimation of performance distribution on CIFAR-100, CUB-200, and ImageNet-R, EDGE better captures upper/lower bounds and reduces distribution distance versus RS, remaining robust across different model backbones and CLIP encoder sizes.", "questions": "1) The paper emphasizes that RS leads to inaccurate variance estimation and poor coverage of extreme cases, and that EDGE improves the estimation of the performance distribution. While this is well motivated, it raises a fundamental question about what the community should prioritize in continual learning evaluation. In many continual learning scenarios, our focus may be more on relative model ranking rather than on absolute accuracy metrics.\n\n2) The proposed framework relies on the assumption that semantic similarity between class labels, derived from CLIP text embeddings, correlates with inter-task similarity and reflects task difficulty. How robust is this assumption beyond natural image datasets? In domains such as medical imaging or synthetic datasets, label names may inadequately represent visual or distributional similarities between tasks.\n\n3) The framework may be vulnerable to data contamination, especially since both the evaluated continual learning (CL) methods and the CLIP model used for task similarity computation are pre-trained on extensive web-scale datasets. Have the authors considered how data contamination—resulting from either model pre-training or benchmark overlap with CLIP—could affect task similarity estimates and, in turn, impact the reliability of the EDGE protocol?\n\n4) When evaluating continual learning methods that are pre-trained (e.g., on CLIP or similar datasets), it's important to note that if the downstream datasets overlap with the pre-training corpus, task difficulty may be misrepresented. Tasks with greater overlap might seem \"easier\" not due to actual similarity, but because of data leakage from the pre-training phase.\n\n5) How does RS improve with larger sample sizes? Specifically, how many RS sequences are required for RS to match EDGE in distribution approximation or ranking stability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper addresses the limitations of evaluation protocols in CIL, a critical area gaining increasing attention.\n- The theoretical analysis is clear and robust, highlighting the fundamental limitations of random sampling (RS) protocols.\n- The proposed method is straightforward and easy to implement. Utilizing CLIP-based label embeddings to evaluate inter-task similarity appears appropriate for most tasks.\n- The comparison of RS and EDGE is analyzed from various perspectives, emphasizing the limitations of random sampling.\n- Both non-pretrained and pretrained CIL methods were assessed using two evaluation protocols (random sampling & EDGE), offering valuable insights for researchers.", "weaknesses": "- Using CLIP text embeddings for class similarity in non-natural image datasets, like medical imaging or remote sensing, may lead to inaccuracies due to potential misalignment between label semantics and visual similarity.\n\n- The comparison relies solely on random sampling, without addressing other potential baseline evaluation methods. \n\n- The assumption that inter-task semantic similarity perfectly correlates with task difficulty may not be universally valid, particularly in areas where semantic similarity fails to encompass the full complexity of tasks.\n\n- The application scope of this evaluation protocol requires further discussion. Specifically, how crucial is it to capture performance distribution and absolute accuracy compared to solely focusing on relative model ranking? Is this type of evaluation necessary for all continual learning scenarios, or is it primarily relevant for certain deployment-critical applications?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8VaATCmacT", "reviewer": "Reviewer_6DZz", "summary": "This paper focuses on the evaluation protocol of continual learning. Under the standard evaluation protocol, Random Sampling (RS) tends to be unfair, as it emphasizes the mean rather than the deviation. Consequently, the EDGE protocol is proposed for improved sampling of task sequences. First, the author identifies that the difficulty of the continual learning (CL) sequence is highly correlated with inter-task similarity. Therefore, EDGE samples \"easy\" and \"hard\" sequences based on the similarity calculated using the CLIP model. Experimental results show that EDGE provides a better estimation of the performance range of CL methods.", "questions": "For the paper to be deemed acceptable, I believe the following question is crucial:\n1. How do Random Sampling (RS) and EDGE perform with an increased number of sampled runs?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed relationship between task similarity and difficulty is intuitive and interesting.\n2. Under the existing setup (3 runs for each method), the proposed EDGE can be much stronger compared with RS.\n3. The source code is partially included in the submission.", "weaknesses": "1. It is true that recent continual learning (CL) papers consistently use Random Sampling (RS) in their evaluations. However, the experimental setting of this manuscript assumes that only three runs are used for the evaluation, which does not fully simulate a realistic Class Incremental Learning (CIL) evaluation protocol. According to basic statistical theory, the estimation provided by the RS protocol will be much closer to reality if more runs are employed. As this paper focuses on the fairness of the evaluation protocol, it is essential to consider scenarios where the number of runs exceeds three for a factual and equitable comparison, and to examine how performance differences scale with an increased number of runs.\n2. EDGE fails to account for cases where analytic continual learning methods are evaluated, such as GACL [A] and RanPAC. The performance of these methods is usually minimally impacted by task order. This may explain the similarity in RS evaluation for RanPAC and the EDGE evaluation, despite the fact that RanPAC fine-tunes the backbone using the training samples.\n3. As an evaluation protocol, EDGE only takes final accuracy into consideration. However, it is well known that final accuracy is not the sole important factor. To evaluate the dynamics of the CL learner, metrics such as anytime inference [B] and plasticity/forgetting metrics are also vital, yet these are unfortunately not discussed within the proposed metrics.\n4. Methods such as ConvPrompt [C] and SEMA [D] take inter-task similarity into account. Including such methods in the benchmarking would be beneficial.\n\nReferences:\n\n    [A] Zhuang, Huiping, et al. \"GACL: Exemplar-free generalized analytic continual learning.\" Advances in Neural Information Processing Systems 37 (2024): 83024-83047.\n    [B] Koh, Hyunseo, et al. \"Online continual learning on class incremental blurry task configuration with anytime inference.\" arXiv preprint arXiv:2110.10031 (2021).\n    [C] Roy, Anurag, et al. \"Convolutional prompting meets language models for continual learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n    [D] Wang, Huiyi, et al. \"Self-expansion of pre-trained models with mixture of adapters for continual learning.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sNBX8PEky9", "reviewer": "Reviewer_rrMk", "summary": "This paper critiques the current evaluation methodology used in class-incremental learning (CIL), where a small number of randomly sampled class orders are used to estimate model performance. The authors argue that this common practice does not accurately approximate the true expected performance across all possible class sequences. To address this, the paper proposes an improved evaluation approach designed to more reliably estimate true model performance.", "questions": "- For experiments using pre-trained models, what dataset is used to pre-train the backbone?\n- Why are RS and EDGE results very close in Table 1? Does this imply that EDGE only offers minor improvements over RS in most realistic cases?\n- Can methods that perform well on difficult sequences also automatically perform well on easy sequences? If not, what may be the reason behind it?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper investigates an evaluation issue in CIL that is often overlooked in the literature, which is an interesting direction for the CL community.\n\n- The paper is clearly written and well structured, with both theoretical and empirical analysis supporting the proposed evaluation method.\n\n- Highlighting the impact of class ordering contributes valuable insight toward more trustworthy benchmarking in CIL.", "weaknesses": "- The motivation of the paper raises concerns. The introduction uses real-world scenarios (e.g., self-driving cars) where the learner does not have control over class arrival order. However, the proposed method assumes class ordering can be controlled or enumerated for evaluation, which conflicts with the motivation.\n- Although the common RS (Random Sampling) evaluation does not estimate the true model accuracy, it does enable fair comparison among methods if all algorithms are evaluated using the same sampled sequences. Since CIL benchmarks are used for comparison of CIL algorithms rather than real-world deployment, the evaluation of existing literature is not entirely unreasonable.\n- A key practical question would be whether a method that performs well on difficult sequences also performs well on easy sequences. A more actionable direction may be benchmarking both cases and identifying algorithms that consistently perform well regardless of class order.\n- The existence of hard vs. easy class sequences is already known in CIL. Thus, the novelty of the paper is mainly limited to proposing a new evaluation metric. Without demonstrating an algorithm that improves under the new metric, the contribution may feel incomplete.\n- The experiments show limited benefit in the practical application of ranking existing CL algorithms (referring to Table 2).  \n  - Example: CIFAR-100 (hard sequence): both True and RS rankings identify iCaRL as best. Here, EDGE does not contribute any new information in terms of ranking existing CL algorithms\n  - Example: ImageNet-R (easy sequence): True says iCaRL as best, but RS and EDGE disagree. Here, neither estimator consistently reflects the true ranking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BpJ7N5YHFv", "reviewer": "Reviewer_Qscv", "summary": "This paper critiques the evaluation protocol in Class Incremental Learning (CIL), which relies on averaging results over a few random class-order sequences. The authors theoretically and empirically demonstrate that such Random Sampling (RS) severely underestimates variance and produces biased means due to the factorial explosion of possible class sequences. To address this, they propose EDGE (Extreme case–based Distribution & Generalization Evaluation), a new evaluation protocol that identifies easy and hard cases using inter-task similarity derived from CLIP text embeddings. They claim that, evaluating models on these extreme and median sequences better approximates the true performance distribution.", "questions": "1. How robust is the inter-task similarity heuristic across modalities or non-visual datasets? Could EDGE generalize beyond CLIP-based text embeddings?\n2. For large-scale settings where enumerating even three sequences per dataset is costly, how would EDGE be adapted or approximated?\n3. Does emphasizing extreme sequences risk over-penalizing models optimized for stability rather than adaptability?\n\nI am willing to change my review if the concerns are addressed.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Challenges a long-standing but under-scrutinized assumption in CIL benchmarking, introducing a valuable distributional perspective.\n2. Provides clear theorems quantifying the inadequacy of random sampling.\n3. Demonstrates consistency across model types (pre-trained and non-pre-trained) and multiple datasets.", "weaknesses": "1. Lack of real-world validation: Although the authors repeatedly motivate their work through real-world continual learning scenarios (e.g., autonomous driving, uncontrolled class emergence), all experiments are performed on simple disjoint data streams. They do not include any real-world or realistic task-order settings, such as those in Real-World Continual Learning[1] or I-Blurry[2]/Si-Blurry[3] setups, which explicitly simulate gradual or overlapping task transitions.\n2. Dependence on CLIP embeddings: The approach assumes that CLIP’s text-space similarity correlates with inter-task difficulty, which may not hold in all domains (e.g., non-visual or specialized data).\n3. Edge cases unexplored: The paper could discuss scenarios where inter-task similarity fails to predict performance variance (e.g., highly regularized models).\n4. Overlooked fairness implications: Focusing on extreme sequences might overemphasize worst-case behavior, potentially penalizing stable models that perform consistently across moderate task orders.\n\ncitations: \n[1]Online Class-Incremental Learning For Real-World Food Image Classification (Siddeshwar Raghavan, Jiangpeng He, Fengqing Zhu)\n[2] Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference (Hyunseo Koh, Dahyun Kim, Jung-Woo Ha, Jonghyun Choi) \n[3]Online Class Incremental Learning on Stochastic Blurry Task Boundary via Mask and Visual Prompt Tuning (Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, Gyeong-Moon Park)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dx81MMdIlq", "reviewer": "Reviewer_SxFj", "summary": "The paper is about fixing misleading evaluation in class incremental learning (CIL). The authors propose EDGE, an extreme-case-based protocol that samples hardest, easiest, and random class sequences guided by inter-task similarity computed from CLIP text embeddings.", "questions": "In the EDGE protocol design section, it mentions that the extreme sequence generation algorithm uses CLIP text encoding to construct class similarity matrices. Could you provide more details on how this process works and how it contributes to generating high and low similarity sequences? Additionally, what are some potential limitations or challenges associated with using CLIP for this purpose?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper reframes CIL evaluation from mean/variance over a few random orders to estimating the full performance distribution.\n2. EDGE remains effective across datasets (CIFAR‑100, CUB‑200, ImageNet‑R), backbones (ResNet, ViT), and CLIP encoder sizes.\n3. This paper is fluently written and easy to read.", "weaknesses": "1. Missing baselines.  RS is treated as a weak baseline (3 sequences with fixed seeds), while more principled sampling exists (stratified, Latin hypercube, coverage maximization, Bayesian optimization).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)