Boosting Multiple Views for pretrained-based Continual Learning
Authors: Quyen Tran
Tung Lam Tran
Khanh Doan
Toan Tran
Dinh Phung
Khoat Than
Trung Le
Venue: ICLR 2025 Poster (2025)
Keywords: continual learning
ViT-pretrained continual learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Recent research has shown that Random Projection (RP) can effectively improve the performance of pre-trained models in Continual learning (CL). The authors hypothesized that using RP to map features onto a higher-dimensional space can make them more linearly separable. In this work, we theoretically analyze the role of RP and present its benefits for improving the model’s generalization ability
in each task and facilitating CL overall. Additionally, we take this result to the next level by proposing a Multi-View Random Projection scheme for a stronger ensemble classifier. In particular, we train a set of linear experts, among which diversity is encouraged based on the principle of AdaBoost, which was initially very challenging to apply to CL. Moreover, we employ a task-based adaptive backbone
with distinct prompts dedicated to each task for better representation learning. To properly select these task-specific components and mitigate potential feature shifts caused by misprediction, we introduce a simple yet effective technique called the self-improvement process. Experimentally, our method consistently outperforms state-of-the-art baselines across a wide range of datasets.

Published as a conference paper at ICLR 2025
BOOSTING MULTIPLE VIEWS FOR PRETRAINED-BASED CONTINUAL LEARNING
Quyen Tran't Lam Tran Khanh Doan', Toan Tran', Dinh Phung?, Khoat Than?* Trung Le
Qualcomm Al Research Hanoi University of Science and Technology
Monash University
ABSTRACT
Recent research has shown that Random Projection (RP) can effectively improve the performance of pre-trained models in Continual learning (CL). The authors hypothesized that using RP to map features onto higher- dithee-admensionll space can make them more linearly separable. In this work, we theoretically analyze the role of RP and present its benefits for improving the model's generalization ability in each task and facilitating CL overall. Additionally, we take this result to the next level by proposing Multi-Viee Random Projection scheme for stronger ensemble classifier.
particular, we train a set of linear experts, among which diversity is encouraged based on the principle of AdaBoost, which was initially very challenging to apply to CL. Moreover, we employ a task-based adaptive backbone with distinct prompts dedicated each task for better representation learning. To properly select these task-specific components and mitigate potential feature shifts caused by misprediction, we introduce simple yet effective technique called the self- eelf-improvement process. Experimentally, our method consistently outperforms state-of-thed tteh-erttttt baselines across a wide range of datasets.
INTRODUCTION
Continual Learning (CL) is a field of Machine Learning that focuses enabling deep neural networks to learn continually from a sequence of tasks with different data distributions. Sequential training on such data often leads to catastrophic forgetting (French) 1999) of old knowledge, where model parameters are overwritten by new tasks's learning. To overcome this problem, recent CL methods proposed leveraging the generalizability of pre-trained models (Han et al. 2021 Jia al. 2022) as
frozen backbones to continually solve the series of emerging tasks overtime Janson et al. 2022
Wang et al. 2022c Zhou al. 2023 Smith et al.) 2023 Li et al.) (0224).
Among these pretrained-based CL methods, one promising work is RanPAC (McDonnell et al. 2023), which is based on Random projection (RP) facilitate the classification ability of models. In particular, this method proposes using a random matrix to project the features obtained from pre-trained backbone onto a new space with significantly higher dimensions (e.g., from d 768 to d' 10, 000). The motivation for utilizing RP is based on their empirically supported but not explicitly theoretically proven hypothesis, that features are more linearly separable and easier to classify in higher-dimensionn by a nonlincar activation function, showing that the margin of a training data point in the higher- dimensional space is almost surely larger by the rate of D(vd)) This means that RP increases the (nonlinear) separability between classes and tasks. We also analyze RP's implications regarding generalization ability for each specific task and facilitating CL overall.
space. In work. theoretically explain the roles of RP followed
In addition, take advantage of RP for enhancing separability, straightforward solution is to set the dimension of the projected space (i.e., d') to be significantly large. However, this naive idea would be infeasible training the classifier involves computing the inverse of a Gram matrix of size d' x d'. To overcome this obstacle while leveraging the multiple-view learning paradigm for
Equally contributed.
* "Qualcomm Viemam Company Limited

---

Published as a conference paper at ICLR 2025
stronger classifier, we instead consider handling K projected spaces with smaller manageable dimensions and employing K corresponding projection matrices. We refer to each space as an atomic view (i.e., d²). During training, the classifiers these views will be encouraged to diversify through exploiting the principle of Adaboost (Freund & Schapire 1997) in CL for the first time. We also huge view, which is composed of k S K atomic views,
theoretically argued that computations could be conducted through similar operators on these smaller views by some relaxation. Therefore, given the responses from K atomic views, we can generate up 2 1 huge-view responses and implement a voting strategy to obtain the final prediction, harnessing the potential of the boosting algorithm in CL.
Furthermore, we argue that the first task adaptation strategy (Panos et al. 2023) in RanPAC, while bridging the gap between pre-trained data and CL. tasks, may cause overlapping representations from later tasks, potentially limiting model performance as subsequent tasks may differ from the first. To effectively adapt incoming tasks, we follow prompt-based CL. methods (Wang et al., 2022c; [Smith et al. 2023), assigning each task to different prompts and utilizing the prompted model to obtain features for RP. However, we observe that occasionally, incorrect prompt selection can lead to feature shifts between the training and testing phases, resulting in poor performance. Therefore, to mitigate the risk of selecting the wrong prompt, we propose a self-im eefrovmppeeeentt process for appropriately selecting prompts for each task sample.
Finally, we name our method as "Boosting Multiple Views for pretrained-based Continual Learning (BoostCLL)" and summarize our main contributions follows:
We theoretically analyze the benefits of random projection (RP) mapping representations onto higher-dimensional space, which explicitly unsolved in the previous work. We show that RP increases the margin of each training instance, making features more separable. Besides, RP can increase a model's generalization ability for each specific task and facilitate CL. overall.
Based on our theoretical results, to handle vastly nigh-dimensional space, we propose a novel Multi- View random projection scheme for training a strong ensemble classifier. This scheme involves leveraging an adaptive AdaBoost strategy, which initially poses a significant challenge when applied directly to CL.
Additionally, together with integrating task-specific prompts into the CP-based backbone for better adaptation, we propose a self-improvement process, simple but effective strategy to help select prompts more accurately when inference.
We empirically evaluate the effectiveness of our method against current state-of-the-att pre-trained -e--raiied--bsedd baselines across various benchmarks. Specifically, denoting BoostCL- m(d': our approach with K views and d' dimensions for each view, consider the following variants: BoostCL-m (d' = 768;K K = 13), BoostCL-m (d' = 10, 000; = 15), and BoostCL-m (without random projection; K 15) for comparison with RandAC and other baselines. Our variant BoostCL-m (d' = 10, 000:K = 15) outperforms all baselines by a significant margin. Interestingly, BoostCL-m (d' 768:K 13), which uses the same random- dimension as RanPAC, also surpasses the baseline by remarkable margin, showcasing the effectiveness of our multi-view learning strategy. Notably, the variant BoostCL-m (without random projection;K = 13), applying multi-view without random projection, also significantly outperforms RanPAC and other baselines, reinforcing the value of our multi-view learning approach.
2 RELATED WORK
Class Prototype-hased approach leverages pre-trained models and introduces prototype-based strategies for continual learning (CL.. In particular, the nearest class mean (NCM) classifier applied
to prototypes from frozen pre-trained models is a strong baseline (Janson et al. 2022). Later, ADaM (Zhou et al. 2023) refines this by adapting the pre-trained model to the first task with techniques
like tuning (Jia et al., (022) or adaptors (Chen et al., 2022). Building on this, recent work
(McDonnell et al. 2023 Goswami et al. 2023) Zhuang et al. 2023 ZHUANG et al. 2022) further enhance NCM by using second-order statistics, resulting in ad advanced classifiers with closed-for
solution.
2

---

Published as a conference paper at ICLR 2025
Prompt-based approach. Wang et al. (2022d c Smith et al. (2023) typically assigns set of prompts to each task, enhancing the adaptability of the backbone to downstream tasks and enabling the ability to distinguish classes within each task. However, the absence of explicit constraints
lead to feature overlapping between classes from new and previous tasks. Therefore, recent methods employ some types of contrastive loss (Wang al. 2023 Li et al. 2023) utilize Vision Language models (Wang et al. 2022a; Nicolas et al. 2024) to better separate features from tasks.
Although the methods in these two approaches achieve impressive performance, they only consider classifiers with a single view, resulting in a limited performance. To create a stronger classifier, we propose a novel Multi-view Random Projection scheme with an ensemble classifier, including many diverse views driven by the principle of Adaboost (Freund & Schapire (997), where responses from sub-classifiers of different views are then combined and voted on to produce the final prediction.
BACKGROUND
3.1 PROBLEM FORMULATION
We consider Continual Learning setting, where a model learns from a sequence of T classification tasks without revisiting old task data during training task II) access during inference. Each task t E {1,...,7) 7} has dataset Dr containing ne i.i.d. samples (ar), v)) of Yd classes
In this work, we design our model with the following components: a pre-trained ViT backbone f.) parameterized by Φ; a set of frozen random projection matrices {U*} and U e Rdxd' (d' d). where K is the total number of atomic views, d and d' are the dimensions of original and projected latent space, respectively; and two classification heads with weight matrices W, W for class predictor and prompt predictor, respectively. Similarly to other prompt-based works, we incorporate a set prompts P into fa and denote the backbone after incorporating the prompts as fa.P.
3.2 THE CLOOSE-FORM SOLUTION FOR THE CL CLASSIFIER OF RANPAC
RanPAC (MeDonnell et al. 2023 is CP-based method that proposes using random projection matrix U Ra to project feature vector fo (z: E R of each sample onto significantly higher dimensional space and then apply a nonlinear activation function a to obtain representation = a(zU) E 1R Let X = {a:; }
be the set of samples up to task / where N = -Ei= n4-
The corresponding projected representations and labels are Ž E N and Y E N.N.KK Ks, where Ke is the total number of classes up to task t. This method aims to learn linear classifier W Rd' that can obtain knowledge from all tasks so far without forgetting by solving the following problem:
min Zw Y13 W
W
The closed-form solution of is W (Z" - which can be updated incrementally across the arriving tasks. Specifically, let G. = ZT E Rd xd' and C, = Z E Rdd x, be the Gram matrix and class prototype matrix computed at the end of task I, respectively. We have:
. G: = G 2,- 2, where G 2 2,, and 2, is projected representations from task i.
- + Y where C₁ 2 and Y, is one-hot vector labels from task i. #T (as(Gr (GT + -I)--cy, where Cy
After task T, to classify Biest, we simply perform: ypreat = argmax,, test is the nh column of CT.
3.3 THE BOOSTING PRINCIPLE OF ADABOOST/SAMME
AdaBoost (Freund & Schapire 1997 and SAMME (Hastic et al. 2009) are ensemble learning methods that combine multiple weak classifiers {9 (-))k to create strong classifier G(.) by sequential training weighted training data D = [[x,, p These methods focus misclassi- fied examples in each iteration l to improve the model performance iteratively. Directly applying AdaBoost/SAMMEE to CL poses significant challenge because the data for all tasks is not always available, and the computational cost for a sufficiently large number weak learners is infeasible. Please refer Appendix B for more details.
3

---

Published as a conference paper at ICLR 2025
4 FRAMEWORK
In this part, we first analyze the theoretical role of RP and its implications (Section 4.1). Then, we introduce our proposed Multi-view Random Projection strategy for our classifier (Section 4.2). Finally, we explain how we adapt the CP-based backbone to new tasks and select task-specific components to improve accuracy (Section
4.1 RANDOM PROJECTION ONTO MARGIN, GENERALIZATION AND CL OVERALL
DIMENSIONAL SPACE CAN IMPROVE INSTANCE
In RanPAC McDonnnel et al. 2023 the authors hypothesized that performing a random projection (RP) onto higher- digher-dimensionll space can help the transformed features become more linearly separable. Nonetheless, they did not provide explicit proof, leaving the motivation and the hypothesis unsolved. In this section, theoretically analyze the role of RP and show its vital implications.
Improving margin. According to Sokolić et al. (017). given a classifier g. the margin of an instance (z. y) can be defined as (s. g, ] sup {v/ : 1 2111 S = g(z") ) = y, Vz E Rd). When
taking an RP with a transform matrix U E Rdxd" each instance s E R x y will have a projection s" = (2TU.y) E IR" x V, where y is the label space.
Given training set S, we define the margin of g over the training set S as 7(S, ".,g,,R, R²) min.es (ss, g, IRd). The training set S over Ed induces the training set S" over IRdd through the transform matrix U. Similarly, we define the margin over S" According to /Sokolć et al. (2017) and our Corollary A.3 in Appendix A classifier with higher margins over a training set S generalizes better to general distribution from which instances in S are sampled.
To see the benefits of RP, we propose the concept of Bayes margin measure the highest margin of the classifiers in hypothesis family.
Definition 4.1 (Bayes margin). Let H be the set of measurable functions that map from TRd to V. The Bayes margin over the training set S is defined (n,S H) suPyeR (S, g, R*).
Note that H. is large enough to cover all hypothesis spaces often used in machine learning. Using a smaller space may not correctly reflect the role of RP. Similarly, let H' be the set of measurable functions that map from Rd to y. The Bayes margin of 5" is: "inn "in(S", H') supheH 7S" h, R.). We have the following result about the role of RP (see Appendix [ proof and more discussions).
Theorem 4.2. Let U be random matrix of size d' x d, with d' >> d, whose elements are independent copies of a Gaussian random variable with unit variance and O meun. Consider s = (z, y) and its projection s" (o(2² U), y) where the element-wise nonlinearity is invertible and expansive For every (0,1), 1). with probability at least 1 (C.e)d' -c.d where C and e are some positive constants, we have:
Rim H') 2 76n (S, (S. Vd
According to this theorem, the Bayes margin of sample in the new space can increase at rate of O( O(Vd', ,, thus potentially enhancing generalization ability. This property of RP is intriguing and has never been known before. As discussed next, this property should lead to significant consequences for each task of interest and task sequence.
Implications to generalization ability and CL. Denote Zelass and 2laok as the sample domains whose members belong to class c, and task respectively. The Bayes margin of each class e is defined min Tia (ri (S, H) S 3 (z, c), z E Zeluse]. The Bayes margin of a task is defined (7in (S, H): S 3 = (z.v), E Z.ask3. The following implications readily follow:
as Yelass,
as "tusk,
RP can increase generalization ability for a task. As shown in Theorem 4.2 the margin of every instance increases in the new space, resulting in Telass.c and 7mskkt increase rate of O(vd'). Thus, classes in the new space become more separable. Moreover, there
A function a is expansive if it satisfies (a(x) - ov)) 2 |r y/ for all æ. y in their domain. Note: ReIU can work well probably because it approximates functions that satisfy the two conditions above, For example, axx) = x E log(1 + exp(x)) where & R, which is used in our experiments (see Appendix G.4
4

---

We have Published as a conference paper at ICLR 2025
is
connection between the margin and the generalization ability of model (Bartlett et al. 2017; Sokolić et al.) 2017). Consequently, the generalization error for each task f will be smaller. This means the trained model can generalize better unseen data. More discussions about this aspect can be found in Appendix[A]
RP can increase lask separability and CL overall. As discussed above, the margin T for each task increases, indicating rapid task separability in IR This improvement in task separability is crucial in CL. aiding task detection processes. Coupled with enhanced generalization ability for each task, this provides a theoretical basis for why RP effective
in CL. This is consistent with previous observations (Kim et al. 2022 Wane al 2023)
4.2 MULTIPLE-VIEW RANDOM PROJECTION SCHEME
lugh view (U,W)
4.2.1 THE MOTIVATION
(A)
atorric siere
As shown experimentally in RanPAC and theoretically proven in Section 4.1 projecting feature vectors onto a nigher-dimensional space will make representations more separable. Therefore, to improve the model's performance, naive workaround could be increasing the dimension d' as large as possible. However, this idea unexpectedly infeasible since it requires inverting Gram matrix of size d' x d' (See Section 3). To enable learning in a huge di- mensional space and take advantage of the multiple-vin view learning paradigm for a stronger classífier, we consider K projected spaces with smaller, manageable dimensions (i.e., K single atomic views), which all share the same frozen backbone, and each view is generated using a ran- dom matrix U*, k E f1,..., K}.
W
Browtny
d2
Sooeng
27
W
pvle) /y
In what follows, we demonstrate that we select k S atomic views (d,,) and then combine them to produce a huge view with a higher dimension (e.g., d' = krd"' ), the computation of the original huge view can be significantly simplified via operations on these corresponding atomic Ji=1
(B)
Took
Took
Atamic view
A
Atamic view
M
A
views. In particular, let G, 2 E Ne and { (each Z' E IR N.N') are Gram matrix, representation matrix on a huge view and corresponding atomic views respectively. For each huge view, we train linear expert W = (G + -IT,, where Z [2 k=
Figure I: (A) Multiple-view strategy: in each task, atomic views are trained in turn, 1tth view is trained fix the error of the previous one via sample weight A Then, the output of huge view is represented via the answers of l atomic views, where {a*) (=²))= are aux- iliary weights of these views obtained from training. (B) The process of up- dating sample weight over CL tasks.
the theorem below:
Theorem 4.3. we approximate the Gram matrix G ZTZ by block matrix G diag( Zi T 7i) then the optimal linear classifier on the huge view W
[T 1 - Y with G = (Z Zt E "TR'''KKK is the corresponding optimal linear classifier on the 7th atomic view.
(W*yTiT E RRdd' x.e where W (G
Theorem 4.3 states that under some relaxation, the optimal solution over the huge view can be decomposed into the optimal one for each atomic view. Moreover, in Appendix C.3 we prove that (G + -I can approximate well (G + AT) when À is sufficiently large. This theorem provides an approximate and effective solution to compute the optimal linear expert for each huge view based on the respective atomic views. Furthermore, the following corollary specifies how the prediction of the linear expert in a huge view can be represented by those atomic views.
Corollary 4.4. Consider a sample x with the representation in the huge view is = [ where ( z is the representation in the ith atomic view. Let (æ, v) 2W be the huge-view probability prediction vector. Then s" (x. = Eh_, sxxx (x where s² (x. v) specifies the corresponding response in the tth atomic view.
5

---

Published as a conference paper at ICLR 2025
This corollary provides convenient way make huge- view prediction using atomic- responses. With K atomic views, we can create up to 2 1 huge-vies view answers. We then use these responses in voting strategy, which will be discussed in the next subsection.
4.2.2 LEARNING DIVERSE AND COMPLEMENTARY MULTIPLE VIEWS VIA THE BOOSTING PRINCIPLE
Based on the theoretical arguments in the previous sections, we propose producing the output of the linear expert in each huge view through those of the atomic views. However, we observe that naively training these classifiers separately does not improve the performance as they yield roughly the same outputs. Therefore, creating divergent and complementary linear experts in the atomic views is essential to obtain an efficient ensemble classifier.
Our main idea is that as tasks arrive, we continuously maintain K Gram matrices G.:K for K atomic views up to task f, which can be computed incrementally. Inspired by the AdaBoost principle, we aim for K classifiers the training process, even as more tasks are introduced.
these atomic views remain supplementary and complementary throughout
We now assume that we have K Gram matrices G7-1 for K atomic views up to the task / 1 and we need incrementally build up the Gram matrices G.K.K when task arrives In particular, we apply our mulriple-view training strategy to sequentially train a set of linear classifiers for K different atomic views upon the frozen backbone. Inspired by AdaBoost (Freund & Schapire 1997) and SAMME (Hastie et al. 2009), the ttth view (k > 1) will be learned to complement the previous one, using the sample weight vector A computed based on error rate of the (k - 1)th view (see Figure D. We describe this process for each task in more detail as follows:
For the first view:
First, we start with training the linear classifier gi for the first atomic view (k 1) in which all data points possess the same weight of 1 (i.e., A! = [1,... 1), following the rule from RanPAC (see Section 3) with the Gram malrix G G 2 diag( (At) 21 where 2 is the data of task t on the view k through the random projection matrix U².
After that, we compute error rate err! Σ. A 1 of (#t) and auxiliary weight of
E A -rr + log( 1) for this atomic view where ot is the linear classifier
log(
erry on the first view up to the task
We update the sample weight vector w.r.t task view t: 2 as follows:
A = diag (Lexp (at ( }] (ed..)))D))
A!
where 1 is the indicator function returning 1 if A is true and O.
We normalize the sample weight vector A so that its zero-norm is 1.
For the k-th view, which was learned based on the (A: 1)-th view:
We possess the sample weight vector A for the k-th view. We update the Gram matrix G = G 2," diag( A 2 which corresponds to the weighted linear classifier o w.r.t. the sample weights across the tasks A A Note that 2 represents the data of task t on the view ls through the linear random projection k
Next, compute error rate err
- AÑ, 1 (m/) and auxiliary weight of Ai, -err + log(()'} - 1) for this atomic view.
log( Em
a
We update the sample weight vector w.r.t task t, view + 1 as follows:
A diag ([exp fartt i ot (#:) } (oL.w))).))
A..
(3)
We normalize the sample weight A that its zero-norm

---

Published as a conference paper at ICLR 2025
It is worth noting that given the view k, we indeed train the weighted linear classifier ge with the sample vectors Ai on this view across the tasks from to t by solving the following optimization problem (OP):
min ddagg( A ) 2* w Y W
2
where A [A Ii......... and 2 1
1==1.... E NRx'" (i.e., the stack of data of tasks from to t on the view k). The closed-fore forem solution of the OP
(4 is given by:
-1 W (2*) diag **) + AI)
Z diag I)
where Zt and Y are respectively features vectors on hth view and labels of data far. This solution can be updated incrementally across tasks using the Gram matrix G
After finishing all T tasks, we obtain K sub-classifiers (see Algorithm Appendix D for the overview) and use them in our proposed voting strategy to make predictions.
The proposed voting strategy. Given a testing example æ, we first allow the linear classifiers in the atomic views to make their inferences to obtain the prediction probabilities p'(x), ..... ...,p" ( (æ). The voting strategy conducted according to the following steps:
o Given threshold 7 E [0, 1J, we select linear classifiers on the atomic views whose confidence level greater than y (i.e., max,, pz 2 7). Denote this set by A (I,...,,
@ We then create 21-A1 - huge views from A (where |A] is the cardinality of A). For each huge view (i) ...,, rb) C A, we compute its prediction probability vector: py(xr) =kEV a x (x))
where a² = + -i=at af-
We carry out majority voting on 2| responses of vectors pv(x) to obtain the ultimate label.
In this way. in addition to K weak learners, we have at most 2 K -1 stronger ones. Then, the probability of obtaining the correct answer will be higher through majority voting.
Discussion: It is worth noting that directly applying AdaBoost to CL is challenging because this algorithm requires the data and sample weights of all tasks, which are not always available in CL. Fortunately, thanks to RanPAC's ability to accumulate knowledge from all tasks, we developed an adaptive version of AdaBoost. However, our method is not a simple combination of AdaBoost and RanPAC. In particular, besides adjusting the calculation of the auxiliary weight a and normalizing sample weights within the same atomic view k to form A*, a crucial operation is to produce 2|-11 - huge views and exploit voting to obtain the final result. This is because the standard version of AdaBoost relies on large number of weak learners (usually hundreds or thousands or more), which is impractical for working in high-dimensional space according to the design RanPAC.
4.3 TASK-ADAPTIVE BACKBONE AND SELF-IMPROVEMENT PROCESS
This section presents how we improve the backbone of the CP-based method by integrating task- specific prompts and our proposed self-improvement process to select these components properly to achieve higher accuracy.
Prompt-based task adaptation for CP-backbone. Since RanPAC only adapts the pre-trained backbone with data from the first task, later task class representations might not be well-separated and could overlap with those of the first task. Therefore, we aim to learn task-adapted representations before applying RP, ensuring distinct separation between different classes and clustering within the same class (see Figure?? 3 Appendix PoppediixG.4ffommm for illustration). In this work, exploit the prompt-base mosept strategy from HiDE (Wang et al. (203) for training. Specifically, each task is devoted to a prompt P, and then trains the current task using Cross-Entropy (CE) loss and contrastive-based regularization (Reg) loss as follows:
min CE (P.. + &Rg (P:), O.P.
(5)
where 3 > 0 is a orf parameter and 0 is the weights of temporary classification head of prompted
model fa.p. Please refer to Appendix E for more details.
Prompt selection process. To predict prompts when inference, we learn an auxiliary classification head atop the pre-trained backbone fo. Speciffically, given sample æ., random matrix U
7

---

Published as a conference paper at ICLR 2025
to project (x) onto a high dimensional space (e.g., d' = 10, 000) and train linear classifier W incrementally like in RanPAC to predict the class labels. Eventually, based on the predicted class label, we can infer the task id t' and the selected prompt P. for (See Figure 4 Appendix G.4
Self-improvement process- a two-step prompt selection strategy. Ideally, the selected prompt P.. for æ should coincide with the prompt P. corresponding to the ground-truth task IDD of æ. However, we observe that W can sometimes give wrong responses so that P. (x) can shift much from P. (a:). To further relieve the shift in selecting wrong prompt, we propose two-step prompt prediction as follows:
First, we use the prompt ID prediction branch based on fa (az) to predict the prompt Pe. Second, we input fa. (x) into the main class prediction branch to predict the class label and then map it to a task id t" corresponding to the prompt P....
Finally, we use the prompt P.", to make the final class prediction using fe. (a)).
The intuitions behind this self infrooiproveenntt process include (i) the main class prediction branch predicts the class labels and hence the task identity better than the other and (ii) if the main class prediction branch predicts incorrectly the label v" its task t" instead of the ground-truth label and task id / of æ (i.e., v² A L" & L ), the representations of class /askk L" and y/task / tend to stay close. Hence there is a less shift between fo.P...a and P, leading to less error using the linear classifier W to predict on fo, (w)) rather than (w) (see Figure 4B in Appendix G.4
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Benchmarks. We examine widely used CIL benchmarks, including Split CIFAR-100, Split ImageNet- R, 5-Datasets, and Split CUB-200 (Please refer Appendix F.I for more details).
Baselines and Metrics. We compare our method with notable methods exploiting pre-trained
models for CL scenario, which belong to CP-based approach: RanPAC (McDonnell et al.) 2023). ADaM (Zhou et al. 2023), SLCA (Zhang et al.) 2023) and prompt-basee approach: L2P Wang et al. (2022d). DualPrompt Wang et al. (2022c). S-Prompt++ Wang al. (2022b). CODA-Prompi (mith ct al. (2023), HiDe-Prompt Wang et al. (2023), CPP et al. (2024). We present two key metries: the Final Average Accuracy (FFAA,, denoting the average accuracy after the last task, and the Final Forgetting Measure (FFM) for all tasks (Appendices) & p. The implementation is described in detail in Appendix F.4
For convenience, we denote two variations of our method without the self- self-improvement process: (i) BoostCL-s is the version that uses the task-adaptive backbone with single classifier for the Ca-ba backbone, while (ii) BoostCL-m((d (Ld KK) is the version that utilizes our proposed ensemble classifier with the boosting strategy for K multiple views each of which has the dimension of d'.
5.2 EXPERIMENTAL RESULTS
Our approach achieves superior results compared to baselines. Table I presents the main results of all the methods. The key observation is that the gap between our method (BoostCL) and the runner-up method is around 2% to 4% in terms of FAA on all considered datasets. Notably, with nearly the same number of additional projection parameters, hence the same high-dimensional space, BoostCL-m (d' = 768; K 13) still outperforms RanPAC, showing our boosting and voting strategies are beneficial. Furthermore, the absence of Random Projection, BoostCL-m (w/o RP: K 15), requiring no additional projection parameters, also surpasses RanPAC about 1% on Split CIFAR- 100 and Split ImageNet-R.
Additionally, the results show that BoostCL-s outperform the basclines, and BoostCL-m consistently improves upon this single-view version, indicating the effectiveness of the proposed task-adaptive backbone and multi-view strategies, respectively. Finally, the self-improvement process enhances BoostCL-m's 's performance by around 1%.
Time/space complexity of our ensemble classifier
Table 2 compares our ensemble classifier and RanPAC's classifier on Split CIFAR-100. Firstly, although the classifier of BoostCL--m (d' 10, 000; K 15) requires memory 15 times larger than
8

---

Published as a conference paper at ICLR 2025
Table 1: Overall performance comparison. We provide FAA and FFM of all methods, with standard deviation taken over at least 3 runs of different random seeds. The results corresponding to the best FAA among baselines are underlined.
Split CIFAR-100
Split ImageNet-R
S-Dulasets
Split CUB-200
Mothod
FAA (t)
(P
FAX (8)
FFM
BAA
PFM (4)
FAA (1)
(F)
1.2
83.06 +0.17
6.58 =0.40
63.65 +0.12
7.51 +0.17
81.84 +0.96
4.58 +0.53
74,52 +0.92
11.25 +0.23
DualPrompt
86.60 10.19
4.45 10.16
68.79 1:031
4,49 10.14
77.9110.4
13.17 =0.71
82.0510.955
3.56 1.0.5
S-Prompt-+
88.81 10.18
3:87 #0.05
69.68
1:.12
3.29 20005
86 86.19:50665
4.67 #0.72
83,12 1:.51
2.72 20.64
CODA.P
86.94 +0.63
=0.18
70:03 +0.47
+0.22
+0.53
17.22
74.34 +0008
12,05 +0.41
HiDe-Prompt
92.61 +00.88
3.16-0.100
75.06 +0.12
2.17 +0.19
93.92 +0.33
0.31 +0.12
86.62 +0.35
1,98 +0
CPP
01.12:0.112
=0.18
74.88 +0.07
+0.03
92.92 +0.17
=0007
82.35 +0020
+0.32
ADaM
87.60
72:30
74.15
87.10
91.50
77.00
RaaPAC
92,20
77.90
82,85
90.30
BousCCI-s K=1)
94.03 ::0.33
1.83 =0.12
78.52 10.12
3.12 20.15
94.08 +:025
0.25 10.15
90.92 4:0.32
2.21 =:011
BoosCCI-m (d' (d'7768 K-13)
04.56++0:23
1.75 =0.12
78.65 +0.12
3.12 +0.15
94 94.66++0.41
0,23=0.28
91,25+0052
2.12+00119
BoostCL-m (wle RP; K=13)
94.65.10:25
78.82
94.9610.14
0.25
91.60:10.48
2.1240021
BoostCL-m (d' 110k; K-15)
95.45 #0.25
1.72 =0.15
79.62 #0.10
3.03 10.15
95.62 #:0.30
0:22 #0.13
92.05 +0.31
2.08 #20.12
BoosICL
96.55 10.32
12.12
80.42
10.15
1.012
96.73
0.30
1..03
0.33
10.15
RanPAC (d' = 10, 000), this expense can be worth compensating because our BoostCL-m achieves
higher FAA by margin of more than 3%. Additionally, considering BoostCL (d' = 768; K 13). which requires less memory than RanPAC, we can see that with the help of our ensemble strategy. it can still improve the FAA compared to RanPAC and even BoostCL's single view (d' = 10, 000). We also conduct experiments on the model using RanPAC's backbone adapting with our ensemble classifier (RanPAC++). The results once again confirm the effectiveness of our ensemble classifier. Furthermore, in the case where RanPAC uses d' 10, 000 x 15 = 150, 000 (RanPAC*), which is the same as the total dimensions in BoostCL, the cost to store the Gram matrix (with size d' x d') or its inversion is too high and impractical. Meanwhile, BoostCL works well with this setting, as it only needs to deal with submatrices.
Regrading forward time, comparing with the original RanPAC (d' = 10, 000), forwarding the classifier of BoostCL (d' = 10, 000; K = 15) requires more time. This is obvious because we an ensemble strategy for prediction instead of using only one classification head like RanPAC. This cost is offset by the performance improvement we achieve. In addition, when discussing the potential of using projection matrices on high-dimennional space, we claimed in our paper that our process helps alleviate the "substantial computational cost when computing the inverse of d' x d' Gram matrix for solving the linear classifier" which is done in RanPAC. In this table, we compare our BoostCL with RanPAC* (d' 10, 000 15) and mark that this version of RanPAC infeasible to train because the cost when computing the inversion of a Gram matrix of size d'(== 10, 000 x 15) is Odd") for some constant e E [2.3. 3.0), according to Geéron (017)) Therefore, BoostCL. has solved this computational crisis, saving the time cost for a factor of O(15)) Training time for our methods and baselines is reported in Appendix G.3
Here we note that the times reported in Table 2 are the forward time of the classification head. The training times of our approach and baselines are reported in Appendix G.3 We also provide additional experimental results to scrutinize other aspects of our multiple- view strategy, including the effects of the total number of atomic views and the dimension of each corresponding projected space, the role of voting strategy, the applicability of our ensemble classifier (Appendix G.T) and the time/storage complexity (Appendix G.3).
Effect of task-adaptive backbone
The results in Table with the outperformance of BoostCL-s over CP-based baselines, indicate that employing task-specific prompts adapt to CL tasks is beneficial, confirming our argument.

---

Published as a conference paper at ICLR 2025
Table 2: Computation and storage cost of the classification head of methods, on Split CIFAR-100 100.
Metric
Method
Forward (s)
train?
RanPAC (4' 10, 000)
10*
10 4 10
BoosCCL K
10*
10 2.58
10
94.03
BoosCCI. (d' 768; 13)
less than 100 10*
8:31 10 +
10
94.56
Yes
RanPAC* (d'
100 15 10*
2.07 2.05 10
No
BoosCLL (d' 000); K 15)
100 15 10
3.31 10 + 4.26 10
95.45
Yes
BoostCL. (get outpurs)
2.68 10 + 3.65 x 10
BeosCCL (voting)
6.25 x 10 1 6.31 x 10
depicting the latent space of RanPAC's and our model on Split CIFAR-100, supports these results by showcasing better clustering of classes with more adapted tasks.
Additionally, Figure
Effect of self-improvement process
Employing multiple steps for prompt prediction enhances model accuracy in general, as evident in Table [ where steps improve FAA by around I % across all datasets. In addition, Ta- providess detailed perspective, showing that the number of steps increases, model performance further improves. The most significant im- provement is observed when transi- tioning from 1 step creases of 1.1% on Split CIFAR-100 and 0.8% on Split IeageNet-R. This improvement tends to gradually converge in subsequent steps, with growths of less than 0.2% from step 4 to step 5 in both datasets. This limitation may stem from the simplicity of the method itself; however, it could serve as an interesting suggestion for future research.
RamPAC
Our
steps, with in-
Figure 2:
t-SNE visualization for Split CIFAR- 10FAR-100- on data of Task 1 (class O - 9) and Task 2 (class 10 + 19).
Table 3: Performance when increasing the number of prompt prediction steps
Number of steps
Benchmark
3
Split CIFAR-100
95.45
96.55
96.85
97.03
97.05
Split ImageNet-R
79.62
80.42
80.62
80.65
80.66
CONCLUSION
In this work, we first consider the role and implications of a technique that is experimentally effective in previous work, using random projection for pre-trained models in CL. Then, the benefits of RP in high-dimensional space motivated us embark a journey of theoretical thinking and propose a multi-view strategy for an efficient classifier, in which the principle of Adaboost is adapted to overcome inherent obstacles and applied for the first time in CL. In addition, our self-improvement process technique, although simple, also shows significant effectiveness in selecting proper task- specific prompts. The experimental results demonstrate a positive impact of our proposed method in improving model quality while only applying to linear classifiers.
10

---

Published as a conference paper at ICLR 2025
Limitations: A primary limitation lies in the computational and storage costs for ensemble classifier. Like other ensemble methods, we need to train and store many weak learners than usual (see Appendix G.3) However, thanks to the voting strategy, we do not need to train many weak learners to achieve significant performance as the original methods (Adaboost and SAMME often need hundreds of weak learners to show improvement). Another potential limitation can be the computational overhead of the self-improvement process. As we discussed computational complexity in Section G.3.3 two rounds of this process can already bring improvement, and its time complexity is not too significant, so it could be used with any prompt-based based methods. Nevertheless, cases where downstream data is too different from that of the pre-trained- ViT, the simple design of the prompt predictor may hinder its performance. Consequently, adding more self-improvement steps may not increase accuracy.
ACKNOWLEDGMENT
Trung Le and Dinh Phung were supported by ARC DP23 grant DP230101176 and by the Air Force Office of Scientific Research under award number FA2386-23-1-4044,
REFERENCES
Peter L Bartlett, Dylan Foster, and Matus J Telgarsky, Spectrally- normallizd margin bounds for neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017.
Yaroslav Bulatov. Notmnist dataset. 2011.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16678 2022.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning. In European Conference Computer Vision, pp. 386 402. Springer, 2020.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-135, 1999.
Yoav Freund and Robert E Schapire. A decision-theorett generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-139) 1997.
Aureélien Geéron. Hands-on machine learning with Scikit-Learn and TensorFlow concepts, tools, and techniques to build intelligent systems. O' O'Reilly Media, Sebastopol, CA, 2017. ISBN 978-1491962299.
Dipam Goswami, Yuyang Liu, Bartlomiej Twardowski, and Joost van de Weijer. Fecam: Exploit- ing the heterogeneity of class distributions in exemplar- free continual learning. arXiv preprint arXiv:2309. 14062, 2023.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, et al. Pre-train trained models: Past, present and future. Al Open, 2:225-255 2021.
Trevor Hastie, Saharon Rosset, Ji Zhu, and Hui Zou. Multi-class adaboost. Statistics and its Interface, 2(3):349-366 360, 2009.
Paul Janson, Wenxuan Zhang, Rahaf Aljundi, and Mohamed Elhoseiny. A simple baseline that questions the use of pretrained-mote in continual learning. arXiv preprint arXiv:2210.044// 2022.
Menglin Jia. Luming Tang, Bor-Chun Chen, Claire Cardie. Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
11

---

Published as a conference paper at ICLR 2025
Kenji Kawaguchi, Zhun Deng, Kyle Luh, and Jiaoyang Huang. Robustness implies generalization via data-dependent generalization bounds. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 10866-10894, PMLR, 2022.
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, and Bing Liu. A theoret- ical study on solving continual learning. In Sanmi Koyejo, S. Mohamed, A. Agar- wal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Informa- tion Processing Systems 35: Annual Conference on Neural Information Processing Sys- tems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 2022, 2022. URL http:: http://ppprr...t....ccopppeeeeee 1BF44068000000666ccca00002222220 : CostrrcttCooferennce html
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-25) 23244, 1998.
Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N Metaxas. Steering prototype with prompt-tuning for rehearsal-free continual learning, arXiv preprint arXiv:2303.09447, 2023.
Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N Metaxas. Steering prototypes with prompt-tuning for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 2523-2533, 2024.
Mark McDonnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, and Anton van den Hengel.
Ranpac: Random projections and pre-trained models for continual learning. arXiv preprint arXiv:2307.02251, 2023.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Julien Nicolas, Florent Chiaroni, Imtiaz Masud Ziko, Ola Ahmad, Christian Desrosiers, and Jose Dolz. Mop-clip: mixture of prompt-tuned CLIP models for domain incremental learning. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2024, Waikoloa, HI, USA, January 3-8, 2024, pp. 1751-1761, IEEE, 2024. doi: 10. N001109/ACV5770..200...00
Utthttts:: /ttpss://oof /oo::///oiiioo org 110/100.11111011 1wt.1111/9/////11 701 2024 00178
Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: A strong replay-free baseline for iuss-incremental learning, arXiv preprint arXiv:2303. 13199, 2023.
Mark Rudelson and Roman Vershynin. Smallest singular value of a random rectangular matrix. Communications on Pure and Applied Mathematicc: A Journal Issued by the Courant Institute of Mathematical Sciences, 62(1)):1777-1739 2009.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda- Continual decomposed attention-based prompting for rehearsal-free free continual learning. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp. 11999-1119,, 2023.
Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing, 65(16):4265-4280, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Liyuan Wang. Jingyi Xie. Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt- based continual learning: Rethinking obscured sub-optimality. arXiv preprint 200 0.74234, 2023.
12

---

Published as a conference paper at ICLR 2025
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained trans- formers: An occam's 8 razor for domain incremental learning. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 2022, 2022a. URL 25886d7a7cffee 35866774ccc4333444447220663333 Abstract Conference. html
miperrs..iip....mm mips.cc/pape..
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers:
An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022b.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022c.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy. and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022d.
Yihong Wu and Yingxiang Yang. Lecture 14: Packing, covering, and consequences on minimax risk, 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
Huan Xu and Shie Mannor. Robustness and generalization. Machine learning, 86(3):391-444 2012.
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. SLCA: slow learner with classifier alignment for continual learning on a pre-trained model. In IEEE/CVF International Conference Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 19091-19101. URL //ttt: http://o........1..111..
IEEE, 2023. doi: 10. 100/CCC5100000023.000
ICCV51070 2023 01754
Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learn- ing with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338, 2023.
HUIPING ZHUANG, Zhenyu Weng, Hongxin Wei, RENCHUNZI XIE, Kar-Ann Toh, and Zhiping Lin. ACIL: Analytic class-incremental learning with absolute memorization and privacy protection. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL http::ttt///ootm http://///ppeeeeee.eeeeeee Id=vc4QUfqr4do
Huiping Zhuang, Zhenyu Weng, Run He, Zhiping Lin, and Ziqian Zeng. Gkeal: Gaussian kernel embedded analytic learning for few-shot class incremental task. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7746-7755, June 2023.
13

---

Published as a conference paper at ICLR 2025
Appendices
A WHY DOES RANDOM PROJECTION ONTO HIGHER DIMENSION SPACE HELP INCREASE THE MARGIN?
We first consider the role of RP without activation function. Let H. be the set of all measurable functions that map from Hd to V. and Ht' be the set of all measurable functions that map from Id to V.. The following theorem shows the benefit of random projection onto high-ding -igh-dimensional space. Theorem A.1. Let U be a random matrix of size d' x d, with d' z d, whose elements are independent copies of a Gaussian random variable with unit variance and O mean. Consider = (z, v) and its projection s (zT U, U) For every O, with probability at least 1 (C.c) -d+11 e e-edd where C and e are some constants, we have
(S, ,?') 2 (i,((,)0)()) (VVa' - Vd
This theorem suggests that the margin in the new space after applying RP should increase fast as d' increases. The increase rate is O(vd'. This property of RP is intriguing and has never been known before. Note that such an increase may not appear in the cases of class overlapping due to (i, H) for some sample s. Therefore, RP will provide a crucial benefit in the cases of (nonlinear or linear) separability.
In practice, RP is often followed by fixed nonlinear operation. One main reason is that this nonlinearity can keep the crucial benefits of RP when used before, e.g., linear models or neural networks. Without nonlinearity, the effect of RP can be absorbed into trainable linear operation. Therefore, we need to incorporate nonlinearity into the analysis of RP to explain the overall benefits in practice.
In general, nonlinearity often poses challenges for analyzing RP. We restrict our analysis to nonlinear operations which are invertible and expansive F Note that such invertible and expansive nonlinearities are prevalent, such o(x) bxz in domain [1, oo) for any b > 1, oxx)
be in domain [0, oo), (x) = b (in( domain [-a, a} for some small a > 0, (ax) = & In(x) in domain (0, 1), (x)) bæ. ReU (2) max(0, z) min z),, etc.. This leads to Theorem 4.2
A.1 MARGIN AND GENERALIZATION ERROR
We next recall a well-known connection (Bartlett et al. 2017 Sokolić et al. 2017: Xu Mannor 2012) between the margin and generalization ability of classifier.
Consider a learning problem specified by a model (or hypothesis) class H. an instance set X. and loss function é: H. x X + Rzo. where each input instance E X has corresponding output y E y. However, without loss of generality, we omit the output for simplicity of presentation. Given a distribution P defined on X, the quality of model h E H is measured by its expected loss F(P, h) = Ex~ ( x) The empirical loss F(S,h) h) [ST Eaes ((h, x) is defined from a finite set S (m) (m1)...,,..) C X of size n.
Let [X =U X be partition of X into disjoint nonempty subsets. Denote S, SSX,
and 14 = |S|| as the number of samples falling into X, meaning that n j==1 ny. Denote
T (i E [K] S X, + 0).
The following result a simple consequence from algorithmic robustness (Kawaguchi al. 2022 Xu & Mannor 2012).
Theorem A.2. Consider a model and a dataset S which consists of n i.i.d. samples from distribution P. Let E(h, x) be the loss of h at instance x, and C₁. = sup=ex E(h, x), e(S) = su] sUP=Ex; 16(h, x) f(h, s)]. For any > 0, the following holds with probability at least 8:
T In(2K/0)
2C T (2K2/
F(P,h) S F(S, h) +c(S)++3 30
+
(6)
n
n
A function is expansive if it satisfies |a((x) - avv)l 2 |ar for all x. y in their domain.
14

---

Published as a conference paper at ICLR 2025
This theorem implies that the expected loss of model can be bounded by using robustness level c(S) of the model around the training samples. A more robust model can suggest better generalization on unseen data.
Next we connect the margin and generalization ability of a model. Let (V || A) be the A- covering number of X' according to metric || and ((x) be the corresponding covering of X, where is a positive constant. This means = NN' A). When the input space X' S Rd
has diameter at most R, classical fact (Wu Yang 2016) says that W(x, || 1I. - 2RV/
Furthermore, Example 1 in (Xu & Mannor 2012) shows that c(S) for any À S 0.5 and 0-1 loss E, where "6 = max(u x 8|| I t = h(x) = hss) for æ X, E S} denotes the
margin of h for the training set. Combining those observations with Theorem A.2 will lead the following.
Corollary A.3. Given the notations in Theorem A.2
with 0-1 loss l and X C R with diameter at
most R. Denote "hh = max || - 81 S e = a = h(8) for all x: E X, E S) as the margin
of h, and A = In(2/8) + d (In44Rdd/m)) mh > 0, then the following holds with probability at least 1 8:
TA n
A
F(P,h) S P(S +3
n
This result suggests that model with larger margin can lead to smaller bound for the expected loss, suggesting better generalization ability. Combining this with the RP's ability to increase margin, we can conclude that RP can facilitate better generalization ability of a model in high dimensional spaces.
A.1.1 PROOF OF THEOREM A.2
Proof. Firstly, make the following decomposition: F(P,h)
K
K
=(P,h)
Ex (f, X3]
Ex [h, E X. - F(S, F(S, (8)
1=1
Secondly, consider A = =(PPh) - Ei=l
-EK
EE x) E X.).
Observe that:
n; Ex (f, x)) E X4]
A1 F(P,h)
(9)
n
K
0 Eg (f, x) x E X,]
Eg [h, x]]
(10)
n
Note that (Ri,.., ng) is an iid multinomial random variable with parameters n and (P(X),,.. .... ()xx). Therefore
4
(( E X]]
E [f n Σ
E X3)
(11)
n
K
=
En (f,, x:) X:)
74 P(22)
(12)
n
T (2K(2K/8) + a. 2 log(K//5
<
QV
(13)
n
We have the last inequality with probability at least - 5, according to Theorem in (Kawauuch) et al. 2022). where Q
av2 + ag a = maxer Eg (h, x)|2 E X,,, and d₂ 5 E (= ar) le E Xz}- Note that Ex [s(, at) E X.] Cn. for any index i. It suggests that Q < 3C2 and 0r S Cn. As a result,
T log(2K/0)
20, IT log (2k(2))))
A S 3C
(14)
n
n
15

---

Published as a conference paper at ICLR 2025
Third, consider A2
Ei
E. [e x)) E X
F(),h). Observe that
K
A2
M Ex ( h, x)) E Exi]
Σ
E(h, 8)
#ES
ni Ex ( h x E
e(h, s)
ET
ET aES;
ni:lia [e x) 2:
Σ
#ES.
Eng (h) X;] f(h, 8))
n
ET acS,
En [6(h, x) -(h, s)} X:}
n
ET aES;
Σ
sup [((h, x) E(h,s)) s)
11 iET aES,
zERi
1
Σ ((S)
iET S
cSS)
(15)
Finally, decomposition complete the proof.
g shows that F(P,h) = A1 + Ag. Combining this with 14 and (15
will
A.2 PROOF OF THEOREM
Let $ E S be an instance and On E H be the classifier such that (ss, 9h, Rd) = Vin H)). We first consider the ball that reflects the region around instance (z, v), for which the classifier 9r. has no miselassification. Denoting v 7in (s, H), the ball is:
B B(s (2' Rd || 5. = y)
(16)
The diameter of this ball represents the margin of the classifier at sample s. Every point in this ball will be assigned the same label with z. When projecting this ball onto higher-dimensional space by transform matrix U of size d x d', we obtain an ellipse as
B Vs, = U z' B((,D))
(17)
Denote g'(v)= ((U ((U -UV) for any E R., Note that g' is classifier, defined in the new space, and hence q' H'. This function almost surely exists since UUT is invertible, almost surely Rudelson & Vershynin (2009). Observe the margin of " = (z"U y) w.r.t':
7s8 (("",',RR") = suptu || ell S g(() = e
e R²
(18)
= sup | u|l S H (u(((("))~
Vo R
(19)
= sup(#: ! - xT و 5 g2 = = V= e Rd
(20)
= sup(a: ||(z - = S μ gx(2') =y, E Rd
(21)
This suggests that (s"', g', RRE' represents the smallest diameter of B² v). Furthermore, the smallest diameter of this ellipse Bd is, in fact, the smallest singular value of matrix vU. It means
rss", "'' Rdd) amin (vU) won (U)
(22)
where 7min (U) denotes the smallest singular value of matrix U.
16

---

Published as a conference paper at ICLR 2025
Remember that U is random matrix whose elements are independent copies of Gaussian random variable with unit variance and zero mean. According to Theorem 1.1 in (Rudelson Vershynin 2009), for every > O, we have
Pr(amin (U) <<lVd' - vd- <v(-d--vd-I) S (C.e))--4+ d+1 +e-cd
for some constants C and c. This implies that
Pr(amin (U) 2 elvd' vd- > (C.e) e
As a consequence Pr(7(s"; gf 2 velvd' vd-1)) > (C.e)d'-at e-dd
Note that, by definition, the Bayes margin (s, H') cannot be less than 7(8", RR, due to g' E H.. Therefore
Pr("in(" H²)) 2 velva vd- > 1 - (C.c) d+1 - -6 e.dd
The margin of each instance enlarged by a factor of e e(vdd V 1I) Therefore 7(S, yh, Rd) =
min (8, Uhe IRd) is also enlarged by the same factor, completing the proof.
A.3 PROOF OF THBOREM
Let $ E S be an instance and gh E H be the classifier such that (ss, 9h. R*) = (s, Ht). We first consider the ball that reflects the region around instance & = (z, y), for which the classifier ya does not have any misclassification. Denoting v = =sins, H), the ball is:
B(s v)
{2' E IRd || z/11 S v. gn (2 = y)
(23)
{2' E ERd ||= - 1-11 S v. g, = gh (2))
(24)
The diameter of this ball represents the margin of the classifier at sample Every point in this ball will be assigned the same label with z. When projecting this ball onto higher-dimeesiooall space by transform matrix U of size d x d', we obtain the following ellipse:
BB (s, = (zT U E B(s
(25)
D (UU")))--O))) for any v E Red, where a denotes the inverse function of a. Note that g' is a classifier, defined in the new space, and hence qf E H'. This function almost
Denote ('v))= (p0)))((((U))
surely exists since UUT is invertible, almost surely Rudelson & Vershynin 2009). Observe the margin of s" = (o(="U) y) w.r.gg.
suplu: : [|o(= - w/l S p n g'vv)= = Vo E R
(26)
suplu: : [|a(=TU) - - S n - (UU7)
= gz Vo())
suplu:
- - م (z') = gn Vz E [²} (28)
The last equality comes from the fact that a is an one-to-one mapping which maps each E Rd" to where z' = -UUT)-UU
Since is an expansive function, we have [|a(=TU) a(2TUU)] 2 -LTU~-TUII I( U|. Combining this fact with (28) implies that 7(s". g', Rd is not less than the smallest diameter of BB (s, v)). Furthermore, the smallest diameter of 1 is, in fact, the smallest singular value of matrix vU. It means
7(s", "'"."'RR""" Omin (vU) min (U)
(29)
where mii (U) denotes the smallest singular value of matrix U.
Remember that U is random matrix whose elements independent copies of Gaussian random variable with unit variance and zero mean. According to Theorem 1.1 in (Rudelson & Vershynin 2009), for every O, we have
Pr(amin (U) <<lVd' Vd < (C.c)d'-d4e ddd1 + -edd
17

---

Published as a conference paper at ICLR 2025
for constants C and This implies that
Pr(Amin (U) 2 clvd' vd- > 1 - (C.r) eedd
As consequence
Pr(7(6", "'.",, velvd' vd - 1]) > (C.e) cadd
Note that, by definition, the Bayes margin (s H²) cannot be less than 7(8", ".) due to E H". Therefore
(s(" H²) velVd' Vd 1) > 1 - (C.c) -dt d+1 e.d
The margin of each instance enlarged by a factor of clvd' - Therefore (SS 9h, Rd) = minses (8, Dh. IR)) is also enlarged by the same factor, completing the proof.
Note: Based on the proof, we can see that Theorem 4.2 still holds for the case of any data instance.
B ADABOOST AND SAMME ALGORITHMS
Algorithm AdaBoost / SAMME
1: Input: Dataset D { 1 < < N} size N, number of classes C' (3/: (1,2...., C); 4A S šis N), number of weak learners M
2: Initialize sample weight A = [A, M
3: for m = to do
4: Train classifier ((m) with D and A
5: Compute error rate:
(rp(m) Ei A, (r.)
6: Compute the auxiliary weight (m))
AdaBoost:
( 1 - (ry(m) err(m)
a log
SAMME:
- crp(m)
(m) = log
+ logg( - 1)
crr(m)
7: Update sample weight
A₂ = Ar (xpfa((m
11tm) 1. SisN
8: Re-normalize A:
A
A
v1 VISiSN
EN A
9: end for
10: Output:
M (m). 11)m)
v(4) = arg max
m=
In the realm of classification problems, boosting stands as a formidable strategy. AdaBoost Freund Schapire (1997) and SAMME Hastie et al. (2009), which addresses Adaboost's limitations in handling cllti-class situations, ensemble learning techniques that combine multiple weak classifiers [g ( to create a strong classifier G(.) by sequential training on weighted training data {xx,, H including classes, focusing more on misclassified examples in each iteration & to improve the model performance iteratively as follows:
For & = L, weak learner g is trained on with the sample weight vector A = [A, I
After training, 9h will be evaluated the error rate (ry(k) ΣNV A 1 1g((="))'">> and
18

---

Published as a conference paper at ICLR 2025
then the auxiliary weight (k) log( the sample weight of D for the learning in the next iteration: A, = A, exp(a!m)
( ())
+ log(C - 1), which is used update
1 (m))) V1 S N. Following this, the weight of wrongly predicted samples will be increased.
For k the weak learner ge will be trained D with updated A from the previous iteration. In which, the sample with higher corresponding weight will be focused more during the training process. Therefore, the later weak learner will focus more the mistake of the previous one and will learn to complement that previous classifier. The next operations are similar to the case when 1.
Finally, the prediction of G for test sample x is represented by: y(x) = arg max, Σx= (m)) (m)=
Detailed pseudocode of both algorithms provided in Algorithm 1
Note that applying Adaboost to CL is inherently challenging:
Adaboost's ' algorithm requires the whole training set, which is not always available in CL when only accessing the current task's data during training.
Regarding experimental results, Adaboost often gives superior results with a large enough number of weak learners, usually hundreds thousands. If there too few learners, Adaboost will not show a clear advantage (see the results Table
Fortunately, these challenges have been addressed for the first time in our work, as discussed in section 4.2.2
C PROVING THE THEORY OF OUR MULTI-VIEW RANDOM PROJECTION SCHEME
C. PROOF OF THEOREM
If we approximate G by G, the optimal solution is W (G + XI)
Y We further have
(ZI)TZ)
+
AI
0
0
AI
0
(Z²)
2
+
AI
0
0
0
0
0
(2
7
XI
G
AI
0
0
0
0
G
+
XI
0
0
0
0
0
0
+
AI
(G'
+ XI)
0
0
(G+ -
0
(G2
+ XI
0
0
0
0
0
0
0
(G
+ XI)
Besides, 2- - Therefore, we reach
W
W2
W
W
19

---

Published as a conference paper at ICLR 2025
C.2 PROOF OF COROLLARY 4.4
We derive as follows
W
(x.1) = = W 21²
22
2
T
W
W
8i (x, y)
C.3 THE TIGHTNESS OF THE INVERSION APPROXIMATION
We prove that (G + AI) approximates well (G + AI) when À becomes large enough.
Firstly, we show that both G and G are positive semi-definite matrices. For any arbitrary vector E kdd
Gv Gu v² 7 2r = =(20)
2 0
Ž Z
0
0
0
2 T
D Gv 8
0
2
0
0
0
Z
Z
0
0
0
Z
Z
0
0
0
0
0
0
0
T
0
0
0
0
0
0
Zk T
0
0
0
0
Z
Z
0
x
Ž1
0
0 Dlot0"
2
0
22
0 0
u+
0
0
+ T
0
0
z v
Z
= =( 1 0
0
v
([
0
v) +
+(1 0
2
0
([
0
2
v)
+
+ +... (( 0
1
7
([
0
z
{{ [v)
0
Hence, each eigenvalue of or G is non-negative real number.
Besides, also notice that both G AI and G + AI are positive semi-definie which leads their eigenvalues are also their singular values.
Consider matrix A, denote:
20

---

Published as a conference paper at ICLR 2025
rank(A): rank of A
Smin (A): ) the smallest eigenvalue of A
Mmin (A): the smallest singular value of A
E(A): Set of all eigenvalues of A
It is well-known that E(A + AI) = (5 + X E E(A)}.
From the above we have
min (G XI)
Emin (G XI)
Ein GG) +
X
X
And similarly Omin (G A.
Secondly
|I(G (G+ + lle
$
$ (G | + A1)- + (rank((G |I(G + AI) -112
S VKd' (II(G + A1) 112 + |I(G + 112)
=VKd
Tmin (G+AD
min G AI
Assume that À > reach
+ (G++A < 2vKd
X
completing the proof.
D OUR ALGORITHM FOR DIVERSIFYING MULTIPLE VIEWS WITH BOOSTING PRINCIPLE
This section provides concise pseudo-code for our proposed technique, which is presented in Section This step is performed after the backbone is frozen. In general, need to pre-determine the number of atomic views K, and eventually, we obtain prediction matrices W corresponding to these K views. In particular, for each task t, first initialize the sample weight, vector whose elements are all 1, corresponding to n, data samples. Then, we will learn K atomic views in turn; for each view, calculate the prediction matrix based 4 and then update the sample weight vector to learn the next view.
4.2.2
Algorithm 2 Multiple-views learning for adapting T tasks sequentially
1: Input: Data {D;- = (X,, Y:) Number of atomic views: K
2: for task = 1 to T do
3: Initialize observation weights for data samples of Dr: A [A 1]
4: for view
K do
5: Calculate W of 4th view incrementally using (4 with weights A
6: Update the weights A according
7: end for
8: end for
9: Output: Prediction matrices W
21

---

Published as a conference paper at ICLR 2025
Looking closer at the algorithm, since the prediction matrices caleulated based accumulating knowledge from data of tasks so far, thus we used A = -AA = 1 for initialization (Line 3), instead of A = [Ag, ni
1 1 as in the original Algorithm 1 Besides, to create the balance when
learning common classifiers for the sequence of tasks, we also perform a normalization step for this vector before forming A for 1tth view.
E PROMPT TUNNING FOR
ADAPTATION
Figure below illustrates more clearly the motivation of method in utilizing task-wise prompt- based task adaptive backbone.
b) RanPAC: Pretrained model PETTL (first task adaptation) Random projection
d) Pretrained model Task-speelific prompt-tuning Multiview ramdom projection
View
R 10230
Projection
Projection 1/
Projection
Ux
Task
Class 2
Clase d
Class
Chass
Task
Class 9
Pretrained model
c) Pretrained model Task-specific prompt-tuning
PETL (first task adaptation)
Figure 3: (a)&(b) Visualization potential overlapping later tasks of RanPAC in original latent space and high dimensional space due to the lack of adaptation to these tasks. (c) With task-wise prompting, the original features of the classes tend be better distinguished. (d) Multi-view random projection encourages clearer separations in different views.
Inspired by HiDE, we maintain a set of base prompts Q 1:r up to task and compute the prompt Pt for task as P. (1 - Q) + +>= Q;, where Ç is parameter within the range of [0, 1]. During learning task t, the base prompts Q Q1:--1 fixed. After that, we insert P. into the pre-trained backbone and train the current task using Cross-Entropy loss and contrastive-basse regularization loss follows:
-t
min CE (Pr,0) + +BReg (P.) e.P.
(30)
where 3 > 0 is a trade-off parameter and 9 the weights of temporary classification head of prompted model Sa.p.
In particular, to learn representation within the current task, we use Cross Entropy loss:
CE(P,,0) 1 CE(ho(fo.
(31)
n4
where he is the auxiliary classification head employ to learn to representation 2, fa. P. (æ).
In addition, to enforce the separation between features of new task and previous ones, at the end of each previous task, summarize and store the representations of class & in old task to mean
22

---

Published as a conference paper at ICLR 2025
(A)
P
(B)
P
P
A
t
W
W
W
W
fz
fs
fe
Pa
x
2
I
Figure 4: Prompt selection process (A) and Self-improvement prompt selection strategy (B)
vector my by simply averaging those class representations and the following contrastive loss:
D
1 The
Reg (i))
{(z)),
(32)
exp [2) m))
where {(2))
and is set to 0.8.
Σ Dii exp (2 mz//
- exp
Besides, Figure illustrates more clearly our prompt selection process and our proposed self- improvement strategy, which are described detail in Section 4.3
F EXPERIMENTAL SETTINGS
F.1 DATASETS
We follow the protocols in McDonnell et al. 2023) to construct the following common benchmarks:
Split CIFAR-100 Krizhevsky et al. 2009): This dataset comprises images from 100 classes, each of small scale. These classes are divided randomly into 10 separate incremental tasks, with each task featuring distinct set of classes.
Split ImageNet-R Krizhevsky al. 2009): This dataset comprises images from 200 classes, categorized as large-scale. It incl includes challenging examples from the original ImageNet dataset and newly gathered images representing diverse styles. These classes are also randomly divided into 10 distinct incremental tasks.
5-Datasets Ebrahimi et al. (2020): This composite dataset incorporates CIFAR-10
Krizhevsky al. (2009), MNIST LeCun al. (1998), Fashion-MNIST Xiao al. 2017] SVHN et al. (201T), and notMNIST Bulatoy (2011). Each of these datasets is treated as separate incremental task. This structure allows for the assessment of the effects of significant variations between tasks.
Split CUB-200 Wah et al.) (201T): This dataset consists of fine-grained images of 200 different bird species. Like the others, it is randomly segregated into 10 incremental tasks, each comprising unique class subset. Note that use the same number of training samples as in RanPAC (McDonnell et al. 2023), which is larger than that used in SLCA ZZhang et al. (023).
F.2 BASELINES
In the main paper, we use CL methods with pre-trained VIT as the backbone. We group them into CP-based and Prompt-based approaches.
23

---

Published as a conference paper at ICLR 2025
(I) RanPAC (McDonnell et al. 2023
CP-based method that adapts the first task using parameter- efficient transfer learning. then projects the features onto a high dimensional space, and finally incrementally learns linear classifier. Different from our method, RanPAC only performs adaptation at the first task and uses one projection matrix, resulting in potential overlapping of future tasks' features and infeasibility when working with extremely large dimensions.
(2) A (ZhuZ et al., 2023): another CP-based work that proposes a framework which adapts the pre-trained model to the first task using several adapting techniques such as prompt-tuning (Jia et al. 2022), adaptor (Chen et al. 2022) and batch-norm statistics adjustment, so that the representation could be more specific towards the downstream CL task. At prediction, ADaM only uses the nearest class mean classifier, which is shown in (Goswami et al. 2023) to be sub-optimal as it lacks second-order statistics such as the covariance matrix.
(3) SLCA: (Zhang et al. 2023): SLCA is novel method that proposes learning new tasks with a small learning rate for the feature extractor and slightly larger one for the classification head so that the model can reduce the potential representation shift. It additionally implements classifier alignment based on saved features to improve the classification head at inference further. On the one hand, in SLCA, using small learning rate will lead to less-shifted representation. However, this also makes the model become too stable to learn new tasks. This limits the performance of SLCA.
(4) L2P Wang al. (2022d): the first prompt-based work for CL that proposed using common prompt poll, in which the top k most suitable prompts are selected for each sample for training and testing. This might allow knowledge transfer between tasks but also carries the potential risk of catastrophic forgetting. In addition, different from us, L2P has not really considered training classifiers well setting constraints on features from old and new tasks during training, which likely leads to limitations in the model's 's predictability.
(5) DualPrompt Wang al. (2022c) is the prompt-based method that attempts overcome the disadvantages of L2P when attaching complementary prompts to the pre-trained backbone instead of only prompting input. Besides, DualP also integrates additional prompt sets for each task to exploit more task-specific "instruction" in addition to task invariant information from the common pool. However, like L2P, this method does not really focus on learning classification head efficiently. Moreover, choosing the wrong promptlD (for task-specific instruction) during testing could also reduce model performance.
(6) S- S-Prompt++ Wang et al. ) is originally proposed for Domain-incrementee learning. It trains a separate prompt and classifier head for each task, At evaluation, it infers domain id as the nearest centroid obtained by applying K-Means
the training data. To adapt S-Prompt to CIL, S-Prompt++ uses one common classifier head for all tasks. This method also has the same limitations as pointed out in DualP: the learning of classification head and the prediction of appropriate prompts at testing time.
(7) CODA-Promph Smith al. 2023): This prompt- based work uses task task-specific learnable prompts for each task, but similar to I.2P, CODA employs a pool of prompts and keys and then computes a weighted sum from these prompts to generate the real prompt. The weights are defined the cosine similarity between queries keys. At the end of the task sequence, to avoid task prediction, its weighted sum always considers all the prompts. CODA offers improvements over DualP and L2P in that the optimization of keys and prompts occurs simultancously, but it still has not solved the drawbacks we mentioned about DualP above.
(8) HiDe-Prompt Wang et al. (023)): a recent SOTA prompt-based method that decomposes learning CIL into modules: task inference, within-task predictor and a task- adaptivee predictor. The second module trains prompts for each task with a contrastive regularization that tries to push features of new tasks away from prototypes of old ones. To predict task identity, it trains classification head on top of the pre-trained VIT. TAP is similar to fine-tuning step that aims to alleviate classifier bias using the Gaussian distribution of all classes seen so far. Although HiDE solves one of the limitations of the above methods, in comparison with ours, this method still suffers from classifier forgetting, and its taskID predictor may be sub-optimal if the backbone is too different from downstream CL tasks, which we tackle by self-im infrrvemeett process.
(9) CPP Li et al. (024): This recent SOTA also uses a contrastive constraint to control features of all tasks so far during representation learning and achieves roughly equivalent performance to HiDE on
24

---

Published as a conference paper at ICLR 2025
the same settings. Nevertheless, this method still has the advantages that pointed out in HiDE, which we propose to address our work.
F.3 METRICS
In our study, we employed two key metrics: the Final Average Accuracy (FAA) and the Final Forgetting Measure (FFM), To define these, we first consider the accuracy on the i-th task after the model has been trained up to the t-th task, denoted as Ai, The average accuracy of all tasks observed up to the t-th task calculated as AA, report the Final Average Accuracy as FAA = AAr. Additionally, we calculate the Final Forgetting Measure, defined as FFM -- >i= (1.., (Ai, Ai,)). The FAA serves as the principal indicator for assessing the ultimate performance in continual learning models, while the FFM evaluates the extent of catastrophic forgetting experienced by the model.
Σ A.,. Upon the completion of all T tasks, we
F.4 IMPLEMENTATION DETAILS
In our method, we set the contrastive regularization weight / = 0.1, the parameter for prompt construction Ç = 0,1, and the confidence threshold for expert filtering is 0.5. The default values for the number of self-improvement steps is 2, the number of expert views is 15, and the dimension of each atomic view is 10,000.
Our implementation aligns with the methodologies employed in prior research Wang et al. 2022d Smith et al. (2023). Specifically, our framework incorporates former BB16) as the backbone architecture. For the optimization process, utilized the Adam optimizer, configured with typer-parameters 81 set to 0.9 and B3 set to 0.999. The training process conducted using batches of 128 samples, and fixed learning rate of 0.005 applied across all models except for CODA-Prompt. For CODA-Prompt, we employed a cosine decaying learning rate strategy, starting at 0.001. Additionally, grid search technique was implemented determine the most appropriate number of epochs for effective training. Regarding the pre-processing of input data, images were resized standard dimension of 224 224 pixels and normalized within range of [0, 1] to ensure consistency in input data format.
use of pre-trained Vision Trans-
In Table 1 of the main paper, the results of L2P, DualPrompt, S- S-Prompt++, CODA-Prompe and HiDe-Prompt on Split CIFAR- and Split ImageNet-R are taken from (Wang et al. 2023). Their results on the other two datasets are pro- duced from the official code provided by the authors. For CPP, the reported results are also reproduced from their official code. Besides, the results of ADaM and RanPAC on Split CIFAR- 100, Split ImangeNet-R, and CUB are also taken directly from RanPAC paper (McDonnell et al.) 2023), while the remaining is produced by their official code.
Spit: CIFA-1100
06.5
Splt Imagshet.R
96.0
45.5
94.0
M35
43.0
790
78.5
G ADDITIONAL EXPERIMENTS
78.0
715
G. EXPERIMENTS
ABOUT OUR ENSEMBLE CLASSIFIER
10
35
20
20C 2
Number iccpent Views
Dimendan of projected Scace
Effect of the number of atomic views and their dimension. Figure illustrates that over- all, model's FAA gradually improves when increasing the values of these two quantities. Notably, breakthrough occurs when increasing from 10 to views on Split CIFAR- 100 and from 15 to 20 views on Split ImageNet-R R. In addition, on both datasets, the most significant improvement observed when changing the dimension of the
Figure 5: Performance when the number of atomic views and the dimension of each projected space (of each atomic view) vary.
25

---

Published as a conference paper at ICLR 2025
projected space from 5,000 to 10,000, with an increase of more than 2%. Besides, can observe the convergences of FAA as the two quantities reach certain values. This might be due to the limitation of the linear classifiers.
The effectiveness of our voting strategy. Table 4 presents additional experimental results showcasing the advantages of our proposed voting strategy. Our method with voting strategy (B) outperforms the single-view version (D, or BoostCL-s), multi-view without boosting version (C, whose classifiers on different atomic views are trained independently), and AdaBoost (A) on two considered datasets. In particular, using AdaBoost (A) only brings marginal or even negligible improvement compared to using the single view model (D). This may be due to the limited number of views/learners (K = 15), which is insufficient for AdaBoost, typically requiring numerous iterations/weak learners to demonstrate its strength. Meanwhile, with the given number of atomic views, our voting method can generate at most 2 responses, leading to better efficiency. Furthermore, simply using multi-view without boosting will not be effective, as only yields performance equivalent to the single-view version. This is because the output of the respective views is almost the same in most cases.
Table 4: Comparison between our voting strategy and AdaBoost
Benchmark
Method
Split CIFAR-100
Split ImageNet-R
(A) AdaBoost (15 leaners) (Boosting w/o voting)
94.15
78.55
(B) Multi-view (K 15) (Boosting & voting)
95.45
79.62
(C) Multi-view (K' = 15) (w/o boosting)
94.03
78.52
(D) Singe view
94.03
78.52
Applicability of our ensemble classitier. Table 5 shows that our ensemble classifier can flexibly adapt to other prompt-based and CP-based methods and enhance their performance. We observe that as long as fa.p. (x) does not shift much from fa.p, (2), the linear classifier with the weight matrix W on the main label prediction branch can sufficiently generalize to handle this possible shift. The results depict the efficiency of our voting ensemble classifier in enhancing the performance of baselines, where RanPAC ++, SL.CA+, and HIDE++ are the versions of RanPAC, SLCA, and HIDE when using our ensemble classifier, respectively, after training the respective backbone.
Table 5: Using our ensemble classification head for baselines
Benchmark
Method
Split CIFAR-100
Split ImageNet-R
RanPAC
92.20
77.90
RanPAC ++
93.62
78.62
SLCA
91.62
77.10
SLCA ++
92.82
77.56
HiDE
92.61
75.06
HIDE ++
93.82
76.01
BoostCL-s
94.03
78.52
BoostCL-m
95.45
79.25
In the absence of Random Projection. To further demonstrate the benefit of out ensemble classifier, we remove the random projection, i.e. applying the multi-view strategy directly the original latent space. The results of this method are presented in Table (6] It can be seen that without Random Projection, hence no additional parameters for the projection matrix, outperforms RanPAC and HiDE. Here we also include BoostCL-m (d' = 768; K = 13) which has roughly the same number of parameter compared to RanPAC. This version's 's performance is slighly lower than the one without RP, showing that the benefit of random projection tends to occur when d' sufficiently large fc.f, in Figure 5)
BoostCL still significantly
26

---

Published as a conference paper at ICLR 2025
Table Performance (FAA %) of our method with and w/o using RP.
Benchmark
Method
Split CIFAR-100
Split ImageNet-R
RanPAC
92.20
77.90
HiDE
92.61
75.06
BoostCL-s
94.03
78.52
BoostCL-m (d' = 768; K = 13)
94.56
78.65
BoostCL-m (d' = 10, 000; K= 15)
95.45
79.25
BoostCL-m w/o RP & K 13
94.65
78.82
G.2 DETAILED ABLATION STUDY
To summarize each of our proposed components' pros, we provide detailed ablation study in Table with the following remarks:
Line 6 > RanPAC: the merits of Task-adaptive backbone.
Line 4 = Line = Line 6: multi-view without boosting is useless all the views produce the same prediction.
Line Line AdaBoost only brings slight improvement.
Line Line 3: Our voting strategy outperforms AdaBoost.
Line Line 6 and Line Line 2: Self-improvement process is helpful.
(Symbols >, refers to the comparison between two values of FAA).
Table 7: More detailed ablation study CDFIR-CIFAR-100
ID
self-improve
voting
boosting
RP(d'=10k, K=15)
RP(d'=10k, K=1)
FAA (%)
96.55
2
95,45
3
x
X
94.15
4
X
94.03
5
X
X
X
V
94.03
6
X
X
X
94.03
7
V
X
X
94.72
G.3 COMPLEXITY ANALYSIS (COMPUTATION AND STORAGE COSTS)
In this section, present detailed analysis of model's computational compared to baselines. Then, we show details related to our ensemble classifier and the self-improvement process. The reported results were conducted 2 machines with Tesla 2100-SXM2-32GB-12
G.3.1 TIME COMPLEXITY OF MODELS
Table g provides the time complexity of our method (w/o self-improvement process) and the two strongest baselines (RanPAC and HiDE), in terms of forward time and training time.
Table 8: Computational time, measured on Split CIFAR-100 (each image has size 3 x 224 x 224).
Method
Forward time (ms/sample)
Training time (min/task)
RanPAC
12.21 + 1.04
1.42 0.22
HiDE
21.88 1.08
42.62 + 1.35
BoostCI.
27.80 + 1.22
65.52 + 1.60
BoostCL. (backbone)
24.02 1.12
44.17 + 1.52
BoostCL (ensemble classifier)
3.89 = 0.31
21.41 + 1.15
27

---

Published as a conference paper at ICLR 2025
Forward time:
(1) In general, BoostCL takes the largest amount of time, due exploiting both prompt- based backbone and ensemble classifier. However, the cost gap between our method and the strongest prompt-based baseline, HiDE, is insignificant.
(2) The most expensive process is for the backbone forwarding. Meanwhile, the time spent on our proposed ensemble classifier, which works set of high-dimensional linear classifiers, only accounts for a small fraction (14%) compared to the total forward time.
Training time:
We can see that the time spent training our ensemble classifier (d' 10, 000; K 15) accounts for nearly one-third of the total time for each task. The reason is that our approach exploits the boosting strategy so that we have to train atomic views sequentially given that the time for training each view is mostly equal to the time for training each task of RanPAC (d' 10, 000). However, we believe this cost can be offset by the improvement we got.
G.3.2 TIME/SPACE COMPLEXITY OF OUR ENSEMBLE CLASSIFIE
To further analyze the cost and efficacy obtained when using random projection and multiple-view, we compare our ensemble classifier and RanPAC's classifier (Table
Storage cost:
When training BoostCI, (d' 10, 000; K K= 15), we exclusively use one Random Projection matrix for all atomic views. We also store internally, e.g. put to GPU, the Gram matrix and weight vector corresponding to the atomic view that is being trained, while the Gram matrices and weight vectors of other views stored externally. Therefore, essentially, the internal memory utilized for training our ensemble classifier (BoostCL with = 10, 000; = 15) is equivalent to that of RanPAC (d' 10, 000). Thus, in the table below, we only report the number of trained parameters used at testing time.
When testing, we see that the classifier of the version of BoostCL. (d' = 10, 000; K 15) requires 155 times more memory than RanPAC (d' = 10, 000). However, this expense can be compensated by the more-than-3% improvement in FAA that our BoostCL obtains. Moreover, considering BoostCL (d" = 768; K = 13), which requires less memory than RanPAC, we can see that with the help of our ensemble strategy, can still improve the FAA of RanPAC and even BoostCL single view (d' = 10, 000).
For the case that RanPAC uses d' 10, 000 x 15 150, 000, the total dimension in BoostCL. The cost to store the Gram matrix (with size d' x d') or its inversion is too huge and impractical. Meanwhile, BoostCL works well with this setting, as only needs to deal with submatrices.
Regarding the computational complexity:
Comparing with the original RanPAC (d' = 10, 000), forwarding the classifier of BoostCL (d' = 10, 000; K = 15) requires more time. This is obvious because we use an ensemble strategy for prediction instead of using only one classification head like RanPAC. This cost is offset by the performance improvement we achieve.
Last but not least, when discussing the potential of using projection matrices high- dimensional space, we claimed in our paper that our process helps to alleviate the "substantial computational cost when computing the inverse of a d' x d' Gram matrix for solving the linear classifier" which is done in RanPAC. In this table, we compare our BoostCL with RanPAC* (d' 10, 000 x 15) and mark that this version RanPAC infeasible to train because the cost when computing the inversion of a Gram matrix of size d' d'(= 10, 000 15) is (d') for some constant E [2.3,3.00 according to Geéron (017)). Therefore, BoostCI. has solved this computational crisis, saving the time cost for a factor of (119).
28

---

Published as a conference paper at ICLR 2025
G.3.3 TIME CONSUMPTION OF OUR SELF-IMPROVEMENT STRATEGY
We provide the experimental results depicting the time consumption of self-improvement strategy in Table 9 According to the results, such each step costs about 12ms for a sample. However, we just need 2 prompt prediction steps to significantly improve performance. This is worth compensating.
Table 9: Performance when increasing the number of prompt prediction steps (Split CIFAR- -00)..
Number of steps
Metric
1
3
5
FAA (%)
95.45
96.55
96.85
97.03
97.05
Time (ms)
27.80
39.02
50.95
61.84
73.85
G.4 INVERTIBLE AND EXPANSIVE NON-LINEAR ACTIVATION FUNCTIONS FOLLOWING THE RP
The Table below depicts model performance across different datasets when using the ReLU, Leaky ReLU (negative slop 3 0.01), and the activation function a(x) where { = 100, which we used our main experiments. The results show that the obtained FAA corresponding to these activation functions do not differ too much. This may be because these three functions are basically nearly equivalent, plus the property of the datasets make the results not differ much, making ReLU still work well when combined with RP although it violates the invertible condition in the region of x < 0. We leave a detailed explanation for future research.
& log exp(x))
Table 10: Performance when using different non-linear activation functions.
Split CIFAR-100
Split Imagenet-R
Activation function
BoostCL-s
RanPAC
BoostCL-s
RanPAC
Leaky ReLU
93,92
92.12
78.43
77.88
ReLUU
93.93
92.20
78.45
77.90
Our a(z)
94.03
92.25
78.52
77.96
29


Reviews:
[{"review_id": "oWNrHuJK8w", "reviewer": "Reviewer_HTXd", "summary": "This work considers the random projection (RP) strategy for pre-trained models in CL. Motivated by the benefits of RP in high-dimensional space, a multi-view strategy for an efficient classifier is further proposed, in which the principle of Adaboost is adapted to \novercome inherent obstacles and applied for the first time in CL. In addition, a self-improvement process technique, although simple, also shows significant effectiveness in selecting proper task\u0002specific prompts. The experimental results demonstrate a positive impact of the proposed method in improving model quality while only applying to linear classifiers.", "questions": "Does some ablation study be conducted on the proposed modules, Prompt selection process and Self-improvement process?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1)The proposed method BoostCL performs better than existing CL baselines, including Hide-prompt and ranpac.\n(2)This work addressed the challenge when applied AdaBoost directly to CL.\n(3)A self-improvement process, a simple but effective strategy is designed to help select prompts more accurately when inference.", "weaknesses": "As shown in Table 1, the proposed method takes no advantage in anti-forgetting, FFM metric.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ui9y22arNR", "reviewer": "Reviewer_WERc", "summary": "This paper presents BoostCL, a method for pretrained-based continual learning that leverages multiple views and random projection (RP) to enhance performance. The authors theoretically analyze the benefits of RP in higher-dimensional spaces and demonstrate its effectiveness through experimental results.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper provides a novel theoretical analysis of the benefits of random projection in higher-dimensional spaces for continual learning.\n2.\tThe authors conduct a comprehensive set of experiments to evaluate the performance of their proposed method, BoostCL, across various tasks and datasets.\n3.\tThe paper demonstrates the practicality of the proposed method by applying it to real-world continual learning scenarios.", "weaknesses": "1. The paper mentions the motivation behind using random projection, but it lacks a comprehensive discussion on why this approach is superior to other potential methods for feature transformation.\n2. While the paper proposes a novel method for continual learning, it would be helpful to see a more in-depth comparison with other recent methods in the literature. Specifically, how does BoostCL differ from and improve upon existing approaches?\n3. The technical details of the proposed method are somewhat challenging to follow. The paper could benefit from a more clear and concise explanation of the algorithm and its components. \n4. Discussing the scalability of the proposed method to larger datasets or more complex models would be relevant.\n5. The author needs to further analyze the consumption of the proposed method compared to other methods in terms of computational resources and training time.\n6. The paper could benefit from a more organized and focused presentation of the material. The introduction could be more clearly tailored to motivate the problem and the proposed solution. Additionally, the paper includes several appendices that contain detailed proofs and additional experimental results. While these appendices provide valuable information, they could be more effectively integrated into the main text to enhance the clarity and readability of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1u2ybyst4u", "reviewer": "Reviewer_gjF6", "summary": "This manuscript introduces \"BoostCL\", a novel approach to improve continual learning based on pre-trained models. The manuscript provides theoretical analysis to show that feature separability and model generalization ability benefit from random projection. Then a multi-view random projection is proposed to adapt AdaBoost for CL. A self-improvement process is also proposed for appropriately selecting prompts for each task sample. Empirical validation on multiple CL benchmarks is then conducted.", "questions": "1. How to effectively choose the best projection dimension to make appropriate trade-off between performance and computational overhead.\n2. How much gpu resource does the proposed method need, will the gpu memory usage increase rapidly by increasing projection dimension.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The manuscript provides theoretical analysis of Random Projection, demonstrating how RP improves feature separability in high-dimensional space. \n2. The manuscript is clear and well organized. The formulas are presented in an unambiguous manner.", "weaknesses": "A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence; if you believe experiments are insufficient, explain why and exactly what is missing, etc.\n\n1. Although RP and multi-view strategies are theoretically shown to improve feature separability, there is limited discussion on the theoretical upper bound of generalization error.\n2. The manuscript does not provide theoretical guidance on choosing the optimal projection dimension in RP, which can greatly impact model performance for different tasks.\n3. The proposed self-improvement process for prompt selection could increase the model’s inference complexity, but this potential trade-off is not fully analyzed, especially in scenarios with many tasks.\n4. Minor spelling errors, like “sammple” in line 250.\n5. More experiment regarding hyper-parameters like number of views, projection dimensions, and threshold settings for the voting mechanism should be conducted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kIsYSsFFCZ", "reviewer": "Reviewer_H1Kx", "summary": "This paper provides some theoretical analysis on Random Projection (RP) which project features into a higher-dimensional space to improve their linear separability. Furthermore, this paper proposes a Multi-View Random Projection scheme to create a robust ensemble classifier motivated by AdaBoost. Then the proposed method is applied to prompt-based CL methods for a better performance. Experimental results demonstrate its efficacy in various continual learning settings.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper provides theoretical insights into Random Projection (RP), reinforcing its foundation in an intuitive manner.\n\n2.\tThe motivation for the proposed Multi-View Random Projection is clear and well-justified, demonstrating its potential to enhance performance.\n\n3.\tThe paper is well-organized and easy to understand, addressing most of my queries within the main text or the appendix.", "weaknesses": "**Major**\n\n1.\tThe necessity of the huge views should be clarified. As mentioned in the paper, the subsequent atomic views aim to capture the complement knowledge of previous views. However, the final classifier only leverages a part of these views rather than all of them, which may omit some complement knowledge.\n\n2.\tThe theoretical results are interesting, but the method is heuristic. For example, the design of the voting strategy and the self-improvement are quite complex. It seems that a lot of tricks are introduced directly to improve performance.\n\n3.\tThe design of the self-improvement process is not intuitive, which may accumulate errors in multiple-step prediction. Specifically, if an incorrect prompt is chosen in the first step, the model may become biased towards predicting the class associated with that wrong prompt, potentially leading to accumulated errors and hindering improvements in prompt prediction accuracy.\n\n4.\tThe experiments do not include comparisons with simpler ensemble methods, such as randomly sampling views or using all views for the ensemble. Such comparisons can provide more intuitive insights to demonstrate the effectiveness of the proposed method.\n\n**Minor**\n\n1.\tThere are too many notations which may limit the fluency and readability. For example, in line 177, there is no description for the notation $\\gamma (s,g,\\mathbb{R}^d)$ and no explanation for the symbolic $g$. So please simplify the notations as much as possible.\n2.\tTypo in Corollary 4.4 (line 250), “sammple” should be “sample”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)