BECAME: Bayesian Continual Learning with Adaptive Model Merging
Authors: Mei Li
Yuxiang Lu
Qinyan Dai
Suizhi Huang
Yue Ding
Hongtao Lu
Venue: ICML 2025 poster (2025)
Keywords: Continual Learning
Incremental Learning
Model Merging
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We provide therotical insights into model merging in continual learning and propose the optimal solution for adaptive merging.

Abstract: Continual Learning (CL) strives to learn incrementally across tasks while mitigating catastrophic forgetting. A key challenge in CL is balancing stability (retaining prior knowledge) and plasticity (learning new tasks). While representative gradient projection methods ensure stability, they often limit plasticity. Model merging techniques offer promising solutions, but prior methods typically rely on empirical assumptions and carefully selected hyperparameters. In this paper, we explore the potential of model merging to enhance the stability-plasticity trade-off, providing theoretical insights that underscore its benefits. Specifically, we reformulate the merging mechanism using Bayesian continual learning principles and derive a closed-form solution for the optimal merging coefficient that adapts to the diverse characteristics of tasks. To validate our approach, we introduce a two-stage framework named BECAME, which synergizes the expertise of gradient projection and adaptive merging. Extensive experiments show that our approach outperforms state-of-the-art CL methods and existing merging strategies https://github.com/limei0818/BECAME.

BECAME: BsyE Continual Learning with Adaptive Model MErging
Mei Li Yuxiang Lu"
Qinyan Dai
Suizhi Huang
Yue Ding'
Hongtao Lu'
Abstract
0
Continual Learning (CL) strives to learn incre- mentally across tasks while mitigating catas- trophic forgetting. A key challenge in CL is bal- ancing stability (retaining prior knowledge) and plasticity (learning new tasks). While represen- tative gradient projection methods ensure stabil- ity, they often limit plasticity. Model merging techniques offer promising solutions, but prior methods typically rely on empirical assumptions and carefully selected hyperparameters. In this paper, we explore the potential of model merg- ing enhance the plassiityyl trade-off, providing theoretical insights that underscore its benefits. Specifically, reformulate the merg- ing mechanism using Bayesian continual learn- ing principles and derive a closed-form solution for the optimal merging coefficient based on the Laplace approximation that adapts to the diverse characteristics of tasks. To validate our approach, we introduce two-stage framework named BE- CAME, which synergizes the expertise of gra- dient projection and adaptive merging. Exten- sive experiments show that our approach outper- forms ofate-of-the-art CL methods and existing merging strategies. Code is available at https: //github. www.o//mmmemmmmmemmmmmmmmmemeeee 0818///CAM.../.... BECAME.
4-- C.
Stabiliy
Adaptive Merging
0,
02
Plasticity
Figure I. (Left) The training loss landscape for task and task 2, represented by red and blue contours, respectively. Darker color denotes lower loss value. (Right) The training loss values for task 1, task 2, and their sum. Both figures are based on the NSCL-hased experiment on the 10-split CIFAR- 100 dataset. The model initially learns task 1, reaching 0J, which minimizes the loss L1. When learning task 2, the model first obtains 65" using the gradient projection, with minor forgetting indicated by the increase in L1. This solution shows limited plasticity, as La remains high, The model then proceeds to train without constraints, reaching 02, the minimum of L2. By analyzing the trajectory from 0 to Ôz, our method can determine the optimal merging coefficient, achieving the minimal cumulative loss at 02.
gradient descent using gradients computed solely from the current task data, the loss on prior tasks can increase dras- tically (Hadsell et al., 2020). This phenomenon, known as catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999). poses major challenge in CL. Generally, CL mod- els target to mitigate catastrophic forgetting by retaining knowledge from prior tasks (stability) while also being able to assimilate new tasks (plasticity). A balance between sta- bility and plasticity is necessitated achieve strong overall performance across both old and new tasks (Chaudhry et al., 2018; 8; Mermillod et al., 2013; Kim & Han, 2023).
1. Introduction
Continual Learning (CL) aims to equip models to leam in- crementally, similar to human learning (Wang et al., 2024; De Lange et al., 2021; Mundt et al., 2023; Masana et al., 2022; Van de Ven et al., 2022). In the typical CL setting, models can only access the data of the current task due to privacy or storage constraints, yet they are expected to per- form well on both the current and previously learned tasks. Since neural network models typically optimized via
Projecting gradients of new data onto the subspace orthog- onal to the feature space of previous tasks a prominent method to ensure stability (Zeng et al., 2019; Wang et al., 2021b: lb; Saha al., 2021). Nevertheless, this approach often restricts plasticity, as imposes strict constraints on param- eter optimization, reducing the range of feasible gradient directions (Zhao et al., 2023). A promising solution involves merging the model's parameters with those of a complemen- tary model specialized in the current task (Lin et al., 2022a). This idea of merging models of old and new tasks has also been empirically shown to enhance overall performance
"quual contribution 'Shanghai Jiao Tong University. Corre- spondence to: Yue Ding <dingyue @ @jt..cdu..n..........
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
across diverse scenarios (Lee et al., 2017: Marouf et al., 2024). However, these methods often acquire interpretation by assuming that learning new tasks is minimally influenced by previous knowledge, and then approximate the posterior of each task with an independent Gaussian distribution (Lee et al., 2017). Considering the sequential nature of learn- ing, this assumption may not align with the context of CL. where tasks are inherently interrelated. Thus, the funda- mental question remains to be answered: How can model merging effectively improve the stability- trade-off in continual learning?
without constraints to further enhance plasticity (02). By synergizing the complementary strengths of these two mod- els, our approach achieves superior performance compared to using either model independently, as indicated by the low- est cumulative loss in Figure right. Moreover, our method offers a flexible framework that supports a wide variety of existing CL approaches, providing a general solution for consistent performance improvements.
In summary, our contributions follows:
Capitalizing on Bayesian continual learning, we the- oretically prove the existence of merged model that adaptively balances the plassiityyy trade-off and derives closed-form solution for the optimal coeffi- cient based on the Laplace approximation.
Moreover, determining the merging coefficients for models is critical challenge. A straightforward technique is to average the parameters of all models, as commonly adopted in prior work (Lin et al., 2022a; Lee et al., 2017: Marouf et al., 2024). Nonetheless, because different tasks often possess diverse characteristics, such as varying loss land- scapes, assigning equal importance to all models can be suboptimal. While this limitation can be addressed by intro- ducing tunable hyperparameters to adjust the importance of old and new tasks (Lee et al., 2017; Marouf et al., 2024), a major drawback lies in the complexity of this process and the risk of failing to find optimal parameters due to limited grid search. This naturally raises another question: Can we derive the optimal solution in close-form for adaptively merging models?
We propose simple yet effective method that not only substantiates our theoretical insights but also enhances the performance ceiling of various gradient projection methods, offering a general approach for exploiting plasticity.
We evaluate our method on four widely-used contin- ual learning benchmarks, showing that it remarkably outperforms previous state of the art, particularly in metrics measuring plasticity.
For the first question, we provide theoretical insights demon- strating that merging models can indeed yield a better opti- mum in CL (Section 3.2). Specifically, we show that along the path connecting the parameters of the old and new tasks, there always exists point where the cumulative loss across all tasks is lower than at either endpoint. Furthermore, we reformulate the merging paradigm from the perspective of Bayesian continual learning (Kirkpatrick et al., 2017; Ritter et al., 2018; Kao et al., 2021), bridging the gap between prior work and the complexities of task interdependence. Regarding the second question, we aim identify an opti- mal merging point that maximizes overall performance. To this end, we perform MAP estimation based on the posterior of sequentially learned tasks. We demonstrate that the opti- mization objective is convex along the linear merging path, enabling us to derive closed-form solution for the optimal merging coefficient upon the Laplace approximation (Sec- tion 3.3). This solution adaptively integrates the parameters by considering the distinct loss landscapes of the tasks and the impact of parameter shifts, thereby achieving a balance between stability and plasticity.
2. Related Works
Continual Learning. Existing Continual Learning (CL) methods can be broadly categorized into five main types: replay-based (Chaudhry et al., 2019b; Rebuffi et al., 2017: Schwarz et al., 2018). regularization--ased (Kirkpatrick et al., 2017: Li & Hoiem, 2017: Ritter et al., 2018; Shin et al., 2017), architecture- basedd (Mallya & Lazebnik, 2018; Serra et al., 2018; Yoon et al., 2018; Hung et al., 2019; Liang & Li, 2023), representation-based (Javed & White, 2019; Pham et al., 2021; Madaan et al., 2022: Mehta et al., 2023), and optimization et al., 2021b; Saha et al., 2021) (Wang et al., 2024). Gra- dient projection methods, subset of optimization- based approaches, achieve strong performance in mitigating for- getting by constraining updates to the orthogonal subspace of input features, while this may limit adaptability. Subse- quent works (Saha & Roy, 2023; Lin et al., 2022b; Yang et al., 2024) have primarily focused on enhancing model plasticity.
approaches (Zeng et al., 2019; Wang
To validate our findings, we introduce a two-stage frame- work named BECAME. The first stage involves optimizing the model using gradient projection (denoted by O5P in Fig- ure 1), ensuring stability with only minor increase in loss from the previous optimum (0;). The model is then trained
Model Merging. Research model merging generally falls into two categories: merging models fine-tuned a single task to enhance overall generalization (Cha et al., 2021; Wortsman et al., 2022; Jordan et al., 2023), and merg- ing models trained on distinct tasks to produce a unified multi-task model capable of handling all tasks from the
2

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
0
24
50
0
02
OSP
02
02
000
0
0
130
16
99
(a) Train Loss on Task
(b) Train Loss on Task 2
(c) Cumulative Train Loss
30
0
23-
16
19-
0
02
OCP
6
0
6GP
0
(d) Test Accuracy Task
(c) Test Accuracy Task (f) Average Test Accuracy
Figure 2. Visualization of the training loss and test accuracy landscapes for task task 2, cumulative training loss, and average test accuracy. The figures are derived from the NSCL-based experiment conducted the 10-split CIFAR-100 dataset. The model is initially trained on task 1. starting from a random initialization to obtain 0j. Next, the model undergoes training on task with gradient projection, yielding 05"h Further training task 2 is then performed without constraints, resulting in 0z. Our adaptive method then merges 05 and &₂ to locate the optimal point that minimizes training lass along the trajectory from 6 to 0z.
components (Matena & Raffel, 2022; Ilharco et al., 2023; Yaday et al., 2024). Model merging techniques are also applicable in continual learning (Stojanovski et al., 2022; Simon et al., 2022; Lee et al., 2020), where a critical chal- lenge lies in determining the merging strategy. One intuitive approach involves using constant weight related the number of tasks (Lin et al., 2022a). Alternatively, IMM (Lee et al., 2017) and CoMA (Marouf et al., 2024) manually set coefficients through exhaustive tuning.
identity. Each D, is independent of others. The goal of continual learning is to achieve strong joint performance across all T tasks after the model has been trained on each task sequentially. Specifically, while learning task f, only the current training data D; is accessible, yet we aim to minimize the cumulative loss over all tasks learned far Di: U D;:
0 arg min (1; arg min
C, D:). (1)
In contrast to existing merging methods for CL. we de- rive a closed-form solution that not only achieves optimal performance but also eliminates the need for manual adjust- ments across different scenarios. Additionally, our method grounded in Bayesian continual learning, providing ro- bust theoretical foundation that aligns model merging with the core principles of CL.
For simplicity, we denote C, as (, and (1; (1.a) as (1= in following sections.
Considering the nature of incremental learning, is difficult to model the distribution previous tasks (D), DD, 1 and new task I1 simultaneously when training the network the new task, which leads the critical challenge of -laatitty trade-off.
3. Methodology
3.1. Preliminary
3.2. Merge Models for Better Minimum
Under the setting of continual learning, network fo with parameters 0 R1 is sequentially trained on T tasks. Each task t is defined by a subset of training samples D. = ((x(") (n) y (")))"())) following a unique distribution Dr. where (m) represents the input data and y((") the cor- responding labels, with t E 2..,...., T}} denoting the task
To prove the effectiveness of model merging in continual learning, begin with standard setting. Suppose the model has been trained on tasks 1 to - 1, with its param- eters 07- representing the empirical optimal solution for the objective defined in Eq. 1. Next, the model is trained on task via a simple gradient descent optimizer, arriving at 0, by minimizing the task specific loss L.. Given that the two
3

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
models are trained sequentially, their parameters are likely to reside in the same basin of the loss landscapes (Mirzadeh et al., 2021; Neyshabur et al., 2020). Generally, there should not be significant loss barriers along the linear path from 0--1 to dr when the parameters are weighted averaged. The interpolated model along this path could maintain high accu- racy, comparable to both endpoints (Lee et al., 2017; Good- fellow et al., 2015; Marouf et al., 2024). In the following lemma, we prove that there exists point on this trajectory where the merged model can achieve better minimum for the cumulative loss overall learned tasks Lid, indicating proficiency in both previous and new knowledge.
3.3. Determine Optimal Merging Coefficient
While Lemma 3.1 demonstrates that merging previous and current models can provide an improved solution for con- tinual learning, determining the optimal merging coefficient remains a critical challenge. To address this, we derive a closed-form expression for the optimal coefficient upon the Laplace approximation by formulating the continual learn- ing process from the perspective of Bayes' rule (Wang et al., 2024; Ritter et al., 2018; Kao al., 2021).
The posterior after learning task is expressed as:
Lemma 3.1. Given model parameters 0i-1 which minimize loss L1--1, and new parameters 6, optimized from 07_1 to local minima of Lu, there exists a coefficient À E [0.1]
p(O)Di.)) o p(0) II p(D.0) a P(D. (6)
satisfying that
where p(0) denotes the prior over the parameters. This equation implies that the posterior from previously observed tasks becomes the prior for task t, allowing the new posterior to be computed based solely on the new data.
C1:((1-AVi S (in((...(0 C1: (0.))- (2)
Proof. With the merged parameters ((A) = (1 1)0 +
According the maximum a posteriori probability (MAP) estimation, the optimal parameters 07 after learning task t are
2o:, we take the derivative of (I(()())) with respect to X:
(((A)
2X
0. = arg max log p (i::)
(OCI,(0(A)))
OA(A) Ox
(0(A)
(3)
arg max log (D, 10) + (0)
(7)
( (0L1=-1(((A))
(c,(0(2)))
(0. - 87-
00(2)
a0(2)
Alternatively, since the negative log likelihood log (())()) is typically used as the task-specific loss (.),,, which extend Eq. 7 as:
At A = 0, 0(A) = 01-1. given that
BC14 00
0 (as
10=07
0--1 minimizes 11st-1), we have
(0(A) 0X
@h 08
1) S O, (0. )
07 = arg max
log pa PD,)0)) + log
A=
18=8;_1
(8)
(4)
because O is obtained by gradient descent from 0 direction 0,- 18-1 is aligned with the negative gradient of L, at the point 0:-1-
1 so the
arg min
(O).
where the log prior log p(0) is not related to the optimization when it follows uniform distribution. Eq. 8 indicates that the objective formed by MAP is equivalent to minimizing the cumulative loss introduced in Eq. 1.
Similarly, when A = 1, where 0(A) = 6, since the model converges to minima where gradient vanishes, iee. = O, we have
BC
l0=6, 80
As the posterior (P(-. is generally intractable, it is a common strategy to use Laplace approximation (MacKay, 1992), approximating it by a multivariate Gaussian: (t-; A,71) (Huszár, 2018; Kirkpatrick et al., 2017: Ritter et al., 2018), where A is positive semi-definite. Similarly, p(0)Di:) m
OL (0(x)((A)) aA
(DL1a-1
)'
T '0,0- (0: 0--1) 2 0,
A=1
00
0=6,
(5)
p (( R (0)
as learning a new task generally increases the loss over pre- vious tasks, making the direction from 0:-1 to Ô, positively related to the gradient of C1+ at θ:.
(0) = -N(0; (e, Thus, we can rewrite the MAP
in Eq. 7 as:
Since L1 (0))) is continuous in X E [0, 1), with conditions on two endpoints of the merging path established in Eq. 4 and 5, there exists coefficient À such that (I.)((A)) 5
0 = arg max log (D,((()))) - _ - 1--11) A pt-1),
min (frx 1), (0.))
(9)

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
where Me-1 07-1 represents the MAP parameters after
Algorithm Pseudo-codes for BECAME
learning task i 1. Since the MAP is often optimized by
Input: T sequential tasks with data (Di, ......Dr model fo initialized with parameters Og
minimizing the loss function, Eq. 9 can be reformulated as:
1 (0- -0--)" A (0-0; -1-1)- (10) 8 min L,( 2
Output: Trained model parameters 07
1: Train on task D1 to get Bj
2: Compute precision matrix with Fisher Information
A1 = (6))))
Next, we substitute the merged parameters 0()) = (1 2)0211 + vo, into Eq. 10. Defining A0 - O. 0'-1, the objective becomes:
3: for / = 2.3,..,,777 do
Train on task Dr with gradient projection get 6
5: Train on task D. without constraint to get Ó:
1 10000 A11100. (ia(A)= = Lo (A- 1)40) + 2
Compute the current Fisher Information (r,)).
Derive merging coefficient X; via Eq. 20
(11)
Merge models 0; = (1 - >; x²) + X50,
Then we perform a second-order Taylor expansion of the first term around 0,,
Update precision matrix A, = Fi ((;)) + Ar-1
10: end for
C, (O())) ~ C,(,,) + (A (A- - 1)20 VoColond. - 1)>20 V/Colo= A
the precision matrix A, can be computed as the Hessian of the negative log posterior:
+ - -
A = Vi log log (0
(.) + 2a - 1)2 T
V2 92feloab. A0,
(18)
= V log (D, + AL-1
(12)
where VoL; is the gradient and VgCa is the Hessian with respect to 8. As 9 represents the parameters at local minima of L., the gradient VoLrle=e, equals zero, and the Hessian ViColesd, positive semi-definite.
where the first term is the Hessian of the negative log likeli- hood at μ4 0r. Eq. 18 means that the precision matrix is updated recursively, initialized by the Hessian of the nega- tive log prior, which is typically constant.
Taking the derivative with respect to X, we have:
Nevertheless, computing the Hessian can be computation- ally expensive due the large scale of model parameters. Moreover, for the precision A₃ to be valid, it is required to be positive semi-definite, which is not always guaranteed when approximating with the Hessian (Wang et al., 2024; Ritter et al., 2018). To address these limitations, the Fisher Information Matrix (FIM) is used as an approximation of the Hessian (Geisser et al., 1990; Martens, 2020; Kirkpatrick et al., 2017; Ritter al., 2018), as it positive semi-definite by construction:
aCr,(A) = (2 - (V
VoLelom AB + AAO A AA.
0x
(13)
Considering the derivatives at the endpoints X = 0 and
= 1, we have:
(L)
=
VVLooooo A 10S00 SO,
(14)
ON
A=0
(crx(A)
AO
A,-
A0 1 0.
(15)
A=1
(((0) V log (D
(19)
We also consider the second-order derivative:
82 Lia(A) ax2
Therfore, Xj can be calculated by the following equation:
= VT At + AAT A 10 2 0,
(16) indicating that (A) convex along the merging path. With the property that the derivative is opposite at two end- points, there exists a unique optimal X" in [0, 1] that min-
AOT F,
N AOT (FF(0). + + F((O)))00
(20)
Furthermore, the FIM can be simplified with a diagonal approximation (Huszár, 2018; Kirkpatrick et al., 2017).
imizes the objective when the derivative in Eq. 13 equals zero, yielding the closed-form solution:
AO
V
0=0 AO
3.4. BECAME
(17)
AO
9- A. A)
In Lemma 3.1, we establish an upper bound for the cumu- lative loss after merging, (o(A() ), at the lower value of the two endpoints on the merging path. To further tighten this bound, we introduce a two-stage training paradigm. In
Then we continue to introduce how to calculate Aj in prac- tice. By performing a quadratic approximation (()))..),
5

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
Table Performance comparison GPM-based experiments. denote the results from previous work. We reproduce GPM, TRGP, SGP, and GPCNS under our setting for a fair comparison. The best ACC is marked in bold, and the second-hest is underlined.
20-Split CIEAR-100
10-Split CIFAR-100
20-Split MinilmageNet
Method
ACC(%))
BWT(%)
ACC(%)
BWT(
ACC(%)
BWT(%))
Multi-task
85.24 + 0.59
79.52 + 0.59
77.78 + 1.49
EWC (Kirkpatrick al., 2017)
75.30 + 0.70
-6.30 + 0.60
68.80 0.88
-20.00 + 1.00
52.10 1.10
-9.30
1.40
A-GEM" (Chaudhry et al., 2019a)
63.98 + 1.22
-15.00 + 2.00
57.24 + 0.72
-12.00 + 1.00
FS-DGPM' (Wang al., 2021a)
80.50 + 0.40
-3.00 0.40
74.33 0.31
-3.00 +
ER-Res' (Chaudhry et al., 2019b)
79.20 + 0.40
.4900 + 0.50
71.73 + 0.63
-6.00 + 1.00
55.20 + 2.90
-5.70 + 0.80
GPM (Saha et al., 2021)
77.34 + 0.55
0.01 0.07
71.81 = 0.10
-0.11 0.33
63.90 = 1.05
-1.30 = 1.27
GPM + Ours
80.57 + 0.18
10.06:0.1
75.05 + 0.27
0.02 + 0.39
67.62 + 1.40
0.87 = 0.26
TRGP (Lin et al., 2022b)
81.68 + 0.69
-0.13
75.01 + 0.63
-0.01 0.16
62.68 + 2.49
-1.04 + 0.78
TRGP + Ours
82.61 + 0.38
-0.32 0.18
75.87 0.43
-0.77 0.27
65.09 + 1.81
-0.65 1.08
SGP (Saha & Roy, 2023)
80.21 + 0.68
-0.88 + 0.29
74.97 + 0.31
-0,98 + 0.30
66.99 + 2.33
-2.46 + 1.28
SGP + Ours
81.94 + 0.40
-1.22 0.25
76.74
0.36
-1.42
70.06 2.12
-0.26 0.72
GPCNS (Yang et al., 2024)
78.63 + 0.63
-1.93 0.42
71.84 + 0.93
0.85
62.85 + 2.18
-1,90 + 1.75
GPCNS Ours
80.87 + 0.24
-1.96 1 0.24
73,89 0.52
-3.46
64.79 1.71
-2.07 1.66
the first stage, the model is initialized from the last state
Perform second-order Taylor expansion of (1x-1(((PP) around 0;1 and let d == 0e 0 there is:
0--11 and trained with gradient projection to obtain dGP. Then in the second stage, we continue training from 8CP without constraints to reach é,
11.-1 1 + +d VoLLu-1
(23)
The key idea of gradient projection is to maintain the network's outputs when fed inputs from previously seen
+ 2d d VoL2=-110=0p VV d.
tasks (Wang et al., 2021b; Saha et al., 2021; Saha & Roy, 2023), leading nearly identical output logits and minimal influence on the losses of previous tasks, i.e.,
where VoLia-1lo=0 = 0, since 07-1 is an optimum for L1--11. Moreover, there is (0(P)) ss 11=-1(01-1).
we have
L1 ( & (0i-) Thus, we can assume that the projected model remains at the local minima of the previous losses:
d Vo 0.
(24)
Recall that L1--1 - (00g((0/ppp ((0(P (Eq. 8) and Ar- = -V2 1o
QL12-1 08
0c11-1
(Eq. 18), we can re-
0,
(21)
00
A=0;_1
place V2L1=-110-01 with Ar- to get
which implies that Lemma 3.1 stills holds when replacing the starting point 0 on the path with 0CP Additionally, we can approximate the posterior of previous tasks using the new parameters, facilitating smooth loss surface around the path (Lee et al., 2017; Marouf et al., 2024). This allows us to substitute AO with 0: 8GP when determining the optimal coefficient in Eq. 20.
A'A-id=0 0.
(25)
Moreover, by decomposing A via cigendecomposition as A, = Q AQ, where A is a diagonal matrix containing the eigenvalues, there d Ar- d Q AQd
Since A is positive semi-definite, the eigenvalues Au 2 O require that each element of Qd must be zero, leading to A = (Q A)(Qd) = 0. Therefore, we can get A (0CP 0/-1) ) = 0, and the second term in Eq. 22 is zero. Thus, we have:
Proof. To justify this substitution theoretically, we prove that Eq. 10 holds when 6;_ is replaced by &GP.
Given that A is symmetric, we can expand the second term (0- (GP)T Ar-1 OCP) in Eq. 10 as:
(0 (0S) A- (0 000) = (0 17-1)
0; A, T
(26)
(T TA, A 17-1) (0 - 0;-1)
+ (2(0- 07-1) (0 O(P)) T "A- (00 0CP)
This equality validates our substitution of A8 with 0, BGP in Eq. 20.
(22) Then we demonstrate that A, (0,_ BGP) = 0, which can result in that the second term in Eq. 22 is zero so that 0i_ can be replaced by 8CP in Eq. 10.
Additionally, as the loss of new task is substantially reduced compared to 07_ through training, can entail lower
6

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
Table Performance comparison of NSCL--based experiments. denotes results from Connector (Iin et al., 2022a), which does not report the standard deviation. We reproduce Adam-NSCL under our setting for fair comparison.
20-Split CIEAR-100
10-Split CIEAR-100
25-Split TinyImageNet ACC(%)
Method
ACC(%)
BWT(%)
ACC(%)t
BWT(%)
BWT(%)
Multi-task
90.81 0.10
95.06 + 0.18
82.55 0012
OWM" (Zeng et al., 2019)
68.47
-3.37
68.89
-1.88
49.98
-3.64
EWC (Kirkpatrick et al., 2017)
71.66
-3.72
70.77
-2.83
52.33
-6.17
SI" (Ostapenko et al., 2019)
59.76
-8.62
60.57
-5.17
45.27
-4.45
LWF* (Li & Hoiem, 2017)
74.38
-9.11
70.70
-6.27
56.57
-11.19
GD-WILD' (Lee et al., 2019)
77.16
-14.85
71.27
-18.24
42.74
-34.58
Connector" (Lin et al., 2022a)
80.80
-5.00
79.79
-0.92
64.61
-6.00
Adam-NSCL (Wang et al., 2021b)
75.66 + 0.39
-3.85
+ 0.45
72.91 = 0,30
-1.76 = 0.20
58.57 + 0,40
-5.74 + 0.61
Adam-NSCL + Ours
81.88 1 0.37
.4.56 + 0.42
81.66 = 0.51
-0.80 = 0.53
66.49 = 0.48
.4.97 = 0.73
cumulative loss (O()) < (0 This guarantees better upper bound on the merged model's loss, which is consistently lower than that of the projected model, re- sulting in an improved minimum after merging with the second-stage model. The complete pipeline of BECAME outlined in Algorithm
all datasets. Each task has separate classifier head under the task-incremental learning setup.
Implementation Details. For the first task, the training pro- cess is identical to that of the corresponding baselines. For subsequent tasks E 3, ..... , .T), our method involves two stages as mentioned above. All experiments are re- peated with 5 random seeds, and we report the mean and standard deviation of the results. The hyperparameter con- figurations in both stages are mostly consistent with those of the baselines except subtle adjustments for adapting our methods, with details provided in Appendix B.4 to ensure reproducibility.
4. Experiments
4.1. Experimental Setup
Baselines. Our experiments are divided into two categories. The first category is based on GPM (Saha et al., 2021) and its extensions: TRGP (Lin et al., 2022b), SGP (Saha & Roy, 2023), and GPCNS (Yang et al., 2024). We refer to these as "GPM-based experiments". The second category based on Adam-NSCI. (NSCL) (Wang et al., 2021b), which we similarly term "NSCL-based experiments". In addition to these baseline methods, we also compare our approach with
Metries. We use two evaluation metries for the main ex- perimental results: average accuracy (ACC) and backward transfer (BWT) (Lopez-Paz & Ranzato, 2017). ACC mea- sures the overall test performance across all Iearned tasks, while BWT measures the stability. Besides, we use the Intransigence Measure (IM) (Chaudhry et al., 2018) to as- sess plasticity. Definition of the metrics can be found in Appendix B.5.
OWM (Zeng et al., 2019), Connector (Lin et al., 2022a), EWC (Kirkpatrick et al., 2017), SI (Ostapenko et al., 2019), LwF (Li & Hoiem, 2017), GD-WILD (Lee et al., 2019), A-GEM (Chaudhry et al., 2019a), FS-DGPM (Wang al., 2021a), and ER-Res (Chaudhry al., 2019b). Appendix B.1 introduces these methods in more detail.
4.2. Main Results
Visualization. In Figure 2, we present a detailed visualiza- tion of the training loss and test accuracy landscapes for the first two tasks of the 10-split CIFAR-100 dataset. The plots display the performance of different model parameter configurations, specifically 0j, 0CP, 0z, and 0z. The gap between 05P and 6g illustrates the limited plasticity of gradi- ent projection, with the path from 85Th 02 representing the trade-off. The plots clearly demonstrate that the merged model 83 achieves the optimal balance be- tween the performance on both tasks, as shown in panels (c) and (f). The placement of 03 along the path suggests that it precisely identifies the optimal merging point, minimizing the optimization objective and significantly outperforming both BGP and 02 in overall performance.
Datasets. We conduct our experiments on four widely- used benchmarks: 20-Split CIFAR-100 (Krizhevsky et al., 2009), 10-Split 1100R-100, 25-Split TinylmageNet (Wu et al., 2017), and 20-Split MinilmageNet (Vinyals et al., 2016). k-Split means the dataset is evenly divided into k tasks. Details can be found in Appendix B.2.
Architecture. To ensure a fair comparison, we use the same network architectures as those employed in the baselines. For GPM-based experiments, we utilize a 5-layer AlexNet- like network as the backbone for the 10-Split and 20-Split CIFAR-100 datasets, and a reduced version of ResNet- 18 (He et al., 2016) for the 20-Split MinilmageNet experiment. For NSCL-based experiments, we employ ResNet- 18 across
7

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
10. CIFAR-100
20. M Minimma
Table Performance comparison of different merging strate- gies on 10-Split CIEAR-100 (10-S CIFAR) and 20-Split Mini- ImageNet (20-S Mini) based on GPM. The results demonstrate that model merging enhances overall performance. Our method strikes a superior balance by maximizing plasticity while minimiz- ing forgetting.
GPM
DRI
(4SS)) (W(T)%))
[NN BM17150
10-Spilt CIPAR-100
25- Thyylmagtten
oue
(M6(4) (WTT))
Dataset
Strategy
ACC(S))
BWT(%)t
10
71.81
-0.11
1/t (Lee et al., 2017)
74.71
-0.40
Figure 3. BWT and IM in GPM-based and NSCL-based ex- periments. Our method outperforms baselines by reducing IM (plasticity, lower is better) while maintaining a good BWT (stabil- ity, higher is better).
10-S CIFAR
CoMA (Marouf et al., 2024)
74.07
-2.79
CoFiMA (Marouf' et al., 2024)
74.36
-1.77
Ours
75.05
0.02
63.90
-1.30
1/t (Lee et al.. 2017)
66.09
0.57
20-$ Mini
across all three benchmarks. IMM is reported on differ- ent benchmarks, and CoMA/CoFiMA rely on a pre-trained model, which is distinct setting in CL. To ensure fair and accurate comparison, we implement their merging strategies within our framework and use the hyperparameters reported in their papers. Table 3 shows that all merging strategies improve accuracy over the baseline without merging, provid- ing empirical support for our theoretical insights. Among these strategies, our adaptive coefficient achieves the high- est ACC and performs well in terms of BWT. In contrast, CoMA, which uses a fixed coefficient, falls short in BWT as it tends to prioritize the most recent tasks. Performance comparison without gradient projection training stage is available in Appendix C.3.
CoMA (Marouf et al., 2024)
66.91
-0.82
CoFiMA (Marouf et al., 2024)
64.19
0.54
Ours
67.62
0.87
Comparison with state of the art. Table presents the results of the basedd experiments. Our method con- sistently achieves higher accuracy across all three datasets compared to four baselines. Notably, boosts the accuracy of GPM by over 3% on all datasets. Beyond improving the performance of the corresponding baseline, our method applied to GPM also surpasses TRGP, SGP, and GPCNS, which are alternative GPM improvement methods, on the 10-Split CIFAR- 100 and 20-Split MinilmageNet datasets. Furthermore, the increase in accuracy is achieved without compromising the ability to mitigate forgetting, demonstrat- ing a
Paastictt-Stabbiin Trade-off. For a specific task we can find the relationship between the final accuracy AT, BWT, and IM,, formulated as:
trade-off.
Table 2 reports the results of the NSCL-based experiments. As shown in the table, our method remarkably enhances the performance of NSCL. On 20-Split CIFAR-100, our method improves ACC by over 6%. For two more challenging datasets, 10-Split CIFAR-100 and 25-Split TinylmageNet, the improvement is even more pronounced, with an ACC increase of 8%. Additionally, our approach outperforms Connector in all experiments, which is regarded the state of the art among NSCL-based methods.
Ar, A - IM, + BWT;
(27)
where BWT; and IM; are calculated individually for each task. Since A; remains constant for a given backbone re- gardless of the chosen approach, A7, depends on IM, and BWT, Optimal performance is achieved when IM is mini- mized (i.e., better plasticity) and BWT is maximized (i.e., better stability). Figure illustrates this trade-off by vi- sualizing BWT and IM using horizontal bars. The length of each bar represents the absolute value of the respective metric. By substantially minimizing IM, our method outper- forms both GPM and NSCL, enhancing plasticity towards the upper bound while preserving stability across different scenarios.
4.3. In-depth Analysis
Merging Strategy. To validate the effectiveness of our adaptive merging, we compare it with existing merging strategies proposed by Connector (Lin et al., 2022a), IMM (Lee et al., 2017), and CoMA (Marouf et al., 2024). These methods typically merge models using a fixed coefficient based on the number of learned tasks (1/t) or tunable constant based on grid search. CoFiMA (Marouf et al., 2024) further introduces a parameter-wise weigtt- averaging with a tunable hyperparameter.
Balance Across Tasks. In experimental studies, it is often assumed that all tasks hold equal importance during testing. Nonetheless, in real-world applications, task importance can vary due to difference in the number of samples per task. This variation highlights the necessity of achieving inter-task balance. To evaluate this, we include the standard deviation of the final accuracy (STD) as an additional metric. Considering both ACC and STD is essential: when methods attain similar ACC, a lower STD reflects less bias among
Since Connector (Lin et al., 2022a) is also based on NSCL,, we can directly compare it with our approach. As shown
in Table 2, our method consistently surpasses Connector
8

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
Spin CFAR-100
25-5ple Tmyimagetie
Table 4. Efficiency comparison. While our method requires addi-
Accurooyna
tional time and GPU memory due to the second training stage, it achieves state thethe-ate art performance with competitive efficiency.
providing substantial advantage over the baselines.
Ours (STD:
Ours (STD:
Method ACC(%)
Training Time (s)
GPU Mem Usage (MB)
57
24
21
GPM
63,90
434,66
347.26
GPCNS
62.85
258.13
998.41
SGP
66,99
490.84
347.26
Figure 4. Final test accuracy each task in GPM-based and NSCL-based experiments. Our method simultaneously enhances the accuracy and balance across different tasks (STD).
TRGP
62.68
776.72
1854.32
GPM + Ours
67.62
584.14
375.72
SGP + Ours
70.06
675.99
375.72
10-Sein FFAA-100
25-5pit Thyymagetie
dataset. The results indicate that our approach enhances performance while requiring significantly less memory than GPCNS and TRGP, and it achieves shorter training time than TRGP. Although the second stage increases the overall training time, it yields an improvement of more than 3% over the baselines. In addition, updating the Fisher infor- mation is computationally efficient, taking only 0.69s per task comparable to training for single epoch and our method adds no overhead during inference. More results on efficiency are presented in Appendix C.4.
58
NECL (AOR: 57 Ous (ADA 63.70)
NSCL (ADA: 45,77)
Ours (ADA) 52.531
38
3
15
14
15 20
Tuck
Taee
Figure 5. After one epoch accuracy (AOA) in GPM-based and NSCL-based experiments. Note that the training process is the same until the second stage of task 2. Our methods demonstrate superior generalization for learning new tasks compared to the baselines.
5. Conclusion
tasks. As shown in Figure 4, the accuracy curves of our method not only surpass those of the baseline methods but also exhibit smaller fluctuations, indicating more consistent task performance.
In this paper, we demonstrate that model merging can im- prove the overall optimum by identifying point along the merging path where the cumulative loss is minimized. Additionally, we bridge the gap between prior work and the complexities of task interdependence with theoretical insights based on Bayesian continual learning, offering a closed-form solution for the merging coefficient that bal- ances stability and plasticity. Experiments confirm the effec- tiveness of our proposed framework BECAME on multiple CL benchmarks. Our findings underscore the potential of model merging to enhance continual learning, providing flexible and generalizable solution that can be seamlessly incorporated into existing approaches for improvements.
Generalization and Forward Transfer. Forward Transfer (FWT) (Lopez-Paz & Ranzato, 2017) is originally calculated the test accuracy on task t after the model has learned tasks from to t 1, to evaluate how knowledge from previous tasks aids the learning of new tasks. However, before the model sees the data of new task, the classification head cannot correctly map neurons to class IDs, hindering the classification of unseen tasks. Instead, we introduce a new metric called after one epoch accuracy (AOA) assess generalization and forward transfer, which is defined as:
with 1,- 1,
Impact Statement
T
AOA-
epoch=1
(28)
Our research adheres high ethical standards in machine learning, emphasizing transparency, reproducibility, and faimess throughout all experiments. Although our approach demonstrates promising results, it shares the common limita- tions of computer vision models, particularly in data usage. We use publicly available datasets accordance with all relevant legal and ethical guidelines, ensuring careful data handling during selection and preprocessing.
T
where A 1 represents the test accuracy on task af- ter training for one epoch on data D; (in the first stage of our framework). Figure shows that our method ob- tains higher accuracy on nearly all tasks compared to the baselines. highlighting strong inter-task generalizability that accommodates the heterogeneous distribution of new tasks.
To facilitate reproducibility, we provide detailed implemen- tation specifications, including models, datasets, and train- ing setups, in Appendix B. Our code is publicly available at https: / thub. com/ 11 me 0818/ /BECAME.
Efficiency. Despite requiring second training stage, our method maintains good balance between efficiency and performance. As shown in Table 4, we compare the training time and GPU memory usage of our method with those of corresponding baselines on the challenging MinilmageNet
9

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
Acknowledgement
Goodfellow, J., Vinyals, O., and Saxe, A. M. Qualitatively characterizing neural network optimization problems. In International Conference on Learning Representations, 2015.
This work is supported by the National Nature Science Foun- dation of China (62176155) and Shanghai Municipal Sci- ence and Technology Major Project 22021HH2ZX00102)
Hadsell, R., Rao, D., Rusu, A. A., and Pascanu, Embrac- ing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028-100) 2020.
References
Arpit, D., Wang, H., Zhou, Y., and Xiong. C. Ensemble of averages: Improving model selection and boosting per- formance in domain generalization. Advances in Neurul Information Processing Systems, 35:8265-8277,2
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Cha, J., Chun, S., Lee, K., Cho, H.-.., Park, S., Lee, Y., and Park, S. Swad: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems, volume 34, pp. 02405-22418 2021.
Hung, C.-Y., Tu, C.-H., Wu, C.-E., Chen, C.-H., Chan, Y.-
M., and Chen, C.-S. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems, volume 32, 2019.
Chaudhry, A., Dokania, P. K., Ajanthan, T., and Torr, H. Riemannian walk for incremental learning: Understand- ing forgetting and intransigence. In European Conference on Computer Vision, pp. 532-547,2015
Huszár, F. Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences, 115(L1)E2299-EE249)) 2018.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a-gem. In Interna- tional Conference on Learning Representations, 2019a.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In International Conference on Learning Representations, 2023.
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H., and Ranzato, M. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486 2019b.
Javed, K. and White, M. Meta-learning representations for continual learning. In Advances in Neural Information Prucessing Systems, volume 32, 2019.
Chen, C., Du, Y., Fang, Z., Wang, Z., Luo, F., Li, P., Yan, M., Zhang, J.. Huang, F., Sun, M., et al. Model composi- tion for multimodal large language models. In Proceed- ings of the 62nd Anmual Meeting of the Association for Computational Linguisties ( (Volume I: Long Papers), pp. 11246-11262, 2024.
Jin, X., Ren, X., Pecotiuc-Pietro, D., and Cheng, P. Data- less knowledge fusion by merging weights of language models. In International Conference on Learning Repre- sentations, 2023.
Jordan, K., Sedghi, H., Saukh, O., Entezari, R., and Neyshabur, B. Repair: Renormalizing permuted activa- tions for interpolation repair. In International Conference on Learning Representations, 2023.
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. A continual learning survey: Defying forgetting in classifi- cation tasks. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 44(7):3366-333) 3385, 2021.
Kao, T.-C., Jensen, K., van de Ven, G., Bernacchia, A., and Hennequin, G. Natural continual learning: success is journey, not (just) a destination. In Advances in Neural Information Processing Systems, volume 34, pp. 28067- 28079, 2021.
Ebrahimi, S., Meier, F,, Calandra, R., Darrell, T., and Rohrbach, M. Adversarial continual learning. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 386-402. Springer, 2020.
Kim, D. and Han, B. On the stability-plastiiiy dilemma of class-incremental learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 20196- 20204, 2023.
French, R. M. Catastrophic forgetting in connectionist net- works. Trends in cognitive sciences, 3(4):128-13 1999.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Brabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the Na- tional Academy of Sciences, 1 114(13):3521-355 2017.
Geisser, S., Hodges, J., Press, S., and ZeUner, A. The va- lidity of posterior expansions based on laplace's method. Bayesian and likelihood methods in statistics and econo- metrics, 7:473, 1990.
10

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov,
A. D., and Van De Weijer, J. Class-incremental learning: survey and performance evaluation image classifica- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5) 2022.
Lee, J., Joo, D., Hong, H. G., and Kim, Residual continual learning. In AAAI Conference on Artificial Intelligence, volume 34, pp. 4553-4560, 2020.
Matena, M. S. and Raffel, C. A. Merging models with fisher- weighted averaging. In Advances in Neural Information Processing Systems, volume 35, pp. 17703-17716, 2022.
Lee, K., Lee, K., Shin, J., and Lee, H. Overcoming catas- trophic forgetting with unlabeled data in the wild. In IEEE International Conference on Computer Vision. pp- 312-321,
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 109-165. Elsevier, 1989.
Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., and Zhang, B.- T. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems, volume 30, 2017.
Mehta, V., Patil, D., Chandar, S., and Strubell, E. An empirical investigation of the role of pre-training life- long learning. Journal of Machine Learning Research, 24(214):1-50, 2023.
Li, and Hoiem, D. Learning without forgetting. JEEE Transactions on Pattern Analysis and Machine Intelli- gence, (0(12))2935-294) 2017.
Liang, Y.S. and Li, W.-J Adaptive plasticity improvement for continual learning. In IEEE Conference on Computer Vision and Pullern Recognition, pp. 7816-7825, 2023.
Mermillod, M., Bugaiska, A., and Bonin, P. The stability- plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4:504, 2013.
Lin, G., Chu, H., and Lai, H. Towards better plasticity- stability trade-off in incremental learning: A simple linear connector. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 89-98,999988 2022a.
Mirzadeh, S. I., Farajtabar, M., Gorur, D., Pascanu, R., and Ghasemzadeh, H. Linear mode connectivity multitask and continual learning. In International Conference on Learning Representations, 2021.
Lin, S., Yang, L., Fan, D., and Zhang, J. Trgp: Trust region gradient projection for continual learning. In Interna- tional Conference on Learning Representations, 2022b.
Mundt, M., Hong, Y., Pliushch, I., and Ramesh, V. wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge active and open world learning. Neural Networks, 160:306-336. 2023.
Lopez-Paze Paz,D D. and Ranzato, M. Gradient episodic memory for continual learning. In Advances Neural Information Processing Systems, volume 30, 2017.
MacKay, D.J. A practical bayesian framework for backprop- agation networks. Neural computation, 4(3):448-47) 1992.
Neyshabur, B., Sedghi, H., and Zhang, C. What is being transferred transfer learning? Advances in Neural Information Processing Systems, volume 33, pp. 312-523, 2020.
Madaan, D., Yoon, J., Li, Y., Liu, Y., and Hwang, S. J. Representational continuity for unsupervised continual learning. In International Conference on Learning Rep- resentations, 2022.
Ostapenko, O., Puscas, M., Klein, T., Jahnichen, P., and Nabi, M. Learning to remember: synaptic plasticity driven framework for continual learning. In IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 11321 2019.
Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. IEEE Conference on Computer Vision and Puttern Recognition, pp. 7765-7773. 2018.
Pham, Q., Liu, C., and Hoi, S. Dualnet: Continual learn- ing, fast and slow. In Advances in Neural Information Processing Systems, volume 34, pp. 16131-16144, 2021.
Marouf, I. E., Roy, S., Tartaglione, E., and Lathuilière, S. Weighted ensemble models are strong continual learners. In European Conference on Computer Vision, 2024.
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Martens, J. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1 2020.

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
Ritter, H., Botev, A., and Barber, D. Online structured laplace approximations for overcoming catastrophic for- getting. In Advances in Neural Information Processing Systems, 2018.
Information Processing Systems, volume 34, pp. 22379- 2021a,
Wang. L.. Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Saha, G. and Roy, K. Continual learning with scaled gradi- ent projection. In AAAI Conference on Artificial Intelli- gence, volume 37, pp. 9677-9685, 2023.
Wang, S., Li, X., Sun, J., and Xu, Z. Training networks in null space of feature covariance for continual learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 184-193, 2021b.
Saha, G., Garg, I., and Roy, K. Gradient projection memory for continual learning. In International Conference on Learning Reprexentations, 2021.
Schwarz, J., Czarnecki, W. Luketina, J., Grabska- Barwinska, A., Teh, Y. W., Pascanu, R., and Hadsell, R. Progress & Compress: scalable framework for continual learning. In ICML, pp. 4528 2018.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned mod- els improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965-23998, PMLR, 2022.
Serra, J., Suris, D., Miron, M., and Karatzoglou, A. Over- coming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp. 4548-4557. PMLR, 2018.
Wu, J., Zhang, Q. and Xu, G. Tiny imagenet challenge. Technical report, 2017.
Shin, H., Lee, K., Kim, J., and Kim, J. Continual learn- ing with deep generative replay. In Advances in Neural Information Processing Systems, volume 30, 2017.
Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. Ties-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems, volume 36, 2024.
Simon, C., Faraki, M., Tsai, Y.H.,, Yu, X., Schulter, S., Suh, Y., Harandi, M., and Chandraker, M. On generalizing beyond domains in cross-domain continual learning. In IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 9265-9274, 2022.
Yang, C., Dong, M., Zhang, X., Qi, J., and Zhou, A. In- troducing common null space of gradients for gradient projection methods in continual learning. In ACM Inter- national Conference on Multimedia, 2024.
Stojanovski, Z., Roth, K., and Akata, Z. Momentum- weight interpolation of strong zero-shot models for con- tinual learning. In NeurlPS Workshop, 2022.
Yoon, J., Yang, E., Lee, J., and Hwang, S. Lifelong learning with dynamically expandable networks. In Inter- national Conference on Learning Representations, 2018.
Sung, Y.-L., Li, L., Lin, K., Gan, Z., Bansal, M., and Wang, L. An empirical study of multimodal model merging. In Findings of the Association for Computational Linguis- tics: EMNLP 2023, pp. 1563 1575, 2023.
Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y. Language mod- els are super mario: Absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning, 2024.
Van de Ven, G. M., Tuytelaars, T., and Tolias, A. S. Three types of incremental learning. Nature Machine Intelli- gence, (115):1155-119) 2022.
Zeng, G., Chen, Y., Cui, B., and Yu, S. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8):364-377 3322 2019.
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, volume 29, 2016.
Zhao, Z., Zhang, Z., Tan, X., Liu, J., Qu, Y., Xie, Y., and
Ma, L. Rethinking gradient projection continual learning: Stability/plasticity feature space decoupling. In IEEE Conference on Computer Vision and Patern Recognition. pp. 3718-3727, 2023.
Wan, F., Huang, X., Cai, D., Quan, X., Bi, W., and Shi, S. Knowledge fusion of large language models. In Interna- tional Conference on Learning Representations, 2024.
Wang, L., Zhang, M., Jia, Z., Li, Q., Bao, C., Ma, K., Zhu, J., and Zhong, Y. AFEC: Active forgetting of negative transfer in continual learning. In Advances Neural
12

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
The appendix is organized follows:
Appendix A reviews relevant work on gradient projection- continual learning methods and model merging.
Appendix B provides a detailed description of the experimental setup, including supplementary information on the baselines (B.1), dataset statistics (B.2), network architecture (B.3), implementation details (B.4), and evaluation metrics (B.5).
Appendix C is divided into five parts: Section C.I presents experimental results on additional datasets, Section C.2 shows the performance with the average anytime accuracy (AAA) metric, Section C.3 compares different merging strategy without the gradient projection training stage, Section C.4 reports the detailed memory usage of GPM-based experiments and efficiency results of NSCL-based experiments, and Section C.5 includes supplementary plots.
A. Additional Related Work
Gradient Projection Methods. The gradient projection method is a significant branch of optimization- based continual learning approaches. Specifically, during training. the method restricts the update direction of parameters to the orthogonal subspace of the input features each layer of the network. OWM (Zeng et al., 2019) ensures that weight modifications during training for new tasks occur exclusively in directions orthogonal to the input space of previously learned tasks, thereby preventing interference with prior knowledge. Adam-NSCI. (Wang et al., 2021b) constrains network parameter updates to lie within the layer-wise null space of input features from previous tasks and incrementally computes the null space based on the uncentered covariance of these features. GPM (Saha et al., 2021) divides the gradient space into two parts and uses learned representations to preserve past knowledge while enabling new task learning through gradient steps that avoid interference with the stored Core Gradient Space. While gradient projection methods exhibit strong suppression of forgetting, they also limit the model's ability to learn new tasks. Several works aim improve the model's plasticity based gradient projection methods. Connector (Lin et al., 2022a) combines Adam-NSCL with regularization bassed approaches and employs linear interpolation to achieve a better balance between stability and plasticity in the final model. SGP (Saha & Roy, 2023) enhances new learning by not only allowing gradient updates orthogonal to the feature spaces but also projecting on the feature spaces themselves. TRGP (Lin et al., 2022b) introduces a trust region that selectively reuses relevant old task weights, facilitating effective forward knowledge transfer while minimizing interference with previously learned tasks. GPCNS (Yang et al., 2024) projects current gradients into the common null space of previous tasks' gradients, utilizing both old gradient and feature information.
Model Merging. Model merging has emerged as promising direction for efficiently synthesizing the capabilities of multiple fine-tuned models while reducing data and computational demands. Task Arithmetic (Ilharco et al., 2023) introduces the idea of task vectors that encode -ssciiicc knowledge via the difference between fine-tuned and pre-trained weights. Other approaches such as RegMean (Jin et al., 2023) and FisherMerging (Matena & Raffel, 2022) go beyond simple averaging, leveraging local linear regression or parameter importance scores via the Fisher Information Matrix. Further, methods like -erg-Merging (Yadav et al., 2024) and DARE (Yu et al., 2024) improve merging by resolving parameter interference through pruning and sign alignment strategies. When merging heterogeneous models-those with different architectures the challenge becomes more complex. Methods such DAMC (Chen et al., 2024) and FuseLLM (Wan et al., 2024) adopt strategies like parameter decoupling, token alignment, and knowledge distillation. Across these efforts, model merging has been applied to improve generalization (Cha et al., 2021: Arpit et al., 2022), multitask learning (Jin et al., 2023), multimodal fusion (Sung et al., 2023), and continual learning (Marouf et al., 2024).
B. Experiment Details
B.I. Baselines
Since our method is proposed based on gradient projection, the primary objective of our experiments is to validate whether adaptive merging can improve the plasticity of gradient projection methods. We mainly compare method with corresponding gradient projection baselines. Additionally, we include comparisons with the Connector (Lin et al., 2022a), which has the same model capacity and enhances gradient projection by combining it with parameter interpolation and parameter regularization on Adam-NSCL (Wang et al., 2021b). Among all compared methods, OWM (Zeng et al., 2019), Connector, GEM (Lopez-Pat Paz& & Ranzato, 2017), A-GEM (Chaudhry et al., 2019a), and FS-DGPM (Wang et al., 2021a) are related to gradient projection. EWC (Kirkpatrick et al., 2017) and SI (Ostapenko et al., 2019) are egularization-basee
13

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
methods, whereas LwF (Li & Hoiem, 2017), GD-WILD (Lee al., 2019), and ER-Res (Chaudhry et al., 2019b) replay-based methods.
B.2. Datasets
Table and Table 6 summarize the datasets used in our experiments. For GPM-based experiments, the dataset is split into 95% for training and 5% for validation, with no data augmentation applied across all three datasets. In NSCL-based experiments, the entire training dataset is utilized, with data augmentation applied via a random crop with 4-pixel padding and random horizontal flip.
Table 5. Dataset statisties for GPM-hased experiments.
20-Split CIFAR-100
10-Split CIFAR-100
20-Split MinilmageNet
Number of Tasks
20
10
20
Input Size
3 x 32 x 32
3 x 32 32
3 84 x 84
Classes per Task
5
10
5
Training Samples per Task
4,750
4,750
2,375
Validation Samples per Task
250
250
125
Test Samples per Task
1,000
1,000
500
Table 6. Dataset statistics for NSCL-based experiments.
20-Split CIFAR-100 10-Split CIFAR-100
20-Split MinilmageNet
Number of Tasks Input Size
20
10
25
x 32 x 32
32 32
3 x 64 x 64
Classes per Task
10
8
Training Samples per Task
5,000
5,000
4,000
Validation Samples per Task
Test Samples per Task
1,000
1,000
400
B.3. Architecture
AlexNet-like Architecture: This architecture mirrors that of Serra et al. (2018), with the addition of batch normalization applied to all layers except the classifier. It features three convolutional layers with filter sizes of 64, 128, and 256, and kernel sizes of 4 x 4, 3 3, and 2 respectively. These are followed by two fully connected layers, each comprising 2048 units. Activation functions throughout are rectified linear units (ReLU), and x max-pooling is applied after each convolutional layer. Dropout is employed with rates of 0.2 for the first two layers and 0.5 for the remaining layers.
Reduced ResNet18 Architecture: This design is adapted from Lopez-Paz & Ranzato (2017). For the MinilmageNet experiment, the first layer incorporates a convolution with a stride of 2. In both the MinilmageNet and 5-Datasets experiments, the 4 4 wverage-pooling layer preceding the classifier is replaced with a 2 x 2 average caeraag-pooling layer. ReLU activations are used the hidden layers, while the final layer utilizes softmax with cross-entropy loss.
B.4. Implementation Details
GPM-based Experiments. We validate our method using GPM (Saha et al., 2021) and its three improved variants: TRGP (Lin et al., 2022b), SGP (Saha Roy, 2023), and GPCNS (Yang et al., 2024). To ensure fair comparison, the experimental setup is consistent with the corresponding baselines.
The initial learning rate (1r) is reduced by factor of Ir-factor if the validation loss does not improve 1r pat panience ence consecutive epochs. Training terminates when the learning rate falls below 1 min or the maximum number of epochs (n.epochs) is reached. The threshold for selecting orthogonal bases (c()) is initialized as en for the first task and incremented by Ça for each subsequent task. The scale coefficient (a) corresponds to the parameter used SGP Hyperparameters for the GPM- based experiments across different baselines are detailed in Table 7. The experiments on both 20-Split CIFAR-10 1000 and 10-Split CIFAR-100 share identical hyperparameters.
14

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
Table Hyperparameter settings for each GPM-based baseline. indicates that the parameter value sourced from the corresponding papers supplementary materials, while other values derived from the code.
20/10-Split
CIFAR-100
20-Split MinilmageNet
Hyperparameter
GPM
TRGP
SGP
GPCNS
GPM
TRGP
SGP
GPCNS
1r
0.01
0.01
0.05
0.05*
0.1
0.1*
0.1*
0.1"
Ir.min
10-5
10-5
5
10-5
5 10-5
10-3
10-3
10-3
10-3
Ir.patience
6
6
6
6
5
5
5
Ir.factor
2
3
3
n epochs
200'
200
200"
200*
10"
100*
10"
200'
batchsize
64"
64
64
64
10"
64"
10"
64
6D
0.97
0.97
0.97
0.97
0.985
0.985
0,985
0.98
EA
10-3+
3+
x 10-3+
3
10-30
3
10-3
3
10-1+
3 x 10-4+
10-4x
10-3
Q
10"
5
1
3
NSCL-based Experiments. Following Adam- (Wang et al., 2021b), use the Adam optimizer. The learning rate is set to 10-1 for the first task and 5 10-5 for subsequent tasks. To adapt Adam-NSCI. to our method, we decay the learning rate by factor of 0.5 if training loss does not decrease for consecutive epochs, and each stage stop when the condition occurs for the third time. The maximum number of training epochs per stage 80. The batch size 32 for 10-Split CIFAR- 100, and 16 for both 20-Split CIFAR-100 and 25-Split Imagee.
Parameters in the batch normalization layer are regularized using EWC (Kirkpatrick al., 2017) with regularization coefficient of 100. The hyperparameter in Adam-NSCL is 30 for 20-Split CIFAR-100 and 10 for the other two benchmarks. During training
task, only the backbone network and the classifier for the current task are updated.
All experiments are performed on single NVIDIA GeForce RTX 4080 GPU.
B.5. Metrics
We use two evaluation metrics for the main experimental results: average accuracy (ACC) and backward transfer (BWT) (Lopez-Paz& & Ranzato, 2017). ACC measures the overall test performance across all learned tasks, while BWT is quantified by the decrease in test accuracy on previously learned tasks relative to their initial accuracy upon learning. We also use the Intransigence Measure (IM) (Chaudhry et al., 2018) to assess plasticity. IM calculated as the difference between the upper-bound performance under the multi-task setting and the accuracy achieved immediately after learning each task. These metrics are defined as:
ACC Σ AT. BWT
T Σ
A Are. IM Σ
A A
(29)
Here, T denotes the total number of tasks, and Ar,i represents the test accuracy of the model after training on task and evaluated on task i, and A represents the accuracy on task i, evaluated using a model trained from random initialization on the combined datasets Ul D
C. Additional Results and Plots
C.1. Experimental Results on Additional Datasets
In addition to CIFAR-100, MinilmageNet, and TinylmageNet, we evaluate our proposed method on the 5-Datasets (Ebrahimi et al., 2020) and CIFAR-100 100 Superclass (Yoon et al., 2018) benchmarks. The 5-Datasets benchmark is combination of five different datasets: SVHN, CIFAR-10. not-MNIST, Fashion-MNIST" and MNIST. Each sub-dataset contains 10 classes and treated as a separate task. The task order significantly influences performance, learning on the CIFAR-I0 sub-dataset considerably more challenging than the other four sub-datasets.
For both the 5-Datasets and CIFAR- 100 Superclass benchmarks, the network architecture and hyperparameters are identical to those used in the corresponding baselines. The training configuration for the second stage follows that of the first stage in each experiment.
As shown in Table our method consistently improves baseline performance. The performance improvement on the
15

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
Table Additional Performance comparison of GPM-based experiments. run experiment on different dataset order. The best ACC is marked in bold, and the second-best is underlined. Since GPCNS does not provide code for the 5 D-aaaset benchmark, the results marked with are reproduced by us based on available code 10-Split CIFAR-100.
5-Datasets
CIFAR-100 Superclass
Method
ACC(%) T
BWT(%) %)
ACC(%)
BWT(%))
GPM (Saha et al., 2021)
84.70 3.17
-488 4.37
57.49 0.65
-1.03 + 0.37
GPM Ours
84.89 3.10
-3.92 4.61
58.21
0.22
-0.60 = 0.17
TRGP (Lin et al., 2022b)
89.99 0.58
-0.75 0.62
57.85 0.43
-0.93 + 0.60
TRGP Ours
90.63 + 0.87
0.50
58.07 0.56
-0.71 = 0.30
SGP (Saha & 2023)
83.77 + 3.28
+
58.24 0.39
-1.85 + 0.54
SGP + Ours
84.02 + 3.18
66.24
59.03 0.32
-1.13 = 0.28
GPCNS (Yang et al., 2024)
73.89 = 5.92"
- 0.0188200
57.68 0.84
-3.23 = 1.38
GPCNS + Ours
76.92 + 5.30
-15.51 = 7.32
58.41 0.13
-2.33 = 0.23
5-Datasets benchmark is less than on the 10-Split CIFAR-100 due to two primary factors. First, the key driver performance improvement on the 5-Datasets benchmark is enhancing the model's learning capability on CIFAR-10, whereas gradients have minimal influence learning effectiveness for sub-datasets like MNIST. Second, due the model's poor performance on CIFAR-10, 10, it tends to assign a larger weight CIFAR-10 1- during parameter merging, leading limited enhancement for other tasks.
A similar pattern is observed on the CIFAR-100 Superclass dataset, where new task learning under gradient projection may also converge to local optima.
C.2. Results with the AAA metric
Despite ACC, we also compute Averaged Anytime Accuracy (AAA) using the formula: AAA = AA, with AA, T - Aij. In Tables 9 and 10, we present comparative analysis of ACC and AAA metrics different benchmarks side by side. These results confirm that our method continues to yield substantial improvements when evaluated with AAA, with trends closely aligning with ACC.
Table Performance comparison GPM-hased experiments with ACC and AAA metrics. The best AAA is marked bold, and the second-best is underlined.
20-Split CIFAR
10-Split CIFAR
20-Split MinilmageNeet
Method
ACC(%))
AAA(%)f
ACC(%)
AAA(%)
ACC(%)
AAA(%
GPM (Saha et al., 2021)
77.34
77.31
71.81
72.08
63.90
63.02
GPM + Ours
80.57
79.61
75.05
74.22
67.62
66.05
TRGP (Lin et al., 2022b)
81.68
79.96
75.01
74.24
62.68
62.55
TRGP Ours
82.61
80.78
75.87
74.89
65.09
64.48
SGP (Saha Roy, 2023)
80.21
78.78
74.97
73.62
66.99
61.97
SGP + Ours
81.94
79.44
76.74
74.82
70.06
64.98
GPCNS (Yang et al., 2024)
78.63
76.22
71.84
71.50
62.85
60.53
GPCNS Ours
80.87
78.08
73.89
72.94
64.79
61.53
C.3. Merging Strategy Compatison without GP
To isolate the impact of the gradient projection methods, we directly evaluate different model merging strategies in CL setting. Except lack of the first training stage, all other training configurations remain the same as in the "Merging Strategy" part of Section 4.3.
As shown in Table 11, our adaptive merging method achieves the highest ACC on both 10-Split CIFAR100 and 20-Split MinilmageNet, outperforming fixed-coefficient methods like 1/t and CoMA. Furthermore, our method maintains relatively
16

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
Table Performance comparison of NSCL-based experiments with ACC and AAA metrics. best AAA is marked in bold.
20-Split CIFAR
10-Split CIFAR
25-Split TinyImageNet
Method
ACC(%))
AAA(%)
ACC(%)f
AAA(%)
ACC(%)
AAA(%)
Adam-NSCL (Wang et al., 2021b)
75.66
75.43
72.91
74.77
58.77
60.21
Adam Ours
81.88
81.25
81.66
81.94
66.49
66.65
high BWT. indicating better retention of previous tasks. These results demonstrate the robustness and effectiveness of our adaptive merging strategy compared to other merging strategies.
Table I. Performance comparison without gradient projection different merging strategies on 10-Split CIFAR-100 and 20-Split MinilmageNet based GPM.
10-Split CIFAR100
20-Split MinilmageNet
Method
ACC(%)
BWT(%))
ACC(%))
BWT(%)f
Finetune
57.98
-20.19
57.06
-10993
1/6 (Lee et al., 2017)
60.69
-1,44
47.54
0.91
CoMA (Marouf al., 2024)
64.42
-9.79
60.99
-5.00
CoFiMA (Marouf al., 2024)
61.17
-0.38
62.26
-1.40
Ours
65.03
-1.26
64.75
-2.11
C.4. Additional Efficiency Results
Detailed memory usage on GPM-based experiments. To provide detailed comparison, we divide the total memory into three parts: model parameters, the additional memory for variables needed for training future tasks, and the temporary memory required only during training. We further divide the additional memory based on training stage for analysis. Table 12 shows that:
The second training stage introduces only a modest storage overhead. As shown in the 5th and 6th rows of the table, the only additional memory for the second training stage is the precision matrix, which is comparable in size to the model parameters and determined solely by the model architecture rather than by the baseline method.
The increment of the additional memory in the first training stage is direct result of improved overall performance, which follows the inherent feature of the gradient projection methods. After performing model merging, the model's capability of feature extraction is enhanced so that the dimension of output feature space expands and more base vectors need to be stored for future task training.
The memory overhead of our method competitive to GPCNS and TRGP, which require more storage than our method even though they only have one training stage.
Efficiency analysis on NSCL-based experiments. We evaluated the training time and GPU memory usage in NSCL- based experiments, as shown in Table 13. In this table, Adam-NSCL, serves as the baseline for our method, while Connector represents an alternative approach designed to improve upon Adam-NSCL. Our method demonstrates a significant performance improvement over Adam-NSCL. Furthermore, not only surpasses Connector in performance but also requires significantly less training time and memory.
C.5. Additional Plots
Plasticity-Stabilit Trade-off. Figure 6 displays the BWT and IM from the GPM-based experiments, presented as a single plot for each dataset. The symbol + denotes our method applied to the corresponding baseline. Compared to GPM. all other three baselines including GPCNS, TRGP, and SGP show improved plasticity, albeit with some potential reduction
17

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
Table 12. Detailed memory usage on GPM-based experiments.
Additional Memory (MB)
Method
ACC (%)
Model Parameters (MB)
Temporary Memory (MB)
1st Stage
2nd Stage
GPM
63.90
4.74
37.51
305.01
GPCNS
62,85
4.74
584.26
409.41
SGP
66.99
4.74
53,47
289.05
TRGP
62,68
73.23
1359.18
421.91
GPM+Ours
67.62
4.74
55.88
4.73
310.37
SGP+Ours
70.69
4.74
60.49
4.73
305.76
Table 13. Efficiency comparison with NSCL-based methods on 25-Split Tiny ImageNet.
Method
ACC (%)
Training Time (h) GPU Mem Usage (MB)
Adam-NSCA (Wang al., 2021b)
58.57
5.22
1728.71
Connector (Lin et al., 2022a)
64.61
8.53
1766.66
NSCL+Ours
66.49
6.30
1748.77
in stability. Notably, our method further enhances plasticity relative to the baselines, contributing to improved overall performance.
Figure presents the BWT and IM from the NSCL-based experiments. Our method achieves significant improvements in IM.
20-Split CIFAR-100
10-Split CIFAR-100
20-Split MinilmageNet
GPM
GPM
GPM
G
GPM+
GPM+
GPCNS
GPONS
GPONS
G
GPCNS
GPCNS
TRGP
TRGP
TRGP
TRGP
TRGP
TRGP+
SGP
SGP
IM(%)
IM(%) BWT(%)
S
BWT(
SGP+
BWT((%)
SGP+
Figure 6. BWT and IM for GPMe- Experiments. Each dataset is represented in single plot. The symbol indicates method applied to the corresponding baseline.
Balance Across Tasks. Figure 8 depicts the final accuracy across all tasks for the GPM-based experiments, with each column representing a specific dataset. Our method not only surpasses the baselines in ACC but also achieves a low standard deviation. Similarly, Figure presents the results for the NSCL-based experiments.
Generalization and Forward Transfer. Figure 10 and 11 illustrate the accuracy after one epoch across all tasks for GPM-based and NSCL-based experiments respectively.
Coefficient Loss. We sample the merge coefficient from 0 to with step size of 0.05 analyze the impact of different coefficients the cumulative training loss of tasks learned. Figures 12 and 13 illustrate the results of experiments based on GPM and Adam-NSCL, respectively, on the 10-Split CIFAR-100 dataset. The vertical dashed lines in the figures indicate the coefficients computed using our method, which identify point with the minimum cunulative loss for different tasks. comparison of Figures A and reveals that the optimal merge coefficients differ across models on the same dataset. Our method adaptively identifies the optimal merge coefficient.
18

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
20-Spiit CIFAR- 100
10-Spiit CIFAR-100
25-Split TinylmageNet
NSCL
NSCL
NSCL
(M1%) BWT(%)
Ours
Ours
BWT((%)
Ours
BWT(1%)
12
15
12
15
33
12
15
18
Figure? 7. The BWT and IM from the NSCL-based experiments.
20-Split CIFAR-100
10-Split CIFAR-100
20-Split MimilmageNeet
Acturacy (96)
Accuracy
Accuracy (96)
78
63
68
GPM (STD: 4.25)
GPM
GPM (STD: 4.07)
4.01)
(STD: 2.29)
(STD: 3.81)
5
13
15
19
10
15 17
Task
Task
Task
20-Split CIFAR-100
10-Split CIFAR-100
20-5plit MinilmageNet
81
78
Accuracy(%)
80
Accuracy|
(cccuray(%)
68
58
48
GPCNS (STD: 6.60)
GPCNS (STD: 5.13)
GPCNS (STD: 8.64)
Ours (STD: 6.46)
Ours (STD: 5.03)
Ours (STD: 8.61)
5
13
51
Task
Task
Task
20-Split CIFAR-100
10-Split CIFAR-100
20-Split MinilmageNet
Accuracy
82
76
Accuracy1%)
72
62
66
TRGP (STD: 4.46)
TRGP (STD: 3.14)
TRGP (STD: 5,53)
Ours (STD: 4.43)
Ours (STD: 3.08)
Ours (STD: 5.39)
62
5
13
56
42
1
15
Task
Task
Task
20-Split CIFAR-100
10-Split CIFAR-100
20-5plit MinilmageNee
84
Accuracy 1%)
77
Accuracy(%)
74
64
67
54
68
SGP (STD: 5.26)
SGP (STD:
44
SGP (STD: 9.43)
Ours (STD: 4.84)
Ours (STD: 3.68)
Ours (STD: 8.81)
58
57
34
5
11
13
15
19
13
15 17
Task
Figure 8. Final accuracy across all GPM-based experiments.
19

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
20-Split CIFAR-100
10-Split CIFAR-100
25-Spiit TinylmageNet
87
86
Accuracy(S5)
Accuracy(P5)
Accuracy(%6) (%6)
64.
77-
76
54-
44-
66
NSCL (STD: 5.10)
NSCL (STD: 5.31)
NSCL (STD: 8.00)
Ours (STD: 4.70)
Ours (STD: 2.85)
Ours (STD: 7.48)
56
13
25
Task
Task
Task
Figure Final accuracy across all tasks in the NSCL-based experiments.
20-Split CIFAR-100
10-Split CIFAR-100
20-Split MinilmageNet
M
66
Accuracy(%5)
Accuracy(%% (yy%)
Accuracy(%))
56
58-
46
(51
62.60)
(ADA:
(ADA: 66.15)
59.57)
8
12
16
20
2
5
6
10
2
6
10
14
16
18
Task
Task
Task
20-Split CIFAR-100
10-Split CIFAR-100
20-5plit MinilmageNet
66
69-
68
56
Accuracy(9)
Accuracy1%)
Accuracy(%)
58
46
49
36
29
GPCNS (ADA: 56.57)
48
GPCNS (ADA: 61)
26
GPCNS (AOA: 44.48)
Ours (AOA: 56.65)
Ours (ADA: 61.23)
Ours (AOA: 45.09)
20
Task
Task
Task
20-Split CIFAR-100
10-Split CIFAR-100
20-Split MinilmageNet
70.
51
Accuracy
Accuracy(%)
57
Accuracy(%)
41
31
TRGP (ADA; 63.53)
TRGP (ADA: 57.48)
21.
TRGP (AOA: 38.27)
Ours (ACA: 66.32)
Ours (ADA: 58.44)
Ours (AOA: 41.08)
37
11
Task
Task
Task
20-Split CIFAR-100
10-Split CIFAR-100
20-Split MinilmageNett
74
Recurcyy
63
Accuracy/95
64
Accuracy(38)
67
53
43-
54
57
SGP (AOA: 53.38)
44
SGP (AOA: 57.54)
SGP (AOA: 65.
23
Ours (AOA: 55.53)
Ours (ADA: 60.74)
Ours (AOA: 67.29)
13
8
12
16
20
34
37
2
6
10
14
18
Task
Task
Task
Figure After one epoch accuracy GPM--based experiments.
20

---

BECAME: BayEsian Continual Learning with Adaptive Model MErging
20-Split
10-Split CIFAR-100
25-Split TinylmageNet
64
74
Accuracy
Accuracy(%6)
Accuracyr
58
54
34
NSCL (AOA: 60.24)
NSCL (AOA: 57.08)
NSCL (ADA: 75.77)
24
Ours (ADA: 67.81)
Ours (ADA: 63.70)
Ours (ADA: 52.53)
34
2
6
10
12
14
16
18
20
3
6
9
10
2
6
10
12 14
16 18
20
22 24
Task
Task
Task
Figure 11. After one epoch accuracy in NSCL-based experiments.
Cumulative Training LessfTlsk 1-2)
Cumulalive Triinnn Loss/Task 1-3)
Cumulative Training Lossi 1-4)
Cumulative Training Loss(Task 1-5)
0.8
1.4
0.6
2-066102
1.2
A-0.3419
A-0.3228
3.0
A-02769
0.5
8 1.0
0.4
0.8
1.5
2.0
0.6
0.2
0.0
0.2
0.4
0.6
0.8 2
1.0
0.0
6.2
0.4
0/6
0.8
100
0.0
0.2
0.4
0/6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1/0
Merge Coofficient for Task
Merge Confficient for Task
Merge Confiiiiet for Task
Merge Confficiont for Task 5
Figure Cumulative training loss different merge coefficient on 10-Split CIFAR-100 dataset in NSCL-based experiments. The dashed line denotes the merge coefficient calculated by our method.
0.52 Curulaative Treining Laskl"ask
ask 1-2)
Curulattvv Training LossiTask 1-3)
Cumn lative Training Lossi"ask 1-4)
Cumul Training Loss(Task 1-5)
1.00
2.4
0.95
1.8
0-5.399
1-0.3294
Am0.2310
2.2
A-0.1668
0.90
1.6
0.0
0.2
0.4
0.6
0.8
6.0
0.2
0.0
0.6
0.8
1.0
0.0
0.2
0..0
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Merge Coefficient for Task
Merge Coefficient for Task 3
Merge Coefficient for Task
Merge Coefficient for Task
Figure 13. Cumulative training loss versus different merge coefficient 10-Split CIFAR-100 dataset using GPM baseline.
21


Reviews:
[{"review_id": "cgHVhE0BXC", "reviewer": "Reviewer_Nkj6", "summary": "The paper proposes a method for continual learning based on updating the parameters for old tasks under an approach with limited plasticity, e.g. gradient projection methods, and then merging with parameters trained more freely for the new tasks. The paper proposes an approach for determining the merging coefficient in closed form based on the curvature around the minima. The experiments show improved performance over a range of recent gradient-projection based methods and widely known baselines from the literature on a vision classification benchmarks (split CIFAR100, Split TinyImagenet) with AlexNet and ResNet based architectures.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time for the review and respect your concern. However, we respectively argue our current experiments have sufficiently validated our contributions.\n## Experiment, W2\nWe believe our experimental design and results sufficiently support our claims for the following reasons:\n1. We **have already compared various merging methods within the BECAME framework in Table 3**, as discussed in Line 366. Among these methods, CoFiMA is a per-parameter merging method based on Fisher Merging. The results show that our method **outperforms alternative merging strategies in both ACC and BWT**. This empirically supports the superiority of our merging coefficient and the adaptability of model merging in gradient projection (GP) methods. If there are any other baselines you would like us to compare, we are happy to include them.\n2. The BECAME framework itself is a key contribution alongside the merging method, making its validation through experiments essential.\n3. One of our primary motivations is to enhance the adaptability of existing GP methods, and the results in Tables 1 and 2 show that BACAME effectively integrates with various GP methods, significantly improving their performance.\nTo further strengthen our study, we conduct experiments on applying model merging directly to CL without GP:\n\n||10-S CIFAR100||20-S MiniImageNet||\n|---|---|---|---|---|\n||ACC|BWT|ACC|BWT|\n|finetune|57.98|-20.19|57.06|-10.93|\n|1/t|60.69|-1.44|47.54|0.91|\n|CoMA|64.42|-9.79|60.99|-5.00|\n|CoFiMA|61.17|-0.38|62.26|-1.40|\n|Ours|**65.03**|-1.26|**64.75**|-2.11|\n\nThese results further highlight the effectiveness of our merging coefficient compared to other methods.\n## Claims&Theory&W3&Q2\nWe respectfully argue that our theoretical analysis using Laplace approximation is valid for the following reasons:\n1. Given the inherent complexity of deep learning optimization, which is influenced by numerous factors, it is impractical to analyze every possible case explicitly. Therefore, approximation techniques are **not only common but** **also** **essential for deriving meaningful theoretical** **insights****.** This is widely recognized in the field of machine learning.\n2. The use of Laplace approximation in neural networks has been **well-established since 1992** (A Practical Bayesian Framework for Backpropagation Networks, cited by 4264), and numerous studies have built upon this foundational framework. Recent works (Lee et al., 2017; Marouf et al., 2024; Kirkpatrick et al., 2017; Ritter et al., 2018) have successfully **applied Laplace approximation in** **CL**, further validating its applicability\n3. Model merging involves two key components: the model parameters {$\\theta\\_i$}$_n$ and their corresponding weights {$\\lambda_i$}$_n$. In our setting we have {$\\theta_i$}$\\_n$={$\\theta\\_{t−1}^*,\\theta_t$}, and our optimal merging coefficient is derived based on the merging trajectory between these two endpoints.\n4. We have provided additional empirical evidence through loss function visualization (Figs. 1 and 2) and comparative experiments (Table 3), further supporting the reliability of our theoretical analysis.\n## Relation&W1&Q1\n1. Both Fisher Merging and our method are based on Laplace approximation and use Fisher for calculation, yet they differ significantly in many aspects. Below is a detailed comparison:\n\n||Theoretical Basis|Usage of Fisher|Setting|Merging Coefficient $\\lambda$|Number of Merges|Number of Models Merged|Relationship Between Models|Training Dataset|\n|---|---|---|---|---|---|---|---|---|\n|FisherMerging|Laplace Approx.|Weight importance of parameters|Ensemble, Finetune|Hyperparameter|1|n|Trained from a same initialization|Same|\n|Ours|Laplace Approx.|Calculate $\\lambda$|CL|Adaptive|T(tasks count)-1|2|$\\theta_{i+1}$ trained from $\\theta_i$|Different for each task|\n\nOur method fills a crucial gap in existing merging approaches for CL by considering the influence of prior tasks on new task learning.\n2. Using a scalar $\\lambda$ rather than a vector is a deliberate choice for theoretical analysis. Ablation study and Table 3 also indicate that **per-parameter merging does not necessarily yield better results**. **Please refer to Q2** **of the** **response to reviewer H5hN for more details.**\n\n\nWe hope these detailed responses and additional experiments address your concerns.", "claims_and_evidence": "Claims closed form optimal solution for merging coefficient, but this is not accurate and quite misleading to readers who do not go through the full paper. The derivation is based on an approximation of the objective, hence the closed form expression is not guaranteed to be optimal for the true objective. This needs to be clearly acknowledged throughout the paper.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Not in detail.", "experimental_designs_or_analyses": "The baselines are problematic. The experiments mainly demonstrate that the proposed merging approach can be combined with different gradient projection methods. However, the core technical contribution is not compared to alternative approaches for merging and this comparison is essential to evaluating a core technical contribution of the paper (the merging approach).", "relation_to_broader_scientific_literature": "The paper explains its motivations clearly in improving on previous gradient projection methods for continual learning. Merging methods are briefly discussed in the related work section. The method seems very closely related to Fisher merging, with the difference being that the present work uses a scalar rather than a per-parameter merging coefficient. Unfortunately, this relationship is not discussed. In particular, it is not clear to me why we would use a single coefficient when we are using a facotrized posterior approximation rather than a per-parameter one.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n* The paper is clear\n* The approach is well-motivated and the overall pipeline makes sense\n* The merging approach improves results for a range of gradient projection based continual learning methods\n\nWeaknesses:\n* It is not clear to me that the proposes merging technique is not just Fisher merging with a single coefficient.\n* The merging approach is not compared to alternatives, which I find problematic given that it is the core technical contribution of the paper. It is neat that it works well for a range of gradient projection methods, however I find this less relevant than substantiating the claim that the merging is optimal. In particular, Fisher merging (assuming I am correct in suspecting the close relationship) tends to be a relatively weak baseline, at least in the LLM merging literature. So I would suspect that there likely is a better-performing alternative off the shelf.\n* The paper consistently claims the theoretical optimality of its merging coefficient, which appears to be incorrect given that the closed form solution is based on an approximation of the objective.", "other_comments_or_suggestions": "I hope the authors do not take the score as an overly harsh judgement of their work. It is unfortunately the only \"reject\" option that isn't borderline. The overall merging pipeline seems valuable, however I think the theoretical sections (3.3 in particular) and experiments (other merging baselines) will have to be reworked significantly and render this work better suited for resubmission rather than revision.", "questions_for_authors": "* Could you comment on the relationship between your merging technique and Fisher merging?\n* If you believe that your merging coefficient is actually optimal, could you explain why? I appreciate the preceding discussion in 3.2 about the absence of loss barriers. But (a) this seems to be a conjecture rather than theoretically guaranteed and (b) even with an absence of barriers, the approximation of the objective seems to break any guarantees to me.\n\nI just want to state explicitly in advance that I may not end up raising my score if these questions are answered, as my core concern is the lack of baselines for the proposed merging method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kIeHc5mEfR", "reviewer": "Reviewer_H5hN", "summary": "This paper introduces a novel framework called BECAME to address a crucial problem in continual learning, i.e., retaining prior knowledge while learning new tasks to achieve stability and plasticity. From the perspective of Bayes continual learning, BECAME develops a novel merging mechanism to bridge the gap between prior work and the complexities of task interdependence, providing a theoretical demonstration of the stability-plasticity trade-off. Specifically, the optimal merging coefficient for two successive models can be derived via a closed-form solution. Extensive experiments demonstrate the superior performance of BECAME, suggesting its effectiveness in finding an optimal merging model that maximizes overall performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review and the insightful comments on our theoretical framework and experimental validations. Below, we detail our clarifications and additional proof.\n# Theory\n1. We agree that the assumption of independence across tasks is necessary for Eq. 6 and will fix it, while Eq. 1 is derived by definition and imposes no such constraint.\n2. Our proof of Lemma 3.1 utilizes fixed endpoints to show the existence of a merged model with a lower cumulative loss. Importantly, this proof **does not assume that the** **loss function** **is globally convex** with respect to the model parameters. \n# W1\nWe conduct additional ablation studies in response to your advice. And the results indicate our method **consistently outperforms alternative merging strategies**. \n\n**Please refer to the Experiment part of the response to reviewer Nkj6.** We will include them in appendix to further support our method.\n\n# C1\n\nWe promise to include a detailed discussion on model merging in appendix in our revised vision, as well as classical literature on Gaussian mixture models and multivariate mixtures.\n\n# Q1\nThank you for the suggestion, here we supplement a theoretical analysis on why $\\Delta\\theta$ in Eq. 17 can be substituted by $\\hat{\\theta}_t-\\theta_t^{GP}$.\n\nThe key point is to prove that Eq. 10 holds when $\\theta_{t-1}^*$ is replaced by $\\theta_t^{GP}$.\n\nGiven $\\Lambda_{t-1}$​ is symmetric and semi-positive definite, as it is the Hessian of the negative log posterior, we have\n$$\n(\\theta-\\theta_t^{GP})^{\\top}\\Lambda_{t-1}(\\theta-\\theta_t^{GP})=(\\theta-\\theta_{t-1}^*)^{\\top} \\Lambda_{t-1}(\\theta-\\theta_{t-1}^*)+2(\\theta-\\theta_{t-1}^*)^{\\top}\\Lambda_{t-1}(\\theta_t^{GP}-\\theta_{t-1}^*)+(\\theta_t^{GP}-\\theta_{t-1}^*)^{\\top}\\Lambda_{t-1}(\\theta_t^{GP}-\\theta_{t-1}^*).\n$$\nThen we only need to prove $\\Lambda_{t-1}(\\theta_t^{GP}-\\theta_{t-1}^*)=0 \\ (1)$. We define $d=\\theta_t^{GP}-\\theta_{t-1}^*$.\n\n$\\Lambda_{t-1}$ can be decomposed as $\\Lambda_{t-1}=Q^{\\top}AQ$, as $A$ is a diagonal matrix of the eigenvalues. \n\nTo prove (1), we do a second-order Taylor expansion of $L_{1: t-1}(\\theta_t^{GP})$ around $\\theta_{t-1}^*$, yielding: $d\\Lambda_{t-1}d=dQ^{\\top}AQd=0$, since $L_{1: t-1}(\\theta_t^{GP})\\approx L_{1: t-1}(\\theta_{1-t}^*)$ and $\\theta_{t-1}^*$ is an optimum.\n\nGiven that $A_{ii}\\geq 0$, it follows each element of $Q$ must be 0, leading to $\\Lambda_{t-1}d=Q^{\\top}AQd=0$​​​​.\n\nDue to the character limit, we will provide a detailed proof in our revised version.\n\n# Q2\n1. Our theory does not require Eq. 10 to be convex. We only claim that when merging model from$θ_{t−1}^*$ to $\\hatθ_t$, the total loss in Eq.11 **is convex with respect to** $\\lambda$ (Line 236).\n2. We treat $\\hatθ_t$ as a scalar for the following reasons:\n- When $\\lambda$ is scalar, our analysis allows us to readily show that $\\tilde L_{1:t}(\\lambda)$ is convex .\n- From our proof in Q1, when $\\lambda$ is a vector, $\\Lambda_{t-1}\\lambda(\\theta_t^{GP}-\\theta_{t-1}^*)$ may not be 0, preventing the replacement of $\\theta_{t-1}^*$ with $\\theta_t^{GP}$ in our BECAME framework.\n- There exists a linear connector from $\\theta_A$ to $\\theta_B$ when they are trained sequentially [1], as also indicated in the right column of Line 146.\n- Our ablation studies and Table 3 have demonstrated that per-parameter merging does not necessarily yield better results.\n3. We also provide additional experiments and analysis to validate that optimizing Eq. 10 via model merging is more efficient than regularization.\n\n| |10-S CIFAR ACC (%)|BWT (%)|Train Time (s)|20-S MiniImageNet ACC (%)|BWT (%)|Train Time (s)|\n|---|---|---|---|----|----|---|\n|Regularization|59.76|-19.58|218.9|58.90|-12.41|612.5|\n|Limited Reg|62.40|-14.28|341.98|58.24|-11.49|931.1|\n|Ours (Merge)|65.03|-1.26|108.7|64.71|-2.61|232.7|\n|GPM+Regularization|71.99|-7.72|362.5|64.69|-4.19|781.2|\n|GPM+Limited Reg|72.35|-5.34|527.2|64.94|-3.69|1083.1|\n|**GPM+Ours**|**75.05**|**0.02**|274.2|**67.62**|**0.87**|584.14|\n\nNow we present the analysis to further elaborate:\n- Adding regularization to the loss function slows down training. The additional time grows as the parameter dimension increases, whereas model merging without regularization leads to faster training.\n- Regularization is added to prevent the loss of old tasks from increasing. For a specific $\\theta_i$ in $\\theta$, optimization for the new task follows the loss based on the network architecture, while for old tasks it is constrained by the second-order norms. This explains why the results of regularization differ from those of model merging, even with constrained direction. **Our approach naturally avoids this conflict** and consistently achieves the best results.\n- Due to the aforementioned imbalance, weight of the regularization term should be carefully tuned to get optimal performance, while our model merging **does not require such manual tuning**.\n\n[1] Linear mode connectivity and the lottery ticket hypothesis. ICML 2020", "claims_and_evidence": "The claims in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method makes sense for the problem of continual learning.", "theoretical_claims": "Yes, I have checked the correctness of the proofs and theoretical claims in the main body, and I have several concerns:\n\n1. The problem definition missed some important hypotheses. For instance, the data are supposed to be iid; otherwise, Eqs. 1 and 6 can not be established.\n2. The proof of Lemma 3.1 seems to default to the fact that the loss function is convex (line 182), but it has not been stressed out in the preliminary. However, under this circumstance, Lemma 3.1 seems to be a little trivial, as the summation of two convex function ($L_{1:t-1}$, $L_{t}$) must be convex, such that the new minimum is between $\\theta^*_{t-1}$ and $\\hat{\\theta}_t$, Eq. 2 can then be easily obtained by the nature of convex function.", "experimental_designs_or_analyses": "The experiments, conducted on several widely used benchmarks, exhibit outstanding performance. The experimental designs and analyses are sound and make sense.", "relation_to_broader_scientific_literature": "This paper addresses the problem of continual learning from the perspective of Bayes learning. Although the fundamental objective problem is based on prior literature, it presents a novel approach to merging models and achieves stability and plasticity.. Specifically, the linear combination weights can be obtained through a closed-form solution. I believe this research can inspire Bayes continual learning and model merging.", "essential_references_not_discussed": "This paper has fully discussed the related work, and there is no more essential literature to my knowledge.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The method provides theoretical insights into model merging to achieve optimal performance in continual learning. The optimization objective has been proven to be convex and the closed-form solution can be efficiently obtained.\n1. The paper is well written and easy to follow; extensive experiments have validated the effectiveness of the merging strategy.\n\nWeakness:\n\n1. In the first stage of BECAME, the method incorporates the GP method to enhance stability in the previous task. However, the MAP parameter should be $\\theta_{t-1}^*$ in section 3.3. To fully validate the effectiveness of the merging strategy, it is important to conduct ablation studies starting from the previously obtained parameter $\\theta_{t-1}^*$, as well as to compare it with other model merging strategies.", "other_comments_or_suggestions": "This paper is well written and I didn't notice any typos. My advice is to provide a more detailed discussion of the related work about model merging in the appendix like gradient projection. Moreover, more classic methods about Gaussian mixture model and multivariate mixtures could be cited.", "questions_for_authors": "1. It is a pity that the article does not fully incorporate GP into the theoretical framework. However, the authors claim that $\\theta_{t-1}^*$ can be substituted with $\\theta_t^{GP}$, this process seems more like an experimental conclusion. Since the MAP parameter is supposed to be $\\theta_{t-1}^*$ rather than $\\theta_t^{GP}$ in the derivation, I would prefer a general framework that theoretically analyzes the result in Line 267.\n2. This paper first follows the Streaming Bayes theorem to derive the MAP problem in Eq. 7, then utilizes Laplace approximation to deal with the previous posterior $p(\\theta|D_{1:t-1})$. The obtained formulation in Eq. 10 seems to be equivalent to former work. Even if we treat Eq. 10 as a convex problem, the optimal result does not necessarily lie on the line between $\\theta_{t-1}^*$ and $\\hat\\theta_t$, What is the advantage of linear merging rather than jointly minimizing Eq. 10 by treating the latter part as a regularization term? If it does, can we regard Eq. 11 as a sub-problem of Eq. 10 with the constraint that $\\theta$ is a linear combination of $\\theta_{t-1}^*$ and $\\hat\\theta_t$? Does the closed-form solution avoid any explicit parameter calculations compared with other regularization-based methods, given that $\\hat\\theta_t$ still needs to be optimized? What if the updating direction of regularization-based method is restricted to lie between $\\theta_{t-1}^*$ and $\\hat{\\theta}_t$, can we still obtain a similar optimal result? Why or why not?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QXgc2l6dMF", "reviewer": "Reviewer_mavm", "summary": "The paper presents BECAME, a Bayesian Continual Learning framework designed to address the stability-plasticity dilemma in continual learning. The method combines gradient projection methods with model merging to balance retaining prior knowledge (stability) and learning new tasks (plasticity). The key contribution is deriving a closed-form solution for optimal merging coefficients using Bayesian principles. Besides, the proposed method is simple and compatible with various gradient projection CL methods. The proposed method is validated by extensive experiments on multiple benchmarks. It achieves superior performance.\n\n## update after rebuttal\nThe authors have addressed my concerns regarding BWT.  However, the AAA results are often only comparable to or worse than Acc. These results contradict the trends in existing works (AAA typically exceeds Acc by a significant margin). This is a remaining minor concern. \n\nOverall, I appreciate that the proposed method is simple yet supported by theoretical inference of the optimal combination coefficient. Therefore, I am inclined towards the acceptance of the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful review and the constructive suggestions. Your recognition of this work is truly encouraging. Here we present our detailed clarifications and updates, which we believe further enhance our paper.\n## Evaluation&C1\nIn response to your suggestion, we compute Averaged Anytime Accuracy (AAA) using the formula: $AAA = \\sum_{i=1}^T AA_i$ with $AA_i = \\frac{1}{i} \\sum_{j=1}^{i} A_{i,j}$. The results are shown in the tables below:\n\n|            | 20-S CIFAR |           | 10-S CIFAR |           | 20-S MiniImageNet |           |\n| ---------- | ---------- | --------- | ---------- | --------- | ----------------- | --------- |\n|            | AAA        | ACC       | AAA        | ACC       | AAA               | ACC       |\n| GPM        | 77.31      | 77.34     | 72.08      | 71.81     | 63.02             | 63.90     |\n| GPM+Ours   | 79.61      | 80.57     | 74.22      | 75.05     | **66.05**         | 67.62     |\n| TRGP       | 79.96      | 81.68     | 74.24      | 75.01     | 62.55             | 62.68     |\n| TRGP+Ours  | **80.78**  | **82.61** | **74.89**  | 75.87     | 64.48             | 65.09     |\n| SGP        | 78.78      | 80.21     | 73.62      | 74.97     | 61.97             | 66.99     |\n| SGP+Ours   | 79.44      | 81.94     | 74.82      | **76.74** | 64.98             | **70.06** |\n| GPCNS      | 76.22      | 78.63     | 71.50      | 71.84     | 60.53             | 62.85     |\n| GPCNS+Ours | 78.08      | 80.87     | 72.94      | 73.89     | 61.53             | 64.79     |\n\n|           | 20-S CIFAR |           | 10-S CIFAR |           | 25-S TinyImageNet |           |\n| --------- | ---------- | --------- | ---------- | --------- | ----------------- | --------- |\n|           | AAA        | ACC       | AAA        | ACC       | AAA               | ACC       |\n| Adam      | 75.43      | 75.66     | 74.77      | 72.91     | 60.21             | 58.77     |\n| Adam+Ours | **81.25**  | **81.88** | **81.94**  | **81.66** | **66.65**         | **66.49** |\n\nThese results confirm that our method continues to **yield substantial improvements when evaluated with AAA**, with trends closely aligning with ACC. Due to space limit, we will include these results in the appendix.\n\n## Theory&C2\nThank you for your suggestion. We will revise the statement at lines 181–182 to clarify that the log prior remains constant only under a uniform prior.\nAdditionally, we have verified that the parameter prior is predefined and that this adjustment does not affect subsequent derivations or results.\n## W1\nThe observed degradation in the Backward Transfer (BWT) metric in some cases reflects a deliberate trade-off favoring enhanced plasticity and overall performance.\n\n- **Sensitivity of BWT.** BWT of task t is determined by its immediate post-training performance ($A_{t,t}$) and its final performance after learning all tasks ($A_{T,t}$). In some instances, a higher $A_{t,t}$ may lead to a lower BWT even when overall ACC improves.\n- **Stability-Plasticity trade-off.** Our approach intentionally prioritizes increased plasticity to boost performance on new tasks, which may slightly reduce stability (as measured by BWT). However, the overall accuracy benefits from this trade-off.\n\nWe emphasize that our method does not compromise the performance of earlier tasks beyond an unacceptable level; rather, it strategically **balances the stability-plasticity trade-off** to achieve better overall performance.\n## Q1\nYes, our method is **equally applicable to class-incremental learning** (CIL). The theoretical framework and model merging mechanism are independent of task- or class-incremental learning. Since our approach operates directly on model parameters, it is effective in both scenarios.\nTo validate this, we conducted a simple experiment by testing without task IDs, following the standard setup for CIL.\n\n|          | 20-S CIFAR | 10-S CIFAR | 20-S MiniImageNet |\n| -------- | ---------- | ---------- | ----------------- |\n| GPM      | 24.73      | 30.73      | 20.98             |\n| GPM+Ours | **27.12**  | **37.04**  | **27.04**         |\n\n|           | 20-S CIFAR | 10-S CIFAR | 25-S TinyImageNet |\n| --------- | ---------- | ---------- | ----------------- |\n| Adam      | 12.29      | 18.09      | 6.03              |\n| Adam+Ours | **17.84**  | **31.79**  | **13.37**         |\n\nThe results indicate that our method also **improves accuracy in the CIL setting**. The improvement may be attributed to the enhanced balance of performance across tasks achieved through merging (see Line 408). We are delighted to investigate this aspect more in future work.\n\n\nWe hope these detailed responses clarify our revisions and validate the robustness of our approach. We sincerely appreciate your valuable feedback.", "claims_and_evidence": "Yes. This is one of its strengths. For example, it supports the motivation of adaptive model merging using illustration and empirical evidence as shown in Figures 1 and 2.", "methods_and_evaluation_criteria": "The method is simple but makes sense. It is supported by empirical evidence and theoretical reasoning that adaptive model merging achieves better performance. \n\nThe benchmarks and metrics are sufficient to evaluate the proposed methods. But this paper does not report the Averaged Anytime Accuracy (AAA) performance, which is a widely used metric for studying continual learning.", "theoretical_claims": "I have checked the theory part. Most of them are correct. A minor issue might be the statement that \"the log prior log p(θ) is not related to the optimization as it is a constant under a certain initialization\" in the right column lines 181-182. The statement holds in the specific case of a uniform prior, instead of under a certain initialization. It does not apply to MAP estimation in general.", "experimental_designs_or_analyses": "The experimental designs are good. It sufficiently supports the effectiveness of the proposed method. A potential improvement is to use AAA as a metric for evaluating CL methods.", "relation_to_broader_scientific_literature": "This paper proposed a novel CL method, which is simple yet effective. It can be applied to different applications.", "essential_references_not_discussed": "Essential references are discussed.", "other_strengths_and_weaknesses": "Strengths:\n1. The proposed method is simple but novel. And its effectiveness is demonstrated through theory reasoning and empirical results.\n2. The paper is well-written and easy to follow. It provides a clear illustration of motivation as in Figure 1 and Figure 2.\n3. Extensive experimental results to validate its effectiveness.\n\nWeakness:\n1. Combining with the proposed method may produce worse BWT in some cases, but not in most.", "other_comments_or_suggestions": "1. Add AAA as a metric to evaluate CL methods.\n2. Fix the minor issue \"the log prior log p(θ) is not related to the optimization as it is a constant under a certain initialization\" in the right column lines 181-182. The statement holds in the specific case of a uniform prior, instead of under a certain initialization. It does not apply to MAP estimation in general.", "questions_for_authors": "1. Can the proposed method be extended to class incremental settings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "umPNpqhsBR", "reviewer": "Reviewer_EVng", "summary": "The paper proposes BECAME, a Bayesian continual learning framework that adaptively merges task-specific models to balance stability and plasticity. Key contributions include:\n\n* A closed-form solution for merging coefficients derived via Bayesian principles, proving that merging models along a linear path can achieve a lower cumulative loss than individual task-optimized models.\n\n* A two-stage training paradigm combining gradient projection (for stability) and unconstrained optimization (for plasticity), followed by adaptive model merging.\n\n* Some theoretical analysis is provided. \n\n* Extensive experiments on CL benchmarks demonstrate state-of-the-art performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and highlighting these points. Below, we address each concern in detail.\n## Reference\nWe acknowledge the significance of AdaMerging[1] in model merging and promise to cite it to recognize its contribution. Additionally, we will expand the appendix to **provide an comprehensive discussion on model merging**.\n\nAfter a careful review of the AdaMerging paper, we identify several key distinctions between their approach and ours. While both methods address adaptive model merging, AdaMerging operates within a multi-task learning framework where tasks are learned **concurrently**. In contrast, our work focuses on continual learning, where tasks are learned **sequentially**. Our merging strategy is specifically designed to consider the temporal dependencies between tasks.\n\nMoreover, AdaMerging iteratively optimizes the merging coefficient at test time using entropy minimization on unlabeled data. In our approach, the coefficient is determined during training by a closed-form solution.\n## W1\nWhile our two-stage training approach introduces additional computational costs, our results show that the overall training time **remains well below double** **that of baselines**, and the performance improvements justify the extra overhead. For instance, as shown in Table 4, GPM attains 63.90% ACC in 434.66s, whereas our method achieves 67.62% in 584.14s. The second stage adds **less than 35%** to the duration of the first stage, making our approach highly competitive compared to single-stage methods like TRGP, which requires considerably more training time (776.72s) due to complex operations.\n\nThe reasons for this efficiency are as follows:\n- The unconstrained training in second stage incurs a lower per-epoch cost (0.52s) than gradient projection (0.85s) and typically converges in fewer epochs when early stopping is applied.\n- In contrast to the gradient projection, which spends additional time computing the subspace for each task, the second stage avoids this overhead entirely.\n\n## W2\nWe would like to highlight that **our method does not significantly increase storage requirements.**\n\nBelow, we first present the experimental results and then analyze why the BECAME framework is memory-efficient.\n\nTo provide a detailed comparison, we divide the total memory into three parts: model, the additional memory for variables needed for training future tasks and the temporary memory required only during training. We further divide the additional memory based on training stage for analysis.\nThe results are as follows:\n\n|          | ACC (%) | Total Memory (MB) | Model Parameters (MB) | Additional Memory in 1st Stage (MB) | Additional Memory in 2nd Stage (MB) | Temporary Memory (MB) |\n| -------- | ------- | ----------------- | --------------------- | ----------------------------------- | ----------------------------------- | --------------------- |\n| GPM      | 63.90   | 347.26            | 4.74                  | 37.51                               | –                                   | 305.01                |\n| GPCNS    | 62.85   | 998.41            | 4.74                  | 584.26                              | –                                   | 409.41                |\n| SGP      | 66.99   | 347.26            | 4.74                  | 53.47                               | –                                   | 289.05                |\n| TRGP     | 62.68   | 1854.32           | 73.23                 | 1359.18                             | –                                   | 421.91                |\n| GPM+Ours | 67.62   | 375.72            | 4.74                  | 55.88                               | **4.73**                            | 310.37                |\n| SGP+Ours | 70.69   | 375.72            | 4.74                  | 60.49                               | **4.73**                            | 305.76                |\n- **The second training stage introduces only a modest storage overhead.** As shown in the 5th and 6th rows of the table, the only additional memory for the second training stage is the precision matrix, which is comparable in size to the model parameters and determined solely by the model architecture rather than by the baseline method.\n- The increment of the additional memory in the first training stage is a direct result of improved overall performance, which follows the inherent feature of the gradient projection methods. After performing our model merging, the model's capability of feature extraction is enhanced so that the dimension of output feature space expand and more base vectors need to be stored for future task training.\n- The memory overhead of our method is competitive to GPCNS and TRGP, which required more storage than our method even they only have one training stage.\n\n\nWe hope these detailed responses address your concerns and further enhance the robustness of our approach.", "claims_and_evidence": "Yes, the main claims are supported by evidence.", "methods_and_evaluation_criteria": "Yes, this paper evaluates different CL approaches with overall accuracy and backward transfer, which are commonly used evaluation metrics.", "theoretical_claims": "To the best of my knowledge, the proof looks correct.", "experimental_designs_or_analyses": "The experimental designs and analysis are sound and valid.", "relation_to_broader_scientific_literature": "This paper improves continual learning through adaptive model merging. The model merging have been discussed in the related works.", "essential_references_not_discussed": "This submission discussed model merging, but one closely related paper about adaptive model merging [1] has not been discussed. \n\nReference:\n\n[1] AdaMerging: Adaptive Model Merging for Multi-Task Learning, ICLR 2024.", "other_strengths_and_weaknesses": "**Strength**:\n* The paper introduces an approach to continual learning (CL) by leveraging Bayesian principles to derive a closed-form solution for adaptive model merging. \n\n* The theoretical analysis and the derivation of the optimal merging coefficient are reasonable. The authors provide a clear mathematical framework to explain why merging models along a linear path can lead to better minima for cumulative loss across tasks.\n\n\n\n**Weakness**:\n* **Computational Overhead**: While merging itself is efficient, the two-stage training (gradient projection + unconstrained training) doubles training time compared to single-stage baselines.\n\n* **Memory Overhead**: The author only compares the GPU memory consumption. I think this comparison is not fair, it would be better to compare the memory cost of storing additional set of model parameters and others.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a framework designed to tackle the stability-plasticity trade-off in continual learning. This work uses a Bayesian approach to adaptive model merging, which aims to improve performance in continual learning by providing a closed-form solution for the merging coefficients. The paper's main strengths lie in its theoretical contributions, demonstrated through extensive empirical results on common benchmarks like CIFAR100 and MiniImageNet, showcasing improvements over existing methods. Additionally, the framework's compatibility with different gradient projection methods further amplifies its adaptability and potential for broader applicability.

The reviewers had varied scores initially: 2, 3, 4, 1. Much of the concern centred around the lack of clarity in theoretical claims and insufficient comparison with alternative merging methods. The authors provided clarifications and additional experiments during the rebuttal process, addressing these issues. Reviewer Nkj6 adjusted their score (1-2) after recognising that the initial evaluation overlooked the detailed discussion in Table 3, which contrasted various merging strategies. Despite a lot of prompting, there wasn't a whole lot of discussion, so it's hard to know if the initial concerns were fully covered.

The main weakness in the original submission was the lack of precision in theoretical claims about the optimality of the merging coefficients, which the authors acknowledged and promised to amend. There were also concerns regarding computational and memory overheads, but  the authors effectively demonstrated that their method remains competitive in terms of performance enhancements relative to this additional cost. 

This approach is softer than trying to do fully Bayesian continual learning, which has been attempted many times and is problematic for various reasons. Here the Bayesian formulation is used only for merging models, and can be used with any gradient projection method. Considering the reasonable resolution of concerns and the contribution to Bayesian continual learning literature, I recommend accepting this paper to ICML.
Decision: Accept (poster)