Make Continual Learning Stronger via C-Flat
Authors: Ang Bian
Wei Li
Hangjie Yuan
Chengrong Yu
Mang Wang
Zixiang Zhao
Aojun Lu
Pengliang Ji
Tao Feng
Venue: NeurIPS 2024 poster (2024)
Keywords: Continual Learning
Incremental Learning
Primary area: optimization_for_deep_networks
TL;DR: This paper propose a Continual Flatness (C-Flat) method featuring a flatter loss landscape tailored for CL. C-Flat could be easily called with only one line of code and is plug-and-play to any CL methods.

Abstract: How to balance the learning ’sensitivity-stability’ upon new task training and memory preserving is critical in CL to resolve catastrophic forgetting. Improving model generalization ability within each learning phase is one solution to help CL learning overcome the gap in the joint knowledge space. Zeroth-order loss landscape sharpness-aware minimization is a strong training regime improving model generalization in transfer learning compared with optimizer like SGD. It has also been introduced into CL to improve memory representation or learning efficiency. However, zeroth-order sharpness alone could favors sharper over flatter minima in certain scenarios, leading to a rather sensitive minima rather than a global optima. To further enhance learning stability, we propose a Continual Flatness (C-Flat) method featuring a flatter loss landscape tailored for CL. C-Flat could be easily called with only one line of code and is plug-and-play to any CL methods. A general framework of C-Flat applied to all CL categories and a thorough comparison with loss minima optimizer and flat minima based CL approaches is presented in this paper, showing that our method can boost CL performance in almost all cases. Code is available at https://github.com/WanNaa/C-Flat.

Make Continual Learning Stronger via C-Flat
Ang Bian" Wei Li 1. Hangjie Yuan 3,4 Chengrong Yu', Mang Wang" Zixiang Zhao", Aojun Lu², Pengliang Ji Tao Feng??
Sichhan University TTinghua University 3DAO Academy, Alibaba Group
'Zhejiang University 'ByteDance ai''an Jiaotong University "Carnegie Mellon University hj edu.. cn, (ymji1198, fengtao hi) (a1@@gmmil.cm com
Abstract
How to balance the learning 'sensitivity-stabiiii upon task training and memory preserving is critical in CL to resolve catastrophic forgetting. Improving model generalization ability within each learning phase is solution to help CL learning overcome the gap in the joint knowledge space. Zeroth-order loss landscape sharpness-aware minimization is strong training regime improving model generalization in transfer learning compared with optimizer like SGD. It has also been introduced into CL improve memory representation or learning efficiency. However, zeroth-order sharpness alone could favors sharper over flatter minima in certain scenarios, leading a rather sensitive minima rather than a global optima. To further enhance learning stability, we propose Continual Flatness (C-Flat) method featuring a flatter loss landscape tailored for CL. C-Flat could be easily called with only one line of code and is plug-and- and-play to any CL methods. general framework of C-Flat applied all CL categories and a thorough comparison with loss minima optimizer and flat minima based CL. approaches is presented in this paper, showing that our method can boost CL performance almost all cases. Code is available at /ttttt:://ggttiie com/ Saam//aNNaa//-CHII Flat
C-Flat: just line of code suffices for its utilization.
from import C_Flat
2
# initialize optimizer for CL.
4 C_Flat_optimizzr = C_Flat paramss base_optimiizer model, args)
Introduction
Why study Continual Learning (CL)? CL generally acknowledged as a necessary attribute for Artificial General Intelligence (AGI) 22 55 40 67]. In the open world, CL holds the potential for substantial benefits across many applications: egg. vision model needs learn a growing image
set [I 61 62], or, embodied model needs to incrementally add skills to their repertoire [[2]
Challenges. A good CL model is expected to keep the memory of all seen tasks upon learning new knowledge (22). However, due to the limited access to previous data, the learning phase is naturally sensitive the current task, hence resulting in major challenge in CL. called catastrophic forgetting (9), which refers to the drastic performance drop on past knowledge after learning new knowledge. This learning sensitivity-stabiii dilemma is critical in CL, requiring model with strong generalization ability [16] to overcome the knowledge gaps between sequentially arriving tasks.
'Equal Contribution Corresponding Authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Current solutions. A series of works [43 44 33 25 are proposed to improve learning stability by extending data space with dedicated selected and stored exemplars from old tasks, or frozen some network blocks or layers that are strongly related to previous knowledge 668
24
69
57 24
Another group of works seeks to preserve model generalization with regularisation onto the training procedure itself [32 18 311. Diverse weight [45] 28 2 or gradient alignment [22]
9 35 26 strategies
are designed to encourage the training to efficiently extracting features for the current data space without forgetting.
efficient training regime for model generalization starts to gain attentions [27 63] Ordinary loss minima based optimizer like SGD To prevent this. zeroth-order sharpness- minimization seeking neighborhood-fian minima (20) has been proven a strong optimizer to improve model generalization ability, especially in transferring learning tasks. It is also introduced into with dedicated designs to improve old knowledge representation or few- shot learning efficiency. However, given the limited application scenarios [10] 49), the zeroth-order sharpness used in the current work is proved to favor sharper minima than flat solution [70]] It means zeroth-order only can still lead to a fast gradient descent to the suboptimal in new data space than more generalizable result for the joint old and new knowledge space.
Loss landscape sharpness optimization 23
19
65
70
can easily lead to suboptimal results 4 37
I3
some CL works [49 30
Our solution. Inspired by these works, beyond zeroth-order sharpness continual optimization method is proposed as demonstrated in I where loss landscape flatness is emphasized to strengthen model generalization ability. Thus, the model can always converge to a flat minima in each phase, and then smoothly migrate to the global optimal of the joint knowledge space of the current and next tasks, and hence resolve the catastrophic forgetting in CL. We dub this method Continual Flatness (C-Flat or C5) Moreover, C-Flat is a general method that can be easily plug-and-play into any CL approach with only one line of code, improve CL.
Contribution. A simple and flexible CL-friendly optimization method C-Flat is proposed, which Makes Continual Learning Stronger.
framework of C-Flat covering diverse CL method categories is demonstrated. Experiment results prove that Flatter is Better in nearly all cases.
To the best of our knowledge, this work is the first conduct thorough comparison of CL approaches with loss landscape aware optimization, and thus can serve as a baseline in CL.
2 Related work
Continual learning methods roughly are categorized into three groups: Memory-based methods write experience in memory to alleviate forgetting. Some work [43
44 25 51] design different sampling strategies to establish limited budgets in a memory buffer for rehearsal. However, these methods require access to raw past data, which is discouraged in practice due to privacy concerns.
Instead, recently series of works [I0 46 33 50 elaborately construct special subspace of old
methods aim to realize consolidation of the previous knowledge by introducing additional regularization terms the loss function. Some works [32 29
tasks as the memory. Regularization-
enforce the important weights in the parameter space 45 28 2], feature representations 21 or
the logits outputs [32 42
the current model function to be close to that of the old one. Expansion- bused methods dedicate different incremental model structures towards each task to minimize
forgetting [68 33). Some work [48. 24 60 exploit modular network architectures (dynamically extending extra components [57 69 or freeze partial parameters [36] I to overcome forgetting. Trivially, methods in this category implicitly shift the burden of storing numerous raw data into the retention of model [68].
Gradient-based solutions are main group in CL, including shaping loss landscape, tempering the wf of gradient, and other learning dynamics [2 9 One promising solution is to modify the gradients of different tasks and hence overcome forgetting 7 38 e.g., aligning the gradients of current and old one [15] 18 or, learning more efficient in the case of conflicting
objectives 47 56 14]. Other solutions 100 49 focus on characterizing the generalization from the
loss landscape perspectives to improve CL performance and yet rarely explored.
2

---

Loss
Losy
Los
loss of old tasks
Ieleratre line
potturh ridus
Fusd
wel leara
Loid
ondnary performance
colastogiii forgsting
Direct Tuning
(b) Regularization
(c) C-Flat
Figure 1: Illustration of C-Flat overcoming catastrophe forgetting by fine-tuning the old model
parameter to flat minima of new task. loss minima for current task only can cause catastrophe forgetting on previous ones. b) balanced optima aligned by regularization leads to unsatisfying results for both old and new tasks. c) C-Flat seeks global optima for all tasks with flattened loss landscape.
are proposed to optimize neural networks in standard training scenarios towards flat minima. Wide local minima were considered an important regularization in CL enforce the similarity of important parameters learned from past tasks [6]. Sharpness- aware seeking for loss landscape flat minima starts to gain more attention in CL, especially SAM based zeroth order sharpness is well discussed. An investigation [41 proves SAM can help with addressing forgetting in CI., and g proposed combined SAM for few-shot CL. SAM also used for boosting the performance of specific methods like DFGP 58 and FS-DGPM [TO] designed for GPM. SAM-CL [52) series with loss term gradient alignment for memory-based CL.
Sharpness minimization in CL Many recent works 23
19
These efforts kicked off the study flat minima in CL, however, zeroth-order sharpness may be
enough for flatter optimal 70 Thus, flatness with a global optima and universal CL framework is
further studied.
Method
Our solution addresses the learning sensitivity-stabiin dilemma in CL by improving model general- ization for joint learning knowledge obtained from different catalogues domains or tasks. Moreover, general but stronger optimization method enhanced by the latest gradient landscape flatness is proposed as a
tool for any CL approach.
Loss landscape flatness. Let B(0,p) p) (0': |10' 011 < /} denotes the neighborhood of 0 with radius > 0 in the Euclidean space e C Rd, the zeroth-order sharpness at point is commonly defined by the maximal training loss difference within its neighborhood [00, p):
R = ( - Es : B(0,p))-
where (0) denotes the loss of an arbitrary model with parameter 0 on any dataset with an oracle loss function (-). The zeroth-order sharpness ()) regularization can be directly applied to restrain the maximal neighborhood training loss:
2 S (8) (0( ()) max(cs((") o' p)),
(2)
However, for some fixed local minima with a lower loss does not always have lower major hessian eigenvalue [70]. which equals to the neighborhood curvature. It means that zeroth-order sharpness optimizer may goes to sharper suboptimal than to the direction of a flatter global optimal with better generalization ability.
Recently, first-order gradient landscape flatness is proposed as a measurement of the maximal neighborhood gradient norm, which reflects landscape curvature, better describe the smoothness of the loss landscape:
(0) = max[[]V(<(())]] 0' p)).
Unlike zeroth-order sharpness that force the training converging to local minimal, first-order flatness alone constraining on the neighborhood smoothness can not lead an optimal with minimal loss. To maximize the generalization ability of loss landscape sharpness for continual learning task,

---

propose model blocks to be trained are altered regarding the training phase and CL method, (as detailed in the next subsection), we define the the C-Flat loss as follows:
sharpness aware optimizer C-Flat for CL. Considering the data space,
gg (f (0T)) 65T (f" (0") + 1 (fT((T) + A.RR, pp.sT (FT()"))
(t (f" X R p. (f"),,
(4)
with the minimization objective:
mingr (max((<r ( (0)) Ap [V (8" (0 12 0g of B(O ,))
(5)
Ro where (0) is constructed to replace the original CL loss, while (2(0) further regularizes the smoothness of the neighborhood, and hyperparameter À is to balance the influence of Rt as additional regularization to loss function Hence, the local minima within a flat and smooth neighborhood is calculated for generalized model possessing both old and new knowledge.
Optimization. our work, the two regularization terms in the proposed C-Flat are resolved corresoondingy in each iteration. Assuming the loss function (()) is differentiable and bounded, the
RC, gradient of Eg at point 0 can be approximated by
(es(0²)
( (ls((()) with 6 = 0 + V
(6)
And the gradient of the first-order flatness regularization VR (0T)
can be approximated by
(R, V
(7)
V V(r(0"))1
with
0 +p
V Ves
Vos(0")
where
V Vcs V²6)(0")
The optimization is detailed in Appendix algorithm T Note that Ve is the gradient of l with respect to # through this paper, and instead of the expensive computation of Hessian matrix V Hessian-vent vector product calculation is used in our algorithm, where the time and especially space complexity are greatly reduced to o(n) using forward and 1 backward propagation. Thus, the overall caleulation in one iteration takes forward and 4 backward propagation in total.
Theoretical analysis. Given (0) measuring the maximal limit of the training loss difference, the first-order flatness its upper bound by nature. Denoting + c B(0, p) the local maximum point, constant e E [0, E] exists according to the mean value theorem that
0)) marr(s(00) (s()): :0' B)0,
= 6 + c) - ()) (V<=))(((((()))) .cs [+ ||||
S <mar[]V<<<0))l>= E p)) -) p-
(8)
Assuming the loss function is twice differentiable, bounded by M, obeys the triangle inequality, its gradient has bounded variance a², and both the loss function and its second-order gradient are 3 Lipschhit smooth, we can prove that, according
3 63), C-Flat converges in all tasks with " 1/8,p p < 1/48, and vf v of p/ Vi for epoch in any task T.
E ST
<
E
(8"))
n
?
n
8M3
1662
32X 22X2 nd -
EI P.ST (5 ((T(((0)) (0*))
99)
n
VnT
36ynT
B2
where n² is the total iteration numbers of task T, and b is the batch size.

---

Upper Bound. Let (00*)) denotes the Hessian matrix at local minimum 8", its maximal eigenvalue Amaz ((²))())) is a proper measure of the landscape curvature. The first-order flatness proven to be related to the maximal eigenvalue of the Hessian matrix as (0)) p² Amaz (V'6 { )), thus the C-Flat regularization can also be used as an index of model generalization ability, with the following upper bound:
RS -00) + (0) S A)p² Amaz (v)
(10)
3.1 A Unified CL Framework Using C-Flat
This subsection presents an unified CL framework using C-Flat with applications covering Class Incremental Learning (CIL) approaches. To keep focus. the scope of our study is limited in CIL task, which is the most intractable CL scenarios that seek for a lifelong learning model for sequentially arriving class-agnostic data. Most CIL approaches belong to three main families, Memory- -bssed Regularization-be based and Expansion-base based methods.
Memory-based methods store samples from the previous phases within the memory limit, or produce pseudo-samples by generative approaches to extend the current training data space, thus a memory replay strategy is used to preserve the seen class features with S = 5 U Sample'<? iCaRL is one of the early works. It learns classifiers and a feature representation simultancously. and preserves the first few most representative exemplars according to Nearest- Seeares-MMean-ee Mean-of -one----eeeplasss Classification. Thus a loss function gr = &CE + e gKL combining both cross entropy for the current task and a knowledge distillation loss for the previous classes is introduced to balance the learning sensitivity and model generalization to the previous tasks.
sT
Solution: for memory-based method, including, the C-Flat can be easily applied to these scenarios by reconstruct the oracle loss function with its zeroth- and first-order flatness measurement as eq. 11 and trained with algorithm using data set extended with the previous exemplars.
Eg (0*)
RD St (6²)
+ X
(11)
ST
Regularization- based methods seeks for a apply regularization on the model develop to preserver the learnt knowledge. For instance, WA introduces weight aligning the final inference part to balance the old and new classes. Denoting e the feature learning layers of the model, ab = [oo snne]] the decision head for all classes consisting of two branches for the old and new seen data classes, the output is corrected to f(x)= = [woold (#(x)) (ø(x)), 7 spnew (ø(z))]. where y is the fraction of average norm of whold new of all classes.
Gradient Projection Memory (GPM) is another main regularization based group, introducing explicit align the gradient direction to new knowledge learning. It stores a minimum set of bases of the Core Gradient Space as Gradient Projection Memory, thus gradient steps are only taken in its orthogonal direction to learn the new features without forgetting the core information from the previous phases. FS-DGPM further improves this method by updating model parameter along the aligned orthogonal gradient at the zeroth-order sharpness minima dynamic GMP space.
Solution: for regularization-base approaches, the same plug-and-plas strategy can be used to reconstruct the loss function as eq. [1 and optimized by algorithm [
An alternative solution for the gradient- based methods like GPM and the improved FS-DGPM, is to introduce C-Flat optimization at the gradient alignment stage, so that the orthogonal gradient at flatter minima is used to ensure that the training can cross over the knowledge gap between different data categories. The implementation of our C-Flat-GPM is detailed in Appendix algorithm
Expansion- based methods explicitly construct task task- specific parameters to resolve the new class learn- ing and inference problem. For instance, Memory-efficient Expandable Model (Memo) decomposes the embedding module into deep layers and shallow layers that @ = & (øy), where d,, &y correspond to the specialized block for different tasks and the generalized block that can be shared during training phases. An additional block di is added to the deep layers for specified feature extraction for the new classes, where the model can be reconstructed as f Di (IoT-'
(oy)") Thus the new model training is focusing on the task specified component while the shared shallow layers are frozen with loss function e Memo
(CB (N. (bg))67ew (())((,))))) (d))
gT
ST
5

---

Table Average accuracy (%) across all phases using state-of-art methods (span all sorts of CL) w/ and w/o C-Flat plugged in. Maxinum/Average Return in the last row represents the maximum/averse boost of C-Flat towards all methods each column.
Technology
CIFAR-100
ImageNet- 100
Tiny-ImageNNl
Method
Reg.
Mem.
Exp.
B0 Inc5
B0 Inc10
B0_ Inc20
350_Innc10
B50_Inc2 Inc25
B0 30_Ine40
Replay 44
58.83
58.87
62.82
63.89
72.18
43.31
C-Flat
$9.98 f
59.42 f
64.71
63.60
73.37 1
44.95
iCaRL 43
58.66
59.76
61.13
64.78
77.25
45.70
C-Flat
59.13 1
60.40 1
62,93
65.01
76.22 4
46.08
WA [4]
63.36
66.76
73.17
80.81
55.69
w/ C-Flat
65.70 |
67.79
69.16
73.56
83.84
56.06
PODNet (
48.05
56.01
63.45
83.66
85.95
54.24
49.70
56.58
84.31
86.85
55.13
DER [57
69.99
71.01
71.40
85.17
87.10
58.63
w/ C-Flat
71.11 f
72.08 1
72.01
86,64 1
87.96 1
60.14 1
FOSTER 521
63.15
66.73
69.70
84.54
w/ C-Flat
63.58
67.34
70.89
85,40
87.81
58.88
MEMO [8
67.42
69.82
69.91
67.28
83.09
58.15
C-Flat
67.56 |
69.94 |
71.79
69.34 1
83.41 1
58.97 1
Average Return
+1.04% %
+0.66%
+1.34%
+0.62%
+0.90
+0.81
Maximum Return
+2.34 %
+1.07%
+1.89%
+2.06%
+3.03%
+1.64% %
Foster uses KL--based loss function to regularize the three combinations of old and new blocks for stable performance on the previous data. It also introduces an effective redundant parameters and feature pruning strategy maintain the single backbone model using knowledge distillation. DER follows the same framework, and introduces an auxiliary classifier and related loss item to encourage the model to learn diverse and discriminate features for novel concepts.
Solution: for expansion-
approaches. the plug-and -andd strategy is still available. The C- Flat loss can be reformed with the reconstructed model as eq. [2] Thus C-Flat optimization using algorithm [ applied onto the first stage, where the new constructed block are optimized, while the generalized blocks are kept frozen. The final model is obtained after post-processing.
CC (. 6 (lygotd [1oT O7 knew (dg))))
A c St (wword (ineu ([oF- (0g), (6,)))
(12)
To conclude, C-Flat can be easily applied to any CL method with reconstructed loss function, and thus trained with the corresponding optimize as shown algorithm Dedicated design using C-Flat like for the GPM family is also possible wherever flat minima is required.
4 Analysis
4.1 Experimental Setup
Datasets. We evaluate the performance on CIFAR-100, ImageNet- 100 and Tmny-Im Adher- ence to [66] 67], the random seed for class-order shuffling is fixed at 1993. Subsequently, we follow two typical class splits in CIL: (i) Divide all |Y, classes equally into B phases, denoted BO Incy; (ii) Treat half of the total classes as initial phases, followed by an equally division of the remaining classes into incremental phases, denoted B50 _n0__nyy. In both settings, y denotes that lears new classes per task.
Baselines. To evaluate the efficacy of our method, we plug it into 7 top- hop-performing baselines across each CL category: Replay [44 and iCaRL (43) are classical replay-based methods, using raw data as memory cells. PODNet [11] is akin to iCaRL, incorporating knowledge distillation to constraint the logits of pooled representations. WA 64 corrects prediction bias via regularizing discrimination and fairness. DER [57]. FOSTER
54 and MEMO (68) are network expansion

---

Demity (Log
10
Tracer 670.36
Arwa:: 67.48
Domeity (Log Smae)
10"
10"
10"
10"
Trace 429 90
Dessiry (Log Seal))
10"
10"
10
321.36 Amei: 21.07
Srak)
10
10
10"
Tmme: 97.
Anac 6:25
10
10"
10"
10
10
10
10
20
30
40
50
60
10
20
30
40
50
60
5
10
15
20
0
10
15
20
Eigenvalue
Bignnvalue
Eigenvalue
Eigenvalur
(a) MEMO Epoch: 50
(b) MEMO w/ C-Flat
(c) MEMO Epoch: 150
(d) MEMO w/ C-Flat
Figure 2: The Hessian eigenvalues and the traces at epochs 50, and 150 on B0 Inc10 setting (MEMO, CIFAR-100) w/ and w/o C-Flat plugged in.
methods, dedicate modular architectures towards each task by extending sub-network or freezing partial parameters. The aforementioned methods span three categories in CL. 53]: Memory-baser methods, Regularization- bassd methods and Expansion-hase based methods.
Network and training details. For given dataset, we study all methods using the same network architecture following repo [66 677. i.e. ResNet-32 for CIFAR and ResNet-18 for ImageNet. not specified otherwise, the
for all models adhere to the settings in the open-source library 166, 67]. Each task are initialized with the same p and 17, which drops with iterations according to the scheduler from [70]. To ensure fair comparison, all models trained with vanilla-SGB optimizer [7 And the proposed method is plugged into the SGD.
4.2 Make Continual Learning Stronger
Table empirically demonstrates the superiority of our method: Makes Continual Learning Stronger. In this experiment, we plug C-Flat into 7 state-of-the-art methods that cover the full range of CL. methods. From Table we observe that (i) C-Flat presents consistent outperformance on all models, spanning Masory-based methods, Regularization-basel methods, and Expansion- based methods. This superiority indicative of the plug-and-play feature inherent our method, allowing effortless installation with all sorts of CL paradigms. (ii) Across multiple benchmark datasets, including C-Flat exhibits consistent improvement. This underscores its generalization ability and effectiveness against diverse data distributions. (iii) C- Flat presents consistent boosting across multiple incremental scenarios, encompassing BO Inc5, BO Inc10, BO Inc20, B50 Incl0 10, B50 Inc25, and BO Inc40. This consistent boosting reaffirms robustness C-Flat for various CL. scenarios. To sum up, C-Flat advances baselines across each CL category, serves valuable addition to CL. offering versatile solution that complement existing methods.
CIFAR-100, ImageNet-100, and Tiny-ImageN
4.3 Hessian Eigenvalues and Hessian Traces
Hessian eigenvalues. Equation TO delineates the connection between fist-order flatness and Hessian eigenvalues in CL. Broadly, Hessian eigenvalues serve as metric for assessing the flatness of function. Thus we report Hessian eigenvalue distributions Figure for empirical analysis. As shown in Figure 2 models trained with vanilla-SGD exhibit higher maximal Hessian eigenvalues (67.48/21.07 at epuchs 50/150 in Figure 2a and Figure 2c while our method induces a significant drop in Hessian eigenvalues to 28. 28.11/0 1/6.25 epochs 50/150 in Figure 2b and Figure 2d) during CL, leading to flatter minima. Consequently, the performance of CL is tangibly enhanced.
Hessian traces. We calculate the empirical Fisher information matrix an estimation of the Hessian and leverage the trace of this to quantify the flatness of the approximation loss at the convergence point. As depicted in Figure we observe that substantial reduction in the Hessian trace when employing our method compared with vanilla-SGD (670.36/321 .336 drops to 429, 229.90997.34 at epochs 50/150 in Figure 2b and Figure 2d). This observation suggests that our method induces a flatter minimum. These findings not only align with but also substantiate the theoretical insights presented in the methodology section.
7

---

Reghy
(a) Replay w/ C-Flat
(b) WA w/ C-Flat
(c) MEMO w/ C-Flat)
Figure 3: The parametric loss landscapes of Replay (Mem.), WA (Reg.) and MEMO (Exp.) are plotted by perturbing the model parameters the end of training (CIFAR-100, B0_Inc 10) across the first two Hessian eigenvectors.
05.43
10.15
-50
$1.01
SCD
-S1M
-GM
61.5
DAM
-C-f1
C- (18k))
2000
(a) BO Incl0
(b) BO Inc20
(a) Convergence analysis
(b) Training time
Figure 4: C-Flat vs. Zero-order flatness
Figure 5: Analysis of computation overhead
4.4 Visualization of Landscapes
More intuitively, we present a detailed visualization of landscape. PyHessian [59] is used to draw the loss landscape of models. To simplify, we choose one typical method from each category of CL methods (Replay, Wa, MEMO) for testing. Figure cleearly illustrates that, by applying C-Flat, the loss landscape becomes much flatter than that of the vanilla method. This trend consistently holds across various categories of CL methods, providing strong empirical support for C-Flat, and confirms our intuition.
4.5 Revisiting Zeroth-order Flatness
Limited work 10 49 proved that the
Table 2: Revisiting FS-DGPM series using C-Flat.
zeroth-order sharpness leads to flat min- ima boosted CL. Here, we employ zeroth-order optimizer [19] instead of vanilla-SGD to verify the performance of C-Flat. As shown in Figure 4 C-
Method
La-GPM
FS-GPM
DGPM
La-DGPM
FS-DGPM
Oracle
72.90
73.12
72.66
72.85
73.14
C-Plat
73.66
73.57
73.01
73.64
73.72
Flat (purple line) stably outperforms the zeroth-order sharpness (blue line) on all baselines. We empirically demonstrated
Boost
+0.76
+0.45
+0.35
+0.79
+0.58
that flatter is better for CI.,
Former work FS-DGPM (10) regulates the gradient direction with flat minima to promote CL. The FS (Flattening Sharpness) term derived from FS-DGPM is a typical zeroth-order flatness. We revisit the FS-DGPM series (including La/FS-GPM, DGPM, La/FS-DGPM
[10] 46) evaluate performance using C-Flat instead of FS (see aigorithm 2. Table 2 yields two conclusions: (i) C-Flat boosts the GPM [6) baseline pluggable regularization term. This not only extends the frontiers of CL methods, incorporating gradient- based solutions, but also reaffirms the remarkable versatility of C-Flat. (ii) Throughout all series FS-DGPM, C-Flat seamlessly supersedes FS and achieves significantly better performance. This indicates that C-Flat consistently exceeds zeroth-order sharpness. Hence, reconfirming that C-Flat indeed a simple yet potential CL. method that deserves to be widely spread within the CL community.
8

---

73
MAA Begle -MEMO
MIEMO
SAN KAM WC-MM
SAM GAN C-E
(a) CL methods
(b) p on CL. methods
(c) Effects of p
(d) Effects of p scheduler
Figure 6: Ablation study about À and p. (a) and (b) represents the effect of X and p on different CL methods (WA, Replay, MEMO). (c) and (d) represents the effect of p and p scheduler on MEMO
with different optimizers (SGD (red line), SAM, GAM, C-Flat).
4.6 Computation Overhead
To assess the efficiency of C-Flat, we provides a thorough analysis from the convergence speed and running time with CIFAR-100/BO Inc20 on Replay. As shown in Figure C-Flat compared with SGD and other flatness-aware optimiziters. We train C-Flat optimizers on CL benchmarks with 20%, 50%, 100% of iterations and approximately 60% of epochs, while holding the other optimizers at 100%. Figure first shows that C-Flat converges fastest and has the highest accuracy (purple line), meaning few tterations/epochs with C-Flat is enough to improve CL. Figure 5 shows i) Compared with SGD, with only 20% of iterations and 60% of epochs (pink line) using C-Flat, CL performance improved using slightly less time; ii) C-Flat surpasses GAM with similar time as SAM when setting the iterations/ peraais ratio to 50%/66%; iii) Models trained with C-Flat for 100 epochs outperform those trained with other optimizers for 170 epochs. To sum up, we show that C-Flat outperforms current optimizers with fewer iterations and epochs. This indicates the efficiency of C-Flat.
To discuss practicality better, provided tier guideline, which categorizes C-Flat into L1 L3 levels, as shown in Table 3 L1 de- notes the low-speed version of C-Flat, with
Table 3: A tier guideline of C-Flat.
Level
Speed
Boost (SGD/SAM)
slightly lower speed than SAM and the best per- formance; 1.2 follows next; 1.3 denotes the high- speed version of C-Flat, with faster speed than SGD and performance close to L2.
L1
SGD SAM > C-Flat
+2.39%%/+1.9
L.2
SGD > C-Flat > SAM
+1.52%/41, 1.04%
L3
C-Flat SGD
+1.51%/+1 1.03%
4.7 Ablation Study
We perform ablation study in cases: (i) the influence of and p different CL methods; (ii) the influence of and its scheduler on different optimizers.
We first present the performance of C-Flat with varying À and p. As described in Eq. 13, À controls the strength of the C-Flat penalty (when equal to O, this means that first-order flatness is not implemented). As shown in Figure ba compared with vanilla optimizer, C-Flat shows remarkable improvement with varying A. Moreover, P controls the step length of gradient ascent. As shown in Figure 6b, C-Flat with p larger than 0 outperforms C-Flat without gradient ascent, showing that C-Flat benefits from the gradient ascent.
For each CL task T, same learning rate '7' and neighborhood size pT initialization are used. By default, p. [p_. p+] is set as a constant, which decays with respect to the learning rate nj E [n_, 7+] by P.- = P__ + (e) [p] (nī nT.). Figure 6c and Figure 6d present a comparison on initialization 0. -n
and (p_ P++ scheduler. C-Flat outperforms across various settings, and is oversensitive to hyperparameters in reasonable range.
4.8 Beyond Not-forgetting
As is known to all, forward, and in particular backward transfer, are the desirable conditions for CL. 22 Here, we thoroughly examine the performance of C-Flat both aspects. Forward Transfer
(FT) means better performance each subsequent task. Backward Transfer (BT) means better

---

Figure 7: Analysis of BT and FT. RR refers to Relative Return on w/o and w/ C-Flat.
CIFAR-I00 B0_lnc5
Method
C-Fatt
RT+2.10%
iCaRL
80.25
82.20
FT+2.43%
old 46.32
47.44
BT+2.42%
PODNer
nEW 62.65
64.75
FT+3.35%
old 58.50
61.35
BT+2.85%
05 110
FOSTER
new 62.05
63.05
FT+1.61%
Figure 8: Loss and forgetting of old tasks.
performance on previous tasks, when revisited. We count the performance of new and old tasks on several CL benchmarks before and after using C-Flat. As observed in Table 7 C-Flat consistently improves the learning performance of both new and old tasks. This observation indicates that C-Flat empowers these baselines with robust forward and backward transfer capabilities, that learning task should improve related tasks, both past and future. But, thus far, achieving a baseline that maintains perfect recall (by forgetting nothing) remains elusive. Should such a baseline emerge, C-Flat stands poised to empower it with potent backward transfer, potentially transcending the limitations mere not-forgetting.
Moreover, one of our contributions to prove the positive effect of low curvature on overcoming forgetting. Intuitively, we visualized the change in loss and forgetting of old tasks in CL. Figure shows the lower loss or less forgetting (red line) for old tasks during CL. This is an enlightening finding.
Conclusion
This paper presents versatile optimization framework, C-Flat, confront forgetting. Empirical results demonstrate C-Flat's consistently outperform on all sorts of CL methods, showcasing its plug-and-play feature. Moreover, the exploration of Hessian eigenvalues and traces reaffirms the efficacy of C-Flat in inducing flatter minima to enhance CL. In essence, C-Flat emerges as simple yet powerful addition to the CL toolkit, making continual learning stronger.
Acknowledgments
This work was supported in part by the Chunhui Cooperative Research Project from the Ministry of Education of China under Grand HZKY20220560, in part by the National Natural Science Foundation of China under Grant W2433165, and part by the National Natural Science Foundation of Sichuan Province under Grant 2023YFW20009.
References
Davide Abati, Jakuh Tomezak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Eht- eshami Bejnordi. Conditional channel gated networks for task-aware continual learning. CVPR, 2020.
[2] Afra Feyza Akyürek, Ekin Akyürek, Derry Tanti Wijaya, and Jacob Andreas. Subspace regularizers for few-shot class incremental learning. ICLR, 2022.
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In ICML, 2022.
[4] Carlo Baldassi, Fabrizio Pittorino, and Riccardo Zecehina. Shaping the learning landscape in neural networks around wide flat minima. Proceedings of the National Academy of Sciences, 2020.
[5] Prashant Bhat, Bahram Zonooz, and Elahe Arani. Task-aware information routing from common represen- tation space lifelong leaming. ICLR. 2023.
10

---

[6] Sungmin Cha, Hsiang Hsu, Taebaek Hwang, Flavio P Calmon, and Taesup Moon. Cpr: plassiferr-roojetioon regularization for continual learning, ICLR, 2021.
[7] Arslan Chaudhry, Mare Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficientt lifelong learning with a-gem. arXiv preprint /8 /8/2.00420 2018.
[8] Runhang Chen, Xiao- Jing, Fei Wu, and Haowen Chen. Sharpness-aware gradient guidance for few-shot class-incremental learning. Knowl. Based Syst., 299:1 1 10030, 2024.
[9] Matthias Delange, Rahaf Aljundi, Mare Masana. Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[10] Danruo Deng, Guangyong Chen, Jianye Hao, Qiong Wang, and Pheng-Ann Heng. Flattening sharpness for dynamic gradient projection memory benefits continual learning. NeurlPS, 34, 2021.
[11] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small tasks incremental learning. In ECCV, 2020.
[12] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian lehter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 2023.
[13] Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. NeurlPS, 2022.
[14] Yunshu Du, Wojciech M Czamnekki Siddhant Jayakumar, Mehrdad Farajtahar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv: /8/2.0222222 2018.
[15] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 3762-3773. PMLR, 2020.
[16] Tao Feng, Kaifan Ji, Ang Bian, Chang Liu, and Jianzhou Zhang. Identifying players in broadcast vidcos using graph convolutional network. Pattern Recognition, 124: 108:108503, 2022.
[17] Tao Feng. Mang Wang, and Hangjee Yuan. Overcoming catastrophic forgetting in incremental object detection via elastic response distillation. In CVPR, 2022.
[18] Tao Feng, Hangjie Yuan, Mang Wang. Ziyuan Huang, Ang Bian, and Jianzhou Zhang. Progressive learning without forgetting. arXiv preprint ar 2 2022.
[19] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprini arXiv:2010.01412, 2020.
[20] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. ICLR, 2021.
[21] Qiankun Gao, Chen Zhao, Bernard Ghanem, and Jian Zhang. R-dfeil: Relation-guided representation learning for data-free class incremental learning. ECCV, 2022.
[22] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12): 1028- 1040, 2020.
[23] Haowei Hc, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. NeurlPS, 32, 2019.
[24] Zhiyuan Hu, Yunsheng Li, Jiancheng Lyu, Dashan Gao, and Nuno Vasconcelos. Dense network expansion for class incremental learning. In CVPR, 2023.
[25] Kishaan Jeeveswaran, Prashant Bhat, Bahram Zonooz, and Elahe Arani, Birt: Bio-inspired replay in vision transformers for continual lcarning. ICML, 2023.
[26] Xisen Jin, Arka Sadhu, Junyi Da, and Xiang Ren. Gradient-based editing of memory examples for online fask-free continual learning, NewrIPS, 2021.
[27] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang, On large-batch batch training for deep learning: Generalization gap and sharp minima, arXiv preprint arXiv:1609.0483. 2016.
11

---

[28] Do-Yeon Kim, Dong-Jun Han, Jun Seo, and Jaekyun Moon. Warping the space: Weight space rotation for class-incremental few-shot learning. In ICLR, 2022.
[29] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- BarkkaBBaaiska,,, et al. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.
[30] Yajing Kong, Liu Liu, Huanhuan Chen, Janusz Kacprzyk, and Dacheng Tao. Overcoming catastrophic forgetting in continual learning by exploring eigenvalues of hessian matrix. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[31] Tatsuya Konishi, Mori Kurokawa, Chihiro Ono, Zixuan Ke, Gyuhak Kim, and Bing Liu. Parameter-level soft-masking for continual learning. arXiv preprint arXiv:2306. 14775, 2023.
[32] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., 40<12):2935-2447 2018
[33] Huiwei Lin, Baoquan Zhang. Shanshan Feng, Xutao Li, and Yumming Ye. Per: Proxy-based contrastive replay for online class-incremental continual learning. In CVPR, 2023.
[34] Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning. arXiv preprint arXiv:2202.0293 2022.
[35] Hao Liu and Huaping Liu. Continual learning with recursive gradient optimization. ICLR, 2022.
[36] Yaoyao Liu, Bernt Schicle, and Qianru Sun. Adaptive aggregation networks for class-incremental lcarning. In CVPR, 2021.
[37] Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsich, and Yang You. Towards efficient and scalable sharpness-aware minimization. In CVPR, 2022.
[38] David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. NeurlPS, 2017.
[39] Aojun Lu, Feng, Hangjie Yuan, Xiaotian Song, and Yanan Sun. Revisiting neural networks for continual learning: An architectural perspective. IJCAI, 2024.
[40] Mare Masana, Xialei Liu, Bartlomicj Twardowski, Mikel Menta, Andrew Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[41] Sanket Vaihhav Mehta, Darshan Patil, Sarath Chandar, and Emma Struhell. An empirical investigation of the role of pre-trainigg in lifelong learning. J. Mach. Learn. Res., 24:214:1-214:50 2023.
[42] Youngmin Oh, Donghyeon Back, and Bumsub Ham. Alife: Adaptive logit regularizer and feature replay for incremental semantic segmentation. NeurIPS, 2022.
[43] Sylvestro-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, 2017
[44] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In NeurlPS, volume 32, 2019.
[45] Tim GJ Rudner, Freddie Bickford Smith, Qixuan Feng. Yee Whye Teh, and Yarin Gal. Continual learning via sequential function-space variational inference. In ICML, 2022.
[46] Gobinda Saha, Isha Garg, and Kaushik Roy. International Conference on Learning Representations, 2020.
Gradient projection memory for continual learning. In
[47] Ozan Sener and Vladien Koltun, Multi-task learning as multi-ohjective optimization. NewrIPS, 2018.
[48] Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In /CML, pages 4555-4564, 2018.
[49] Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu, Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima. NeurIPS, 2021.
[50] Wenju Sun, Qingyong Li, Jing Zhang, Wen Wang, and Yangli-ao Geng. Decoupling learning and remembering: A bilevel memory framework with knowledge projection for insk-incremental learning. In CVPR, 2023.
12

---

[51] Zhicheng Sun, Yadong Mu, and Gang Hua. Regularizing second-order influences for continual learning. In CVPR. 2023.
[52] Lam Tran Tung. Viet Nguyen Van, Phi Nguyen Hoang, and Khoat Than. Sharpness and gradient aware minimization for seemory-basedd continual learning. In Proceedings of the 12th International Symposium Information and Communication Technology, SOICT 2023, Ho Chi Minh, Viemam, December 7-8, 2023, pages 189-196. ACM, 2023.
[53] Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental leamning Nature Machine Intelligence, pages 1185-1197, 2022.
[54] Fu- Yun Wang. Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. In European conference on computer vision, pages 398-414, 2022.
[55] Liyun Wang, Xingxing Zhang, Hang Su, and Jun Zhu. comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00455 2023.
[56] Zirui Wang and Yulia Tsvetkov. Gradient vaccine: Investigating and improving multi-task optimiza- tion in massively multilingual models. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[57] Shipeng Yan, Jiangwei Xie, and Xuming Hc. DER: dynamically expandable representation for class incremental learning. In CVPR, pages 3014-3023, 2021.
[58] Enneng Yang, Shen, Zhenyi Wang, Shiwei I Liu, Guibing Guo, and Xingwei Wang. Data augmented aatness-aware gradient projection for continual learning. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 5607-5616. IEEE, 2023.
[59] Zhewei Yao. Amir Gholami, Kurt Keutzer, and Michael Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pages 581-590. IEFE, 2020.
[60] Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robus continual learning with additive parameter decomposition. In International Conference on Learning Representations, 2020.
[61] Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng. Ziyuan Huang, Dong Ni, and Mingqian Tang. Rlip: Relational language-image pre-training for human-ohject interaction detection. In NeurIPS, 2022.
[62] Hangjie Yuan, Shiwei Zhang, Xiang Wang, Samuel Albanie, Yining Pan. Tao Feng, Jianwen Jiang, Dong Ni. Yingya Zhang, and Deli Zhao. Rlipv2: Fast scaling of relational language-image pre- vre-training. In ICCV,
2023.
[63] Xingxuan Zhang, Renzhe Xu, Han Yu, Hao Zou, and Peng Cui. Gradient norm aware minimization seeks first-order flateess and improves generalization. In CVPR 2023, pages 20247-20257, 2023.
[64] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and faimess in class incremental learning. In CVPR, 2020.
[65] Qihuang Zhong, Liang Ding, Li Shen, Peng Mi, Juhua Liu, Bo Du, and Dacheng Tao. Improving sharpness-aware minimization with fisher mask for better generalization on language models. arXiv preprint arXiv:2210.054997 2022.
[66] De- Zhou, Fu-Yun Wang, Han-Jia Ye, and De-Chuan Zhan. Pycil: A python toolbox for class- incremental learning, 2023.
[67] Da- Wei Zhou, Qi-Wei Wang, Zhi- Hong Qi, Han Ye, De-Chuan Zhan, and Ziwei Liu. Deep class- incrmmnntll learning: A survey, arXiv preprint arXiv:2302.03644 2023.
[68] Da- Wei Zhou, Qi-Wei memory-efficient class-incremental learning. ICLR, 2023.
Wei Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards
[69] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zue Zha. Self-sustaining representation expansion for non-exemplar class-incremental learning. In CVPR, 2022.
[70] Juntang Zhuang, Boging Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-awart training. arXiv preprint arXiv:2203.0800 2022.
[71] Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Machine Learning, Proceedings of the Twentieth International Conference (ICML). 2003.
13

---

A Appendix
A.1 Overview
In this supplementary material, first present more intuitive visualizations of C-Flat, elucidating the loss landscape from local viewpoint (Appendix(A... (A3.1 and each task during CL (Appendix (A.5.2). Next, we provide details the accuracy and runtime trade-offs of other CL methods with C-Flat (Appendix A.6)
A.2 C-Flat
We summarize the pseudo code of C-Flat algorithm T
Algorithm 1 C-Flat Optimization
Input:
Training phase T. training data S model fr' with parameter ( from last phase
T 1, batch size b, oracle loss function é, learning rate O, neighborhood size p. trade-off' coefficient A, small constant c. Model trained at the current time T with C-Flat. Initialization:
Output:
if T=1: then Randomly Initialize parameter 0T=1 "T 1/==1 7/ pT pp else
Reconstruct the model and training set if necessary,
Initialize model parameter 0 according to the training strategy, like randomly initialization or 0t 0 in pre-trained model based approaches, 17" 17, p² p.
Frozen part of the parameter if required.
end if Optimization:
while 8d not converge, do
Sample batch B² of random instances from ST Compute batch's loss gradient J8" V6gT (FT((T))
Compute R. perturbation: 6o p² 1 llate
larr!
Approximate zeroth-order gradient: 90 VtBT (F + +
Compute hessian vector product: lBT = V26 oz or (0) T TV (or))]
Compute Rt perturbation: E1 = p" llate
RRT
Approximate first-order gradient: 91 = BT (TT +()) (0"+
Update: Model parameter: HT HT 7" (go + Ag1): Update training parameters 17 PT according scheduler that the values drop with iterations:
end while
Post-Processing Model and Training data if required. return Model f with parameter f
A.3 C-Flat-GPM
We summarize the pseudo code of C-Flat for GPM family in algorithm 2
A.4 Convergency Proof
Assumptions I: the loss function twice differentiable, bounded by M, with bounded variance o², and obeys the triangle inequality. Both the loss function and its second-order gradient are ree3- -LLischhitz smooth. 1/8,p p < 1/48, and ni." n/vi, of p/ Vi for epoch in any task T.
14

---

Algorithm 2 C-Flat for GPM-family at T > 1
Input:
Training set ST. parameter 6" = 6T loss l, learning rate 7, 72, basis matrix M and significance A from replay buffer. while H not converge, do
Sample batch B²
Compute perturbation e, using C-Flat optimization
Update basis significance: A 1/1 VA ( tc)
Update model parameter: 0² = 6" 1/2 M.MM V l 1 +
Update M and replay buffer.
end while
return Model parameter 6
Claim 1: with Assumptions 1, the convergency of zeroth-sharpness with batch size b is guaranteed by
43
80²
E
Vn. (C(0) c(0*)]
(13)
3bv
hence, the zeroth-order part of C-Flat is bounded:
T
E
R (f (8 MI
VnT (f
43
(f (fT())
802
4MB
80²
(14)
n
3bvn'
VnT
3bvn
Lemma 1: let Eor (0) = b, (f (0), ft (0))) with Assumptions 1, the first-order part is bounded by
n
{f (0 )1
Ell 60r + - (V
n?
5t
n
1=1
82
?
321p2 n nT
0² n Eii
16(2Vn 1)
B||le|111
P 1|
(15)
n
n
B2nT
Theorem 1: with Assumptions 1, by combining the zeroth- and first-order parts, we can prove C-Flat converges in all tasks that,
2 n EE
<
Σ
R (F (07)
nT 1=1
n n
8M3 VnT
16
321² 32X2(2V nT - 1)
+ EIIAR p.s (8T(()))
+
(16)
3byn'
B2
A.5
More Visualizations of C-Flat
In this section, we present additional visualization of the loss landscape involving two cases using PyHessian 5 (i) Changes the loss landscape from localized viewpoints; (ii) Changes in the loss landscape across each task during CL.
A.5.1 Landscapes in a Local Viewpoint
First, we present a more detailed visualization through changes in the local region of the loss landscape. We set minimal radius threshold. At this threshold, more detailed changes are displayed. Similarly, we choose three typical method (Replay 44 WA (4] MEMO (68 from each category of CL. methods for visualization. shown Fig. a tiny view, C-Flat contributes to a flatter surface, a change that more intuitively reveals the mechanism of C-Flat.
15

---

(a) Replay
(b) WA
(c) MEMO
Figure 9: The visualizations of loss landscapes in a local viewpoint (Replay, WA and MEMO)
A.5.2 Landscapes Across Each Task
Second, further visualize the loss landscape of more tasks using PyHessian for intuitive explanations during CL. To simplify, we choose one CL. method (Replay (44) for visualization on
task 2, 12 and 17 with 5 task intervals. As shown in Fig. 10 (a) to (d), the loss landscape all
becomes much flatter than that of the vanilla method across each task. This trend provides strong empirical support for C-Flat.
Task 2
(b) Task 7
(c) Task 12
(d) Task 17
Figure 10: The visualizations of loss landscapes during CL.
A.6 Overhead of C-Flat
To enhance the computing efficiency, we apply C-Flat in a limited number of iterations within each epoch. Remarkably, we observe that without executing C-Flat in every iteration also significantly boost the performance CL (All cases derived from C-Flat improves CL performance). As illustrated in Table 4 10% C-Flat iterations is enough to improve CL performances, and around 50% C-Flat iterations is enough to approach and even exceed the impact full C-Flat training. As consequence, the overhead of 50% C-Flat is at least 30% shorter compared with the full C-Flat training. These observations holds potential for light C-Flat boosted CL. applications.
Table 4: Accuracy and training speed of training with different ratios of iterations using C-Flat. Superscripts denotes the ratio iterations each epoch is trained with 100%, 50%, 20% and 10%.
Method
C-Flat'
C-F
C.
C-Flat
Oracle
Replay
61.02 (100%)
60.98 (67%)
60.63 (40 ())
60.48 (34%)
60.28
iCaRL
43
63.04 (100%)
62.94 (65%)
62.78 (41 %)
62.75 (35%)
62.74
WA 64
68.67 (100%)
68.20 (59%)
67.96 (38%)
68.02 (31 %))
67.75
PODNet
64.35 (100%)
63.82 (60%)
63.27 (39 %))
63.80 (34%) ())
63.05
DER 57
72.25 (100%)
71.82 (59%)
71.82 (39 %
71.44 (31%)
71.52
FOSTER 54
70.24 (100%)
70.43 (70%)
69.99 (52%)
69.71 (47%)
69.30
MEMO
68
69.97 (100%)
70.03 (64%)
70.48 (41 %)
69,90 (32%)
69.71
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly stated the contributions and scope in the abstract and introduc- tion. And our claims match experimental results conducted in various settings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
There is no new theoretical result in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have detailed the settings of our proposed method and experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: We have provided details for reproduction in Appendix. Code will be publicly available upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
public/guides otw.:d////oooeuubbmsssssoooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https wetgguddddeeeuuubbueeee ssinooliccy for more details.
//nips.co
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have detailed all key experimental settings in the main paper, and hyper- parameters all employed CL methods adhere the settings in the open-source libraries unless specifically stated.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not use eтror bars, but present extensive experiment results across CL methods, datasets, incremental settings, and architectures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have details this information in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
There are no ethical issues involved this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is just a basic study on continual learning, and does not directly address societal impacts to the best of our knowledge.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We have not released any data models that have risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we have cited all original papers that produced the code packages and datasets used in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have not introduced any new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "cSQU86QZQJ", "reviewer": "Reviewer_p3Vh", "summary": "The paper proposes a new algorithm agnostic optimisation method, which is tailored specifically for continual learning. This method takes advantage of zeroth order landscape sharpness-aware optimisation and proposes a new method, which improves upon SGD.", "questions": "1) the confidence intervals for the experimental  analysis is crucial for the understanding of the performance improvements. \n2) The comparison is only done in relation to the standard SGD; would it make sense to compare such results with other optimisers such, e.g., Adam? Does this optimiser improve upon the SGD performance in the standard, offline learning scenarios? \n3) The paper only evaluates the optimiser on class-incremental learning benchmarks. I wonder how does the method perform on domain-incremental settings using, for example, the protocol from van de Ven et al (2022)\n4) Should the authors also compare the proposed method with other methods taking advantage of sharpness-aware optimisation?  If not, why?  The concern is that currently the model only compares with a weak baseline of standard SGD, while existing similar methods might provide better performance. \n\nvan de Ven, G.M., Tuytelaars, T. & Tolias, A.S. Three types of incremental learning. Nat Mach Intell 4, 1185–1197 (2022). https://doi.org/10.1038/s42256-022-00568-3\n\n[21] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. NeurIPS, 32, 2019.", "limitations": "1) It would be important to hear more about the choice of parameters: Section 4.1 states that for all methods, the hyperparameters have been chosen to be the same. At the same time, it would be interesting to know what procedure did the authors use to choose these hyperparameters.\n\n2) It might be also good to hear if the authors have any details on failure modes for the method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: the method raises a question which is often overlooked, which is the role of stochastic optimisers in continual learning. While many of the existing methods focus on the continual learning algorithms, this submission considers orthogonal aspect, which is tailoring the optimisation process in accordance with the needs of continual learning. \n\nQuality: the ablation studies show the parameter choice trade-off, time expenditure and the performance of the method; however the lack of confidence intervals does not allow to find out how significant this performance improvement is. The presentation and the derivation of the method look clear.\n\n Clarity: the paper is clear and well-structured in general, however, please refer to the weaknesses section for more detail on omissions.", "weaknesses": "Clarity: there has been a number of omissions in the paper which need to be fixed and which stay in a way of understanding of the paper. This includes, for example, \n\nLine 38: \"Another group of works seeks to preserve model generalization with **regulations** onto the training procedure itself\" regularisation (instead of regulations?) of the training procedure?\n\nLine 40: \"are designed to encourage the training to efficiently extracting **feathers** for the current data space without forgetting\" Features?\n\nLine 188: 'Foster uses KL-based loss function to regularize the three combinations of old and new blocks for a stable performance on the previous data. It also introduces an effective redundant parameters and feature pruning strategy to maintain the single backbone model using knowledge distillation. DER follows the same framework, and introduces an auxiliary classifier and related loss item to encourage the model to learn diverse and discriminate features for novel concepts'\nWhile I understand it is related to the methods described in Table 1, they are only discussed in the next section, it would be easier if the authors cited the FOSTER paper for clarity in  Section 3.1; furthermore, it is capitalised in the table but not in the description.\n\nQuality: The proposed experimental analysis of the method does not come across as fully backing up the claims: from table 1, the accuracy improvement does not looks too big, especially given that there are no confidence intervals given which does not allow us to make the conclusions on whether the advantage is significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**NOTE**: Table R1-R4 and Figure R1 are attached to the one-page PDF.\n\n> Weakness 1: Fix the omissions.\n\nA: Thanks for your kind reminder. We have fixed the omissions and conducted a thorough proofreading for clarity.\n\n> Weakness 2: Significance analysis.\n\nA: i) Note that we set the fixed random seed (seed everything) to ensure that the training results are unique in all experiments during CL [58,59], which shows that our significance is well-supported. \n\nii) Your nice suggestion inspires us that the class order (or task order) often significantly affects the performance of CL methods, a significance analysis of this is crucial. Hence, we set various seeds (1993, 1995, 1997) to split the dataset to further analyze the significance of C-Flat. In Table R1, we report the results, mean, std of running 3 times on different datasets. Table R1 concludes that the results of C-Flat remains significant across 3 datasets on all case. Finally, we will discuss significance in the revision.\n\n> Question 1: Compare with Adam.\n\nA: In Table R2, we further compare C-Flat to Adam. Table R2 shows that C-Flat improves the performance of Adam in most cases. \n\nMoreover, the reasons why we use SGD as a baseline in evaluation instead of Adam are as follows,\n\n+ SGD is the mainstream used in the CL community[58,59]. In CL, SGD performs better than Adam in CNN based vision tasks with higher test accuracy. This can also be concluded from the result of SGD and Adam in Table 2. While Adam outperforms SGD in Transformer-based models, like the foundation model, language model. \n\n+ Adam is better at saddle-point escaping but worse at flat minima selection than SGD[C2]. We analyze the mean escape time $\\tau$ as follows, \n\n$\\tau$ of SGD is $\\log (\\tau)=O\\left(\\frac{B \\Delta L_{a b}}{\\eta H_a}\\right)$; $\\tau$ of Adam is $\\log (\\tau)=O\\left(\\frac{\\sqrt{B} \\Delta L_{a b}}{\\eta \\sqrt{H_a}}\\right)$\n\n$\\tau$ of Adam exponentially depends on the square root of the eigenvalues of the Hessian at a minimum[C2], while $\\tau$ of SGD exponentially depends on the eigenvalue of the Hessian at minima along the escape direction[C1].\n\n+ The saddle-point escape behavior of Adam is approximately independent of saddle-point flatness, whereas SGD is linearly dependent[C2].\n\nTherefore, these observations and conclusions above motivated us to use SGD as a baseline.\n\n[C1] A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. ICLR 2021.\n\n[C2] Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum. ICML 2022.\n\n> Question 2: The performance in offline-learning.\n\nA: Adhere to the protocol [C3], we evaluate the effectiveness of C-Flat in offline learning scenes. As shown in Table R2, C-Flat still provides stable performance gains. This concludes that C-Flat still converges to flat minima even in offline learning cases, showing C-Flat features a good generalization.\n\n[C3] Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning. CVPR 2021.\n\n> Question 3: The performance in domain-incremental settings.\n\nA: Indeed, it should be interesting to see the performance of C-Flat in domain-incremental tasks, and Reviewer 475M is wondering about that as well. We would be happy to conduct a further evaluation on C-Flat.\n\nSo we tested the effectiveness of C-Flat according to the protocol [48]. Table R3 shows that C-Flat still provides consistent performance gains in domain-incremental scenarios. This means that C-Flat smoothly migrates to the global optimal of the joint space of the new and old tasks, thus mitigating catastrophic forgetting. This trend also occurs in task-incremental scenarios.\n\nApart from the typical benchmark, inspired by your suggestion and Reviewer 475M, we further investigated how a larger domain gap affects C-Flat. In this case, the domain shifts much more drastically, thus posing a non-trivial challenge for C-Flat. We trained on ImageNet21K and then evaluated CL performance on IN-R/ObjNet. Please see the Table below, C-Flat surprisingly holds solid performance gains. Such an observation further enhances the technical contribution of C-Flat. You could also refer to Reviewer 475M's question and our response to it.\n\n| Method | IN-R B0_Inc20 | ObjNet B0_Inc10 |\n|:-:|:-:|:-:|\n| iCaRL | 72.13 | 48.06 |\n| w/ C-Flat  |  **72.92** (+0.79) | **49.59** (+1.53) |\n| MEMO | 70.96 | 56.22 |\n| w/ C-Flat | **71.69** (+0.73) | **56.50** (+0.28) |\n\n> Question 4: Compare C-Flat with other sharpness-aware optimization.\n\nA: Actually, sharpness-aware method is relatively new in CL, among the limited works, we took the more general approach FS-DGPM for comparison in Table 2.\nWe compared the accuracy, convergence speed and result with different of parameters of C-Flat with other sharpness-aware method like SAM, GAM in Section 4.6 and 4.7 with Figure 5 and 6. \n\nTo ensure the reliability, we further expanded the scope of the evaluation. In details, we compare C-Flat with more sharpness-aware methods, e.g, SAM, GSAM (newly added), and GAM. Then, the baseline methods used also increased from 1 of Figure 5 to 3, including Replay, WA, MEMO. Table R4 shows that C-Flat still achieves better accuracy. \n\n> Limitation 1: Choice of hyper-parameters.\n\nA: If not specified, the hyper-parameters for tested models are defaulted according to the open-source repo [58,59]. Each task are initialized with the same $\\rho$ and $\\eta$, which drops with iterations according to the scheduler described in section 4.7. \n\n> Limitation 2: Any details on failure modes for C-Flat.\n\nA: As our response to Q1, since the saddle-point escape behavior of Adam is approximate independent of saddle-point flatness, it cannot be as good as SGD at finding flat minima. In this sense, C-Flat is prone to fail on the Adam series, the results in Table R2 reconfirm this. As shown in Table R2, Adam showed a slight drop in some cases. We will discuss these in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H1V9NtKws6", "reviewer": "Reviewer_PuVp", "summary": "Continual learning seeks to learn a series of new tasks without forgetting old ones. This paper explores the impact of a flat loss landscape on catastrophic forgetting.", "questions": "- In Section 2, why should “Gradient-based solutions” be a separate paragraph?\n- In Figure 5, why does the loss become smooth after the 80th epoch, but it is very drastic before?", "limitations": "See Weaknesses", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper applies loss landscape optimization to multiple categories of continual learning methods.\n- The method proposed in this paper is easy to implement and the article structure is well organized.", "weaknesses": "- The contribution of this paper is unclear. The authors claim that they are the first to compare CL methods with loss landscapes. However, there are many works that have discussed the impact of flatness on continual learning (or catastrophic forgetting) [1-5], which are not discussed.\n- The discussion of related work is not comprehensive. First, the connection and difference between the proposed method and the existing loss landscape-based continual learning methods [1-5] are not fully discussed in related work. In addition, there is also a lack of discussion on the difference and connection between the proposed C-Flat and various existing sharpness-aware minimization methods.\n- There is a lack of performance comparison with related work, so it is unclear how the gains compare to other works that improve CL based on flatness [1, 2, 3].\n- The proposed C-Flat has significantly larger computational cost. As mentioned in Section 3, it requires 2 forward and 4 backward propagations.\n- The results in Tab.2 show that C-Flat has only a slight performance improvement.\n\nReferences:\n\n[1] Cha, S., Hsu, H., Hwang, T., Calmon, F. P., & Moon, T. CPR: classifier-projection regularization for continual learning. In ICLR, 2021.\n\n[2] Yang, E., Shen, L., Wang, Z., Liu, S., Guo, G., & Wang, X.. Data augmented flatness-aware gradient projection for continual learning. In ICCV, 2023.\n\n[3] Tran Tung, L., Nguyen Van, V., Nguyen Hoang, P., & Than, K.. Sharpness and gradient aware minimization for memory-based continual learning. In SOICT, 2023.\n\n[4] Chen, Runhang, et al. \"Sharpness-aware gradient guidance for few-shot class-incremental learning.\" Knowledge-Based Systems (2024): 112030.\n\n[5] Mehta, S. V., Patil, D., Chandar, S., & Strubell, E. (2023). An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research, 24(214), 1-50.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**NOTE**: Table R1-R4 and Figure R1 are attached to the one-page PDF.\n> Weakness 1: Clarification of contributions.\n\nA: For the suggested works, they could further validate the importance of C-Flat on a general and stronger flatness-aware CL optimizer, we have cited them and the detailed discussion can be seen in our answer to Weakness 2.  \n\nHowever, all these works are focused on the zeroth order sharpness, i.e. SAM for CL [5], and the improved works are also designed for specific kind of CL approaches like GPM method [2], memory based method [3] or certain CL scenario [4], while wide local minima in [1] is another concept about the model parameter's similarity to the last task.\n\nAs a comparison, 1) we proposed a CL optimizer beyond zeroth order sharpness; 2) C-Flat can be generally applied to any CL methods including the aforementioned ones; 3) as can be seen in our reply to Weakness 4, thanks to the faster convergence speed, we still achieve optimal result with a few overhead compared with other SAM optimizer by performing C-Flat in periodical iterations.\n\nMoreover, for clear contributions, you also can refer to the Strengths 1 of **Reviewer p3Vh**; Strengths 1,2 of **Reviewer Tb4q** and Strengths 1,2 of **Reviewer 475M**. \n\n>Weakness 2: Improve related work.\n\nA: We have cited and discussed these works in Section 2 as follows:\n\nWide local minima was considered an important regularization in CL to enforce the similarity of important parameters learned from past tasks [1]. Sharpness-aware seeking for loss landscape flat minima starts to gain more attention in CL, especially SAM based zeroth order sharpness is well discussed. An investigation [5] proves SAM can help with addressing forgetting in CL, and [4] proposed a combined SAM for few shot CL.\n\nSAM is also used for boosting the performance of specific methods like DFGP [2] and FS-DGPM [9] designed for GPM. SAM-CL [3] series with loss term gradient alignment for memory-based CL. These efforts kicked off the study of flat minima in CL, however, zeroth-order sharpness may not be enough for flatter optimal [62]. Thus, flatness with a global optima and universal CL framework is further studied.\n\nAs for the connection and difference of C-Flat with various existing SAM methods, PLEASE SEE the beginning of Section 3 Method, where we have a very detailed theoretical explanation and analysis. \n\n>Weakness 3: Comparison with related work [1,2,3].\n\nA: The discussion is as follows,\n\ni) Note that C-Flat is an optimizer-level approach, thus, the performance gains on different CL-friendly optimizers should be compared rather than different CL methods. We have already done this in the initial manuscript, please see subsection 4.5/Table 2/Figure 5a/5b. And we further compared C-Flat with more sharpness-aware method, please see Table R4 and our response to Question 4 of Reviewer p3Vh.\n\nii) Note that the flatness function used in DFGP is rooted in FS-GPM. We have demonstrated in Table 2 that C-Flat outperforms this function of FS-GPM.\n\nTherefore, the comparison with some related work you mentioned is out of the scope of this work. However, we still performed evaluations on CPR even if it was not a baseline against which we should compare. C-Flat still presents tangible gains, the results as below,\n\n| Method | Accuracy ↑ | Forgetting ↓ |\n|:--:|:--:|:--:|\n| Rwalk† |57.84| 9.37 |\n| w/ CPR† | 63.66 | 7.69  |\n| w/ C-Flat | **64.73** (+1.07) | **4.79** (-2.9) |\n\n>Weakness 4: Concern on computational cost.\n\nActually, C-Flat excels in seeking well-generalized flat minima, which is demonstrated by the outstanding convergence speed (proved this in Figure 5a, and also see comments of Reviewer 475M). Second, C-Flat does not have to perform in every iteration, this means that repeated propagation can be significantly less. This suggests that C-Flat requires only a few epochs or iterations to converge to global optimum, like 50% of iterations or 60% of epochs (best result).\n\nThus, in practice, we do not need to maintain the same settings as the baseline, which significantly enhances the practicality. Also, note that C-Flat is even faster than SGD in some cases while maintaining high performance, this is surely what the CL community expects. Thus, benefiting from the good optimization nature of C-Flat, this concern is trivial.\n\nMoreover, for easy use of C-Flat, we provided a tier guideline which categorizes C-Flat into L1 to L3 level. PLEASE SEE Weakness 3 of Reviewer 475M and our response to it.\n\n>Weakness 5: Slight performance gains in Table 2.\n\nA: i) Since the FS term in FS-GPM already visits the typical zeroth-order sharpness, at this point, C-Flat partly reconfigures FS-GPM via operate the curvature (Eq.3 and Eq.4). This implies that the gains of C-Flat are founded on the top of the typical sharpness optimization used. Such an experimental setup was intended to reaffirm the versatility of C-Flat. Hence, merely noting that the boost in Table 2 is slight, we do not think it is justified.\nii) Since we set a fixed random seed during CL, the performance gain of C-Flat is fairly solid. This is rare enough for an optimizer-level effort. PLEASE SEE our response to Weakness 2 of Reviewer p3Vh and Table R1 about stability.\n\n>Question 1: About subtitle Gradient-based solution.\n\nA: Gradient-based solutions are a main group in CL, including shaping loss landscape, tempering the tug-of-war of gradient, and other learning dynamics [5, 20]. C-Flat seeks for stable CL by tuning the gradient to flat minima, which is quite different from most gradient-based CL methods. Therefore, we had this paragraph for comparison. We have added more context in line 80 to support the claim that gradient-based solutions focus on dynamic learning to overcome forgetting.\n\n>Question 2: Smooth loss curves after 80th epoch.\n \nA: The decay_steps was set to 80 for all methods, which indicates that the learning rate decay was performed after the 80th epoch. This caused a smooth loss curve.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NRqN0tiHjK", "reviewer": "Reviewer_Tb4q", "summary": "This paper proposes a novel method named C-Flat to mitigate catastrophic forgetting by optimizing for a flatter loss landscape. This method is described as a plug-and-play solution that can be easily integrated into a wide range of existing CL approaches. The paper argues that this approach not only stabilizes the learning process but also enhances the model's generalization across tasks by enabling it to find and utilize flatter minima. The authors demonstrate the effectiveness of their method when applied to different CL solutions on 3 different datasets.", "questions": "1. **Revision of Language**: The authors are encouraged to thoroughly revise the language of the manuscript to enhance readability and clarity. \n\n2. **Typo**: I believe there is a typo in line 98.\n\n3. **Average Boost**: I would like to ask the authors to report the average of the boost alongside the maximum boost in the results section. This could give a better understanding of the effectiveness of the proposed method.", "limitations": "The paper lacks a discrete limitation section or a discussion of the limitations in the conclusion section. I would like to ask the authors to include this in their paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. **Novelty and Effectiveness of C-Flat**: The introduction of C-Flat as a method that emphasizes flatness in the loss landscape to address catastrophic forgetting is innovative. The paper provides extensive experimental results showing that C-Flat improves performance across a variety of continual learning settings and benchmarks.\n\n2. **Theoretical Analysis**: The theoretical grounding provided for the convergence of the proposed loss is valuable and adds depth to the paper.\n\n3. **Visualizations**: The visualization of the loss landscape in Figure 4 effectively illustrates the impact of the proposed method, providing a clear comparative insight into how C-Flat modifies the learning dynamics.", "weaknesses": "1. **Generalization Claims Overstated**: The paper's assertion that it resolves the sensitivity/stability dilemma in CL is overly strong. While the proposed method shows promising results, it would be more accurate to state that it addresses rather than resolves these issues. (see line 96)\n\n2. **Theoretical Section Structure**: The structure of the theoretical analysis section is confusing. It would benefit from a more organized presentation, starting with assumptions, followed by theorems, and a detailed proof. This would enhance the readability and academic rigor of the paper. The assumptions (such as the differentiability of the loss) should be stated first in an assumption environment. Then the theorem or proposition should be stated (for example: Given the assumption above, when <conditions>, the convergence of <loss> is guaranteed) followed by a proof environment. clumping several steps in one equation (Eq. 8) is confusing and should be avoided for clarity. \n\n3.**Oragnization**: From the beginning of line 141 to the end of Eq. 10 seems disconnected from the rest of the paper. The authors should provide context for what they are proposing and explain why the proposed content matters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: Overly strong claims.\n\nA: We appreciate your suggestion! We have revised the manuscript in line 96 to temper these statements.\n\n> Weakness 2: More clear structure for the method. \n\nA: We did shorten that part a lot into one paragraph than a better math environment due to the page limit. We have rewritten that part with highlighted 'assumption' and 'condition' and attached a detailed proof in the appendix. Many thanks for the patient explanation.\n\n> Weakness 3: Highlight the context of Eq.10\n\nA: Thanks for pointing it out. We intended to show some mathematical properties, i.e. the connection of C-Flat to Hessian matrix. Indeed, it does look disconnected. We have added a highlighted \"Upper Bound\" at the beginning of line 141 and some context about this property at the end of line 144 to make it clear.\n\n> Question 1: Improve readability and clarity.\n\nA: We have polished the manuscript again to improve readability and clarity, and fixed minor errors, like errors in line 38,40 and 98.\n\n> Question 2: Report average boost.\n\nA: We have updated maximum boost/average boost in Table 1 of the main paper as below,\n\n| Gains         | B0_Inc5 | B0_Inc10 | B0_Inc20 | B50_Inc10 | B50_Inc25 | B0_Inc40 |\n|---------------|---------|----------|----------|-----------|-----------|----------|\n| Maximum Boost | +2.34%  | +1.07%    | +1.89%   | +2.06%    | +3.03%    | +1.64%   |\n| Average Boost | +1.04%  | +0.66%   | +1.34%   | +0.62%    | +0.90%    | +0.81%   |\n\n> Limitation 1: Discuss limitations.\n\nA: C-Flat potentially has the following limitations. First, all experiments have not yet been validated on pre-training model (PTM)-based CL method. In the era of PTM, exploring the collaborative mechanisms between the proposed method and PTM is essential for advancing CL, which can be a promising future work. Second, C-Flat is under-explored in the transformer-based model, and this architecture is prevalent in the foundation model, which is remaining for future work. As above, we will update this discussion in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NbpHvhbtag", "reviewer": "Reviewer_475M", "summary": "This paper points out that the current sharpness in continual learning (CL) tends to optimize towards a suboptimal space rather than achieving a global solution for continuous tasks. The proposed C-Flat, smoothly migrates to the global optimal of the joint knowledge space of the current and next tasks, thereby promoting CL. The idea is an intriguing one and has the potential to make a significant contribution to the CL community.", "questions": "All experiments are conducted on the complicated Class IL scenarios and consistently achieve improvements. However, still wondering how much the larger domain gap affects the stability of C-Flat. Although Class IL already has fairly complex data distribution changes for CL.\n\nIn addition, please answer WEAKNESS i, and ii.", "limitations": "Please see my comments in WEAKNESS and QUESTIONS.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) the paper is well-structured and presented, compelling narrative. The benefits and motivation generally flowed well. To their credit, the authors provide a theoretical analysis. \n\n2) This paper investigates the relationship between flatness, loss landscape, and CL from an optimization view, and studies the limitation of sharpness applied in current works (e.g., FS-DGPM/F2M etc.), effectively complementing the current body of research. \n\n3) The authors unify the three preceding aspects, and demonstrate the mechanism of loss landscape flatness on several categories of CL methods. The authors show the necessity of a flatter loss landscape for CL cuz favors global optimal during CL. Then, their optimization framework can be plugged into any CL method. Such an approach seems to be non-trivial. Third, this work may be a basic toolkit for the CL community.", "weaknesses": "1) The authors seem made a non-trivial effort in the exploration of the sharpness-aware approach in CL. However, they should elaborate more on the differences with the current CL methods in INTRODUCTION to highlight the contribution of the manuscript, although I note that they did so in RELATED WORK.\n\n2) In Equation 3 and Equation 4, I noticed that C-Flat additionally decreases the curvature around local minima. Does this mean lower loss or less forgetting for old tasks in the next stage? \n\n3) Their method shows an outstanding convergence speed (Figure 5), thus, in practice, the overhead is acceptable (close to SGD, SAM in some cases). Perhaps the authors could put more emphasis on practicality in SECTION 4.6, which could be meaningful.\n\n4) A few terms (e.g., SAM, Figure 1, etc.) lack citations, which may hinder reading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: Improve INTRODUCTION.\n\nA: We have done this for the Introduction section to highlight contributions, including highlighting our CL-friendly optimizers, and also discussing how flatness-related works in current CL (flatness in some data/bases/projection, etc).\n\n> Weakness 2: The relationship between low curvature and less forgetting.\n\nA: Exactly, one of our contributions is to prove the positive effect of low curvature on overcoming forgetting. Intuitively, we visualized the change in loss and forgetting of old tasks during CL. Figure R1 in one page shows the lower loss or less forgetting (red line) for old tasks during CL. This is an enlightening finding.\n\n> Weakness 3: Discuss the practicality.\n\nA: To discuss practicality better, we provided a tier guideline, which categorizes C-Flat into L1 to L3 levels, as shown below,\n\n| Level | Version | Speed                           | Performance (SGD/SAM) |\n|:-----:|:-------:|:-------------------------------:|:---------------------:|\n|  L1   |   Low   |    SGD > SAM > **C-Flat**       |    +2.39% / +1.91%    |\n|  L2   | Medium  |    SGD > **C-Flat** > SAM       |    +1.52% / +1.04%    |\n|  L3   |  High   |      **C-Flat** > SGD           |    +1.51% / +1.03%    |\n\nAs the table above shows, L1 denotes the low-speed version of C-Flat, with a slightly lower speed than SAM and the best performance; L2 follows next; L3 denotes the high-speed version of C-Flat, with a faster speed than SGD and a performance close to L2. We have updated this in the revision.\n\n> Weakness 4: Fix misc.\n\nA: We have fixed them in our manuscript, including citations, minor typos, etc.\n\n> Question 1: The robustness of C-Flat on larger domain gap.\n\nA: As you mentioned, the performance in the Class IL scenario has already demonstrated the robustness of C-Flat given the fairly complex domain shifts. However, we would still be happy to conduct a further test on C-Flat to ensure its reliability. As shown in Table below, we trained on ImageNet21K and then evaluated CL performance on IN-R/ObjNet. Unlike CIFAR, both datasets are acknowledged to have a large domain gap with ImageNet. The Table below shows that C-Flat maintains stability in more difficult cases.\n\n|   Method   | IN-R B0_Inc20   | ObjNet B0_Inc10  |\n|:----------:|:---------------:|:---------------:|\n|   iCaRL    |      72.13      |      48.06      |\n| w/ C-Flat  |  **72.92** (+0.79)  |  **49.59** (+1.53)  |\n|   MEMO     |      70.96      |      56.22      |\n| w/ C-Flat  |  **71.69** (+0.73)  |  **56.50** (+0.28)  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable comments and appreciation of our strengths, e.g., 

+ well-presented and easy to follow (**Reviewer p3Vh/PuVp/Tb4q/475M**); 
+ originality and nice novelty (**Reviewer p3Vh/Tb4q/475M**); 
+ good generalizability for CL (**Reviewer p3Vh/ PuVp/ Tb4q/475M**). 
+ well-supported theoretical and grounded work (**Reviewer Tb4q/475M**).

All suggestions are seriously considered in our rebuttal and we have carefully revised the manuscript to address the concerns. 

**NOTE**: Table R1-R4 and Figure R1 are attached to the one-page PDF.

Meta-review: This paper proposes C-Flat, a novel optimization method for continual learning (CL) that aims to find flatter minima in the loss landscape to mitigate catastrophic forgetting. The method is presented as a plug-and-play solution that can be integrated into various existing CL approaches. The authors also demonstrate that their method results in a performance boost in a variety of cases.

The review process highlighted a number of weaknesses such as not using multiple seeds for calculating the result significance but these were mostly addressed in the rebuttal resulting in a paper that the AC is happy to recommend Accept.
Decision: Accept (poster)