Title: Make Continual Learning Stronger via C-Flat
Venue: NeurIPS 2024
Accepted: yes
Keywords: Continual Learning

Incremental Learning
Abstract: How to balance the learning ’sensitivity-stability’ upon new task training and memory preserving is critical in CL to resolve catastrophic forgetting. Improving model generalization ability within each learning phase is one solution to help CL learning overcome the gap in the joint knowledge space. Zeroth-order loss landscape sharpness-aware minimization is a strong training regime improving model generalization in transfer learning compared with optimizer like SGD. It has also been introduced into CL to improve memory representation or learning efficiency. However, zeroth-order sharpness alone could favors sharper over flatter minima in certain scenarios, leading to a rather sensitive minima rather than a global optima. To further enhance learning stability, we propose a Continual Flatness (C-Flat) method featuring a flatter loss landscape tailored for CL. C-Flat could be easily called with only one line of code and is plug-and-play to any CL methods. A general framework of C-Flat applied to all CL categories and a thorough comparison with loss minima optimizer and flat minima based CL approaches is presented in this paper, showing that our method can boost CL performance in almost all cases. Code is available at https://github.com/WanNaa/C-Flat.

Make Continual Learning Stronger via C-Flat

Abstract

How to balance the learning 'sensitivity-stability' upon new task training and memory preserving is critical in CL to resolve catastrophic forgetting. Improving model generalization ability within each learning phase is one solution to help CL learning overcome the gap in the joint knowledge space. Zeroth-order loss landscape sharpness-aware minimization is a strong training regime improving model generalization in transfer learning compared with optimizer like SGD. It has also been introduced into CL to improve memory representation or learning efficiency. However, zeroth-order sharpness alone could favors sharper over flatter minima in certain scenarios, leading to a rather sensitive minima rather than a global optima. To further enhance learning stability, we propose a C ontinual Flat ness ( C-Flat ) method featuring a flatter loss landscape tailored for CL. C-Flat could be easily called with only one line of code and is plug-and-play to any CL methods. A general framework of C-Flat applied to all CL categories and a thorough comparison with loss minima optimizer and flat minima based CL approaches is presented in this paper, showing that our method can boost CL performance in almost all cases. Code is available at https://github.com/WanNaa/C-Flat .

C-Flat : just a line of code suffices for its utilization.

1 Introduction

Why study Continual Learning (CL)? CL is generally acknowledged as a necessary attribute for Artificial General Intelligence (AGI) [22, 55, 40, 67]. In the open world, CL holds the potential for substantial benefits across many applications: e.g. vision model needs to learn a growing image set [17, 61, 62], or, embodied model needs to incrementally add skills to their repertoire [12].

Challenges. A good CL model is expected to keep the memory of all seen tasks upon learning new knowledge [22]. However, due to the limited access to previous data, the learning phase is naturally sensitive to the current task, hence resulting in a major challenge in CL called catastrophic forgetting [9], which refers to the drastic performance drop on past knowledge after learning new knowledge. This learning sensitivity-stability dilemma is critical in CL, requiring model with strong generalization ability [16] to overcome the knowledge gaps between sequentially arriving tasks.

Current solutions. A series of works [43, 44, 33, 25] are proposed to improve learning stability by extending data space with dedicated selected and stored exemplars from old tasks, or frozen some network blocks or layers that are strongly related to previous knowledge [68, 24, 69, 57, 24].

Another group of works seeks to preserve model generalization with regularisation onto the training procedure itself [32, 18, 31]. Diverse weight [45, 28, 2] or gradient alignment [22, 9, 35, 26] strategies are designed to encourage the training to efficiently extracting features for the current data space without forgetting.

Loss landscape sharpness optimization [23, 19, 65, 70] as an efficient training regime for model generalization starts to gain attentions [27, 63]. Ordinary loss minima based optimizer like SGD can easily lead to suboptimal results [4, 37, 13]. To prevent this, zeroth-order sharpness-aware minimization seeking neighborhood-flat minima [20] has been proven a strong optimizer to improve model generalization ability, especially in transferring learning tasks. It is also introduced into some CL works [49, 30] with dedicated designs to improve old knowledge representation or fewshot learning efficiency. However, given the limited application scenarios[10, 49], the zeroth-order sharpness used in the current work is proved to favor sharper minima than a flat solution [70]. It means zeroth-order only can still lead to a fast gradient descent to the suboptimal in new data space than a more generalizable result for the joint old and new knowledge space.

Our solution. Inspired by these works, a beyond zeroth-order sharpness continual optimization method is proposed as demonstrated in 1, where loss landscape flatness is emphasized to strengthen model generalization ability. Thus, the model can always converge to a flat minima in each phase, and then smoothly migrate to the global optimal of the joint knowledge space of the current and next tasks, and hence resolve the catastrophic forgetting in CL. We dub this method C ontinual Flat ness (C-Flat or C♭ ) Moreover, C-Flat is a general method that can be easily plug-and-play into any CL approach with only one line of code, to improve CL.

Contribution. A simple and flexible CL-friendly optimization method C-Flat is proposed, which Makes Continual Learning Stronger.

A framework of C-Flat covering diverse CL method categories is demonstrated. Experiment results prove that Flatter is Better in nearly all cases.

To the best of our knowledge, this work is the first to conduct a thorough comparison of CL approaches with loss landscape aware optimization, and thus can serve as a baseline in CL.

2 Related work

Continual learning methods roughly are categorized into three groups: Memory-based methods write experience in memory to alleviate forgetting. Some work [43, 44, 25, 51] design different sampling strategies to establish limited budgets in a memory buffer for rehearsal. However, these methods require access to raw past data, which is discouraged in practice due to privacy concerns. Instead, recently a series of works [10, 34, 46, 33, 50] elaborately construct special subspace of old tasks as the memory. Regularization-based methods aim to realize consolidation of the previous knowledge by introducing additional regularization terms in the loss function. Some works [32, 29, 6] enforce the important weights in the parameter space [45, 28, 2], feature representations [5, 21], or the logits outputs [32, 42] of the current model function to be close to that of the old one. Expansionbased methods dedicate different incremental model structures towards each task to minimize forgetting [68, 39]. Some work [48, 24, 60] exploit modular network architectures (dynamically extending extra components [57, 69], or freeze partial parameters [36, 1]) to overcome forgetting. Trivially, methods in this category implicitly shift the burden of storing numerous raw data into the retention of model [68].

Gradient-based solutions are a main group in CL, including shaping loss landscape, tempering the tug-of-war of gradient, and other learning dynamics [22, 9, 41]. One promising solution is to modify the gradients of different tasks and hence overcome forgetting [7, 38], e.g., aligning the gradients of current and old one [15, 18], or, learning more efficient in the case of conflicting objectives [47, 56, 14]. Other solutions [10, 49] focus on characterizing the generalization from the loss landscape perspectives to improve CL performance and yet are rarely explored.

𝜃

𝜃′

𝛩

𝐿𝑛𝑒𝑤

(a) Direct Tuning

𝜃

𝜃′

𝛩

𝐿𝑛𝑒𝑤

(b) Regularization

𝜃

𝜃′

𝛩

𝐿𝑛𝑒𝑤

𝜌

(c) C-Flat

loss of old tasks

loss of new tasks

tolerance line

perturb radius

well learnt

ordinary performance

catastrophic forgetting

Sharpness minimization in CL Many recent works [23, 19, 4] are proposed to optimize neural networks in standard training scenarios towards flat minima. Wide local minima were considered an important regularization in CL to enforce the similarity of important parameters learned from past tasks [6]. Sharpness-aware seeking for loss landscape flat minima starts to gain more attention in CL, especially SAM based zeroth order sharpness is well discussed. An investigation [41] proves SAM can help with addressing forgetting in CL, and [8] proposed a combined SAM for few-shot CL. SAM is also used for boosting the performance of specific methods like DFGP [58] and FS-DGPM [10] designed for GPM. SAM-CL [52] series with loss term gradient alignment for memory-based CL. These efforts kicked off the study of flat minima in CL, however, zeroth-order sharpness may not be enough for flatter optimal [70]. Thus, flatness with a global optima and universal CL framework is further studied.

3 Method

Our solution addresses the learning sensitivity-stability dilemma in CL by improving model generalization for joint learning knowledge obtained from different catalogues domains or tasks. Moreover, a general but stronger optimization method enhanced by the latest gradient landscape flatness is proposed as a 'plug-and-play' tool for any CL approach.

Loss landscape flatness. Let B ( θ, ρ ) = { θ ′ : ∥ θ ′ -θ ∥ < ρ } denotes the neighborhood of θ with radius ρ > 0 in the Euclidean space Θ ⊂ R d , the zeroth-order sharpness at point θ is commonly defined by the maximal training loss difference within its neighborhood B ( θ, ρ ) :

where l S ( θ ) denotes the loss of an arbitrary model with parameter θ on any dataset S with an oracle loss function l ( · ) . The zeroth-order sharpness R 0 ρ ( θ ) regularization can be directly applied to restrain the maximal neighborhood training loss:

However, for some fixed ρ , local minima with a lower loss does not always have a lower major hessian eigenvalue [70], which equals to the neighborhood curvature. It means that zeroth-order sharpness optimizer may goes to a sharper suboptimal than to the direction of a flatter global optimal with better generalization ability.

Recently, first-order gradient landscape flatness is proposed as a measurement of the maximal neighborhood gradient norm, which reflects landscape curvature, to better describe the smoothness of the loss landscape:

Unlike zeroth-order sharpness that force the training converging to a local minimal, first-order flatness alone constraining on the neighborhood smoothness can not lead to an optimal with minimal loss. To maximize the generalization ability of loss landscape sharpness for continual learning task, we

propose a zeroth-first-order sharpness aware optimizer C-Flat for CL. Considering the data space, model or blocks to be trained are altered regarding the training phase and CL method, (as detailed in the next subsection), we define the the C-Flat loss as follows:

with the minimization objective:

where l R 0 ρ S ( θ ) is constructed to replace the original CL loss, while R 1 ρ ( θ ) further regularizes the smoothness of the neighborhood, and hyperparameter λ is to balance the influence of R 1 ρ as an additional regularization to loss function l . Hence, the local minima within a flat and smooth neighborhood is calculated for a generalized model possessing both old and new knowledge.

Optimization. In our work, the two regularization terms in the proposed C-Flat are resolved correspondingly in each iteration. Assuming the loss function l ( · ) is differentiable and bounded, the gradient of l R 0 ρ S at point θ T can be approximated by

And the gradient of the first-order flatness regularization ∇ R 1 ρ ( θ T ) can be approximated by

The optimization is detailed in Appendix algorithm 1. Note that ∇ l is the gradient of l with respect to θ through this paper, and instead of the expensive computation of Hessian matrix ∇ 2 l , Hessian-vector product calculation is used in our algorithm, where the time and especially space complexity are greatly reduced to o ( n ) using 1 forward and 1 backward propagation. Thus, the overall calculation in one iteration takes 2 forward and 4 backward propagation in total.

Theoretical analysis. Given R 0 ρ ( θ ) measuring the maximal limit of the training loss difference, the first-order flatness is its upper bound by nature. Denoting θ + ϵ ∈ B ( θ, ρ ) the local maximum point, a constant ϵ ∗ ∈ [0 , ϵ ] exists according to the mean value theorem that

Assuming the loss function is twice differentiable, bounded by M , obeys the triangle inequality, its gradient has bounded variance σ 2 , and both the loss function and its second-order gradient are β -Lipschitz smooth, we can prove that, according to [3, 63], C-Flat converges in all tasks with η ≤ 1 /β, ρ ≤ 1 / 4 β , and η T i = η/ √ i , ρ T i = ρ/ 4 √ i for epoch i in any task T ,

where n T is the total iteration numbers of task T , and b is the batch size.

Upper Bound. Let ∇ 2 l S ( θ ∗ ) denotes the Hessian matrix at local minimum θ ∗ , its maximal eigenvalue λ max ( ∇ 2 l S ( θ ∗ )) is a proper measure of the landscape curvature. The first-order flatness is proven to be related to the maximal eigenvalue of the Hessian matrix as R 1 ρ ( θ ∗ ) = ρ 2 · λ max ( ∇ 2 l S ( θ ∗ )) , thus the C-Flat regularization can also be used as an index of model generalization ability, with the following upper bound:

3.1 A Unified CL Framework Using C-Flat

This subsection presents an unified CL framework using C-Flat with applications covering Class Incremental Learning (CIL) approaches. To keep focus, the scope of our study is limited in CIL task, which is the most intractable CL scenarios that seek for a lifelong learning model for sequentially arriving class-agnostic data. Most CIL approaches belong to three main families, Memory-based, Regularization-based and Expansion-based methods.

Memory-based methods store samples from the previous phases within the memory limit, or produce pseudo-samples by generative approaches to extend the current training data space, thus a memory replay strategy is used to preserve the seen class features with ˆ S T = S T ∪ Sample t<T . iCaRL is one of the early works. It learns classifiers and a feature representation simultaneously, and preserves the first few most representative exemplars according to Nearest-Mean-of-Exemplars Classification. Thus a loss function l ˆ S T = l CE ˆ S T + l KL ˆ S T combining both cross entropy for the current task and a knowledge distillation loss for the previous classes is introduced to balance the learning sensitivity and model generalization to the previous tasks.

Solution: for memory-based method, including, the C-Flat can be easily applied to these scenarios by reconstruct the oracle loss function with its zeroth- and first-order flatness measurement as eq. 11, and trained with algorithm 1 using data set extended with the previous exemplars.

Regularization-based methods seeks for a apply regularization on the model develop to preserver the learnt knowledge. For instance, WA introduces weight aligning on the final inference part to balance the old and new classes. Denoting ϕ the feature learning layers of the model, ψ = [ ψ old , ψ new ] the decision head for all classes consisting of two branches for the old and new seen data classes, the output is corrected to f ( x ) = [ ψ old ( ϕ ( x )) , γ · ψ new ( ϕ ( x ))] , where γ is the fraction of average norm of ψ old , ψ new of all classes.

Gradient Projection Memory (GPM) is another main regularization based group, introducing explicit align the gradient direction to new knowledge learning. It stores a minimum set of bases of the Core Gradient Space as Gradient Projection Memory, thus gradient steps are only taken in its orthogonal direction to learn the new features without forgetting the core information from the previous phases. FS-DGPM further improves this method by updating model parameter along the aligned orthogonal gradient at the zeroth-order sharpness minima in dynamic GMP space.

Solution: for regularization-based approaches, the same plug-and-play strategy can be used to reconstruct the loss function as eq. 11, and optimized by algorithm 1.

An alternative solution for the gradient-based methods like GPM and the improved FS-DGPM, is to introduce C-Flat optimization at the gradient alignment stage, so that the orthogonal gradient at a flatter minima is used to ensure that the training can cross over the knowledge gap between different data categories. The implementation of our C-Flat-GPM is detailed in Appendix algorithm 2.

Expansion-based methods explicitly construct task-specific parameters to resolve the new class learning and inference problem. For instance, Memory-efficient Expandable Model (Memo) decomposes the embedding module into deep layers and shallow layers that ϕ = ϕ f ( ϕ g ) , where ϕ f , ϕ g correspond to the specialized block for different tasks and the generalized block that can be shared during training phases. An additional block ϕ new f is added to the deep layers for specified feature extraction for the new classes, where the model can be reconstructed as f T = ψ T ([ ϕ T -1 f ( ϕ g ) , ϕ new f ( ϕ g )]) . Thus the new model training is focusing on the task specified component while the shared shallow layers are frozen with loss function l Memo ˆ S T = l CE ˆ S T ( ψ T ([ ϕ T -1 f ( ϕ g ) , ϕ new f ( ϕ g )]))) .

Foster uses KL-based loss function to regularize the three combinations of old and new blocks for a stable performance on the previous data. It also introduces an effective redundant parameters and feature pruning strategy to maintain the single backbone model using knowledge distillation. DER follows the same framework, and introduces an auxiliary classifier and related loss item to encourage the model to learn diverse and discriminate features for novel concepts.

Solution: for expansion-based approaches, the plug-and-play strategy is still available. The CFlat loss can be reformed with the reconstructed model as eq. 12. Thus C-Flat optimization using algorithm 1 is applied onto the first stage, where the new constructed block are optimized, while the generalized blocks are kept frozen. The final model is obtained after post-processing.

To conclude , C-Flat can be easily applied to any CL method with reconstructed loss function, and thus trained with the corresponding optimize as shown in algorithm 1. Dedicated design using C-Flat like for the GPM family is also possible wherever flat minima is required.

4 Analysis

4.1 Experimental Setup

Datasets. We evaluate the performance on CIFAR-100, ImageNet-100 and Tiny-ImageNet. Adherence to [66, 67], the random seed for class-order shuffling is fixed at 1993. Subsequently, we follow two typical class splits in CIL: (i) Divide all ∥ Y b ∥ classes equally into B phases, denoted as B0_Inc y ; (ii) Treat half of the total classes as initial phases, followed by an equally division of the remaining classes into incremental phases, denoted as B50_Inc y . In both settings, y denotes that learns y new classes per task.

Baselines. To evaluate the efficacy of our method, we plug it into 7 top-performing baselines across each CL category: Replay [44] and iCaRL [43] are classical replay-based methods, using raw data as memory cells. PODNet [11] is akin to iCaRL, incorporating knowledge distillation to constraint the logits of pooled representations. WA [64] corrects prediction bias via regularizing discrimination and fairness. DER [57], FOSTER [54] and MEMO [68] are network expansion

Trace: 670.36

λmax: 67.48

(a) MEMO Epoch: 50

Trace: 429.90

λmax: 28.11

(b) MEMO w/ C-Flat

Trace: 321.36

λmax: 21.07

(c) MEMO Epoch: 150

Trace: 97.36

λmax: 6.25

(d) MEMO w/ C-Flat

methods, dedicate modular architectures towards each task by extending sub-network or freezing partial parameters. The aforementioned methods span three categories in CL [9, 53]: Memory-based methods, Regularization-based methods and Expansion-based methods.

Network and training details. For a given dataset, we study all methods using the same network architecture following repo [66, 67], i.e. ResNet-32 for CIFAR and ResNet-18 for ImageNet. If not specified otherwise, the hyper-parameters for all models adhere to the settings in the open-source library [66, 67]. Each task are initialized with the same ρ and η , which drops with iterations according to the scheduler from [70]. To ensure a fair comparison, all models are trained with a vanilla-SGD optimizer [71]. And the proposed method is plugged into the SGD.

4.2 Make Continual Learning Stronger

Table 1 empirically demonstrates the superiority of our method: Makes Continual Learning Stronger. In this experiment, we plug C-Flat into 7 state-of-the-art methods that cover the full range of CL methods. From Table 1, we observe that (i) C-Flat presents consistent outperformance on all models, spanning Memory-based methods, Regularization-based methods, and Expansion-based methods. This superiority is indicative of the plug-and-play feature inherent in our method, allowing effortless installation with all sorts of CL paradigms. (ii) Across multiple benchmark datasets, including CIFAR-100, ImageNet-100, and Tiny-ImageNet, C-Flat exhibits consistent improvement. This underscores its generalization ability and effectiveness against diverse data distributions. (iii) CFlat presents consistent boosting across multiple incremental scenarios, encompassing B0_Inc5, B0_Inc10, B0_Inc20, B50_Inc10, B50_Inc25, and B0_Inc40. This consistent boosting reaffirms robustness of C-Flat for various CL scenarios. To sum up, C-Flat advances baselines across each CL category, serves as a valuable addition to CL, offering a versatile solution that can complement existing methods.

4.3 Hessian Eigenvalues and Hessian Traces

Hessian eigenvalues. Equation 10 delineates the connection between fist-order flatness and Hessian eigenvalues in CL. Broadly, Hessian eigenvalues serve as a metric for assessing the flatness of a function. Thus we report Hessian eigenvalue distributions in Figure 2 for empirical analysis. As shown in Figure 2, models trained with vanilla-SGD exhibit higher maximal Hessian eigenvalues (67.48/21.07 at epochs 50/150 in Figure 2a and Figure 2c), while our method induces a significant drop in Hessian eigenvalues to 28.11/6.25 at epochs 50/150 in Figure 2b and Figure 2d) during CL, leading to flatter minima. Consequently, the performance of CL is tangibly enhanced.

Hessian traces. We calculate the empirical Fisher information matrix as an estimation of the Hessian and leverage the trace of this to quantify the flatness of the approximation loss at the convergence point. As depicted in Figure 2, we observe that a substantial reduction in the Hessian trace when employing our method compared with vanilla-SGD (670.36/321.36 drops to 429.90/97.36 at epochs 50/150 in Figure 2b and Figure 2d). This observation suggests that our method induces a flatter minimum. These findings not only align with but also substantiate the theoretical insights presented in the methodology section.

Replay

Replay + C-Flat

(a) Replay w/ C-Flat

WA

WA +C-Flat

(b) WA w/ C-Flat

MEMO

MEMO +C-Flat

(c) MEMO w/ C-Flat)

0.3

9.3

21

0.0

0.5

tD

0.0

-0.2

1'0-

=0.2

0.2

Eo

55

65

75

Replay

iCaRL

WA

PODNet

DER

FOSTER

MEMO

Average accuracy (%)

w/ zeroth-order

w/ C-Flat

(a) B0_Inc10

55

65

75

Replay

iCaRL

WA

PODNet

DER

FOSTER

MEMO

Average accuracy (%)

w/ zeroth-order

w/ C-Flat

(b) B0_Inc20

30

40

50

0

40

80

120

160

Average accuracy (%)

Epoch

SGD

SAM

GAM

C-Flat

(a) Convergence analysis

63.06

63.54

64.88

65.05

65.16

64.58

65.45

65.02

64.57

62

64

66

0

1000

2000

3000

4000

5000

Average accuracy (%)

Running time (s)

GAM

SGD

SAM

C-Flat (100e)

C-Flat

(b) Training time

4.4 Visualization of Landscapes

More intuitively, we present a detailed visualization of landscape. PyHessian [59] is used to draw the loss landscape of models. To simplify, we choose one typical method from each category of CL methods (Replay, Wa, MEMO) for testing. Figure 3 clearly illustrates that, by applying C-Flat, the loss landscape becomes much flatter than that of the vanilla method. This trend consistently holds across various categories of CL methods, providing strong empirical support for C-Flat, and confirms our intuition.

4.5 Revisiting Zeroth-order Flatness

that flatter is better for CL.

Former work FS-DGPM [10] regulates the gradient direction with flat minima to promote CL. The FS (Flattening Sharpness) term derived from FS-DGPM is a typical zeroth-order flatness. We revisit the FS-DGPM series (including La/FS-GPM, DGPM, La/FS-DGPM) [10, 46] to evaluate performance using C-Flat instead of FS (see aigorithm 2). Table 2 yields two conclusions: (i) C-Flat boosts the GPM [46] baseline as a pluggable regularization term. This not only extends the frontiers of CL methods, incorporating gradient-based solutions, but also reaffirms the remarkable versatility of C-Flat. (ii) Throughout all series of FS-DGPM, C-Flat seamlessly supersedes FS and achieves significantly better performance. This indicates that C-Flat consistently exceeds zeroth-order sharpness. Hence, reconfirming that C-Flat is indeed a simple yet potential CL method that deserves to be widely spread within the CL community.

Limited work [10, 49] proved that the zeroth-order sharpness leads to flat minima boosted CL. Here, we employ a zeroth-order optimizer [19] instead of vanilla-SGD to verify the performance of C-Flat. As shown in Figure 4, CFlat (purple line) stably outperforms the zeroth-order sharpness (blue line) on all baselines. We empirically demonstrated

60

65

70

75

0

0.02

0.05

0.1

0.2

0.3

Average accuracy (%)

λ

WA

Replay

MEMO

(a)

λ

on CL methods

60

65

70

75

0

0.02

0.05

0.1

0.2

0.3

Average accuracy (%)

ρ

WA

Replay

MEMO

(b)

ρ

on CL methods

68

70

72

0.02

0.05

0.1

0.2

0.3

Average accuracy (%)

ρ

SAM

GAM

C-Flat

(c)

Effects of

ρ

68

70

72

[0.02, 0.04]

[0.02, 0.10]

[0.02, 0.20]

Average accuracy (%)

ρ_scheduler

SAM

GAM

C-Flat

(d)

Effects of

ρ

scheduler

4.6 Computation Overhead

To assess the efficiency of C-Flat, we provides a thorough analysis from the convergence speed and running time with CIFAR-100/B0_Inc20 on Replay. As shown in Figure 5, C-Flat is compared with SGD and other flatness-aware optimiziters. We train C-Flat optimizers on CL benchmarks with 20%, 50%, 100% of iterations and approximately 60% of epochs, while holding the other optimizers at 100%. Figure 5a first shows that C-Flat converges fastest and has the highest accuracy (purple line), meaning few iterations/epochs with C-Flat is enough to improve CL. Figure 5b shows i) Compared with SGD, with only 20% of iterations and 60% of epochs (pink line) using C-Flat, CL performance is improved using slightly less time; ii) C-Flat surpasses GAM with similar time as SAM when setting the iterations/epochs ratio to 50%/60%; iii) Models trained with C-Flat for 100 epochs outperform those trained with other optimizers for 170 epochs. To sum up, we show that C-Flat outperforms current optimizers with fewer iterations and epochs. This indicates the efficiency of C-Flat.

To discuss practicality better, we provided a tier guideline, which categorizes C-Flat into L1 to L3 levels, as shown in Table 3, L1 denotes the low-speed version of C-Flat, with a slightly lower speed than SAM and the best performance; L2 follows next; L3 denotes the highspeed version of C-Flat, with a faster speed than SGD and a performance close to L2.

4.7 Ablation Study

We perform ablation study in two cases: (i) the influence of λ and ρ on different CL methods; (ii) the influence of ρ and its scheduler on different optimizers.

We first present the performance of C-Flat with varying λ and ρ . As described in Eq. 13, λ controls the strength of the C-Flat penalty (when λ is equal to 0, this means that first-order flatness is not implemented). As shown in Figure 6a, compared with vanilla optimizer, C-Flat shows remarkable improvement with varying λ . Moreover, ρ controls the step length of gradient ascent. As shown in Figure 6b, C-Flat with ρ larger than 0 outperforms C-Flat without gradient ascent, showing that C-Flat benefits from the gradient ascent.

For each CL task T , same learning rate η T and neighborhood size ρ T initialization are used. By default, ρ T i ∈ [ ρ _ , ρ + ] is set as a constant, which decays with respect to the learning rate η T i ∈ [ η _ , η + ] by ρ T i = ρ _ + ( ρ + -ρ _ ) η + -η _ ( η T i -η _ ) . Figure 6c and Figure 6d present a comparison on ρ initialization and { ρ _ , ρ + } scheduler. C-Flat outperforms across various settings, and is not oversensitive to hyperparameters in a reasonable range.

4.8 Beyond Not-forgetting

As is known to all, forward, and in particular backward transfer, are the desirable conditions for CL [22]. Here, we thoroughly examine the performance of C-Flat in both aspects. Forward Transfer (FT) means better performance on each subsequent task. Backward Transfer (BT) means better

































































performance on previous tasks, when revisited. We count the performance of new and old tasks on several CL benchmarks before and after using C-Flat. As observed in Table 7, C-Flat consistently improves the learning performance of both new and old tasks. This observation indicates that C-Flat empowers these baselines with robust forward and backward transfer capabilities, that is learning a task should improve related tasks, both past and future. But, thus far, achieving a baseline that maintains perfect recall (by forgetting nothing) remains elusive. Should such a baseline emerge, C-Flat stands poised to empower it with potent backward transfer, potentially transcending the limitations of mere not-forgetting.

Moreover, one of our contributions is to prove the positive effect of low curvature on overcoming forgetting. Intuitively, we visualized the change in loss and forgetting of old tasks in CL. Figure 8 shows the lower loss or less forgetting (red line) for old tasks during CL. This is an enlightening finding.

5 Conclusion

This paper presents a versatile optimization framework, C-Flat, to confront forgetting. Empirical results demonstrate C-Flat's consistently outperform on all sorts of CL methods, showcasing its plug-and-play feature. Moreover, the exploration of Hessian eigenvalues and traces reaffirms the efficacy of C-Flat in inducing flatter minima to enhance CL. In essence, C-Flat emerges as a simple yet powerful addition to the CL toolkit, making continual learning stronger.

6 Acknowledgments

This work was supported in part by the Chunhui Cooperative Research Project from the Ministry of Education of China under Grand HZKY20220560, in part by the National Natural Science Foundation of China under Grant W2433165, and in part by the National Natural Science Foundation of Sichuan Province under Grant 2023YFWZ0009.

References

Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR , 2020.

Afra Feyza Akyürek, Ekin Akyürek, Derry Tanti Wijaya, and Jacob Andreas. Subspace regularizers for few-shot class incremental learning. ICLR , 2022.

Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In ICML , 2022.

Carlo Baldassi, Fabrizio Pittorino, and Riccardo Zecchina. Shaping the learning landscape in neural networks around wide flat minima. Proceedings of the National Academy of Sciences , 2020.

Prashant Bhat, Bahram Zonooz, and Elahe Arani. Task-aware information routing from common representation space in lifelong learning. ICLR , 2023.

Sungmin Cha, Hsiang Hsu, Taebaek Hwang, Flavio P Calmon, and Taesup Moon. Cpr: classifier-projection regularization for continual learning. ICLR , 2021.

Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420 , 2018.

Runhang Chen, Xiao-Yuan Jing, Fei Wu, and Haowen Chen. Sharpness-aware gradient guidance for few-shot class-incremental learning. Knowl. Based Syst. , 299:112030, 2024.

Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Danruo Deng, Guangyong Chen, Jianye Hao, Qiong Wang, and Pheng-Ann Heng. Flattening sharpness for dynamic gradient projection memory benefits continual learning. NeurIPS , 34, 2021.

Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In ECCV , 2020.

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. NeurIPS , 2022.

Yunshu Du, Wojciech M Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:1812.02224 , 2018.

Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics , pages 3762-3773. PMLR, 2020.

Tao Feng, Kaifan Ji, Ang Bian, Chang Liu, and Jianzhou Zhang. Identifying players in broadcast videos using graph convolutional network. Pattern Recognition , 124:108503, 2022.

Tao Feng, Mang Wang, and Hangjie Yuan. Overcoming catastrophic forgetting in incremental object detection via elastic response distillation. In CVPR , 2022.

Tao Feng, Hangjie Yuan, Mang Wang, Ziyuan Huang, Ang Bian, and Jianzhou Zhang. Progressive learning without forgetting. arXiv preprint arXiv:2211.15215 , 2022.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 , 2020.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In ICLR , 2021.

Qiankun Gao, Chen Zhao, Bernard Ghanem, and Jian Zhang. R-dfcil: Relation-guided representation learning for data-free class incremental learning. In ECCV , 2022.

Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences , 24(12):1028-1040, 2020.

Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. NeurIPS , 32, 2019.

Zhiyuan Hu, Yunsheng Li, Jiancheng Lyu, Dashan Gao, and Nuno Vasconcelos. Dense network expansion for class incremental learning. In CVPR , 2023.

Kishaan Jeeveswaran, Prashant Bhat, Bahram Zonooz, and Elahe Arani. Birt: Bio-inspired replay in vision transformers for continual learning. ICML , 2023.

Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren. Gradient-based editing of memory examples for online task-free continual learning. NeurIPS , 2021.

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 , 2016.

Do-Yeon Kim, Dong-Jun Han, Jun Seo, and Jaekyun Moon. Warping the space: Weight space rotation for class-incremental few-shot learning. In ICLR , 2022.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS , 2017.

Yajing Kong, Liu Liu, Huanhuan Chen, Janusz Kacprzyk, and Dacheng Tao. Overcoming catastrophic forgetting in continual learning by exploring eigenvalues of hessian matrix. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Tatsuya Konishi, Mori Kurokawa, Chihiro Ono, Zixuan Ke, Gyuhak Kim, and Bing Liu. Parameter-level soft-masking for continual learning. arXiv preprint arXiv:2306.14775 , 2023.

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. , 40(12):2935-2947, 2018.

Huiwei Lin, Baoquan Zhang, Shanshan Feng, Xutao Li, and Yunming Ye. Pcr: Proxy-based contrastive replay for online class-incremental continual learning. In CVPR , 2023.

Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning. arXiv preprint arXiv:2202.02931 , 2022.

Hao Liu and Huaping Liu. Continual learning with recursive gradient optimization. ICLR , 2022.

Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. In CVPR , 2021.

Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In CVPR , 2022.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. NeurIPS , 2017.

Aojun Lu, Tao Feng, Hangjie Yuan, Xiaotian Song, and Yanan Sun. Revisiting neural networks for continual learning: An architectural perspective. IJCAI , 2024.

Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. J. Mach. Learn. Res. , 24:214:1-214:50, 2023.

Youngmin Oh, Donghyeon Baek, and Bumsub Ham. Alife: Adaptive logit regularizer and feature replay for incremental semantic segmentation. NeurIPS , 2022.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR , 2017.

David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In NeurIPS , volume 32, 2019.

Tim GJ Rudner, Freddie Bickford Smith, Qixuan Feng, Yee Whye Teh, and Yarin Gal. Continual learning via sequential function-space variational inference. In ICML , 2022.

Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations , 2020.

Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. NeurIPS , 2018.

Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML , pages 4555-4564, 2018.

Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu. Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima. NeurIPS , 2021.

Wenju Sun, Qingyong Li, Jing Zhang, Wen Wang, and Yangli-ao Geng. Decoupling learning and remembering: A bilevel memory framework with knowledge projection for task-incremental learning. In CVPR , 2023.

Zhicheng Sun, Yadong Mu, and Gang Hua. Regularizing second-order influences for continual learning. In CVPR , 2023.

Lam Tran Tung, Viet Nguyen Van, Phi Nguyen Hoang, and Khoat Than. Sharpness and gradient aware minimization for memory-based continual learning. In Proceedings of the 12th International Symposium on Information and Communication Technology, SOICT 2023, Ho Chi Minh, Vietnam, December 7-8, 2023 , pages 189-196. ACM, 2023.

Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence , pages 1185-1197, 2022.

Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. In European conference on computer vision , pages 398-414, 2022.

Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487 , 2023.

Zirui Wang and Yulia Tsvetkov. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In Proceedings of the International Conference on Learning Representations (ICLR) , 2021.

Shipeng Yan, Jiangwei Xie, and Xuming He. DER: dynamically expandable representation for class incremental learning. In CVPR , pages 3014-3023, 2021.

Enneng Yang, Li Shen, Zhenyi Wang, Shiwei Liu, Guibing Guo, and Xingwei Wang. Data augmented flatness-aware gradient projection for continual learning. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 5607-5616. IEEE, 2023.

Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data) , pages 581-590. IEEE, 2020.

Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. In International Conference on Learning Representations , 2020.

Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, and Mingqian Tang. Rlip: Relational language-image pre-training for human-object interaction detection. In NeurIPS , 2022.

Hangjie Yuan, Shiwei Zhang, Xiang Wang, Samuel Albanie, Yining Pan, Tao Feng, Jianwen Jiang, Dong Ni, Yingya Zhang, and Deli Zhao. Rlipv2: Fast scaling of relational language-image pre-training. In ICCV , 2023.

Xingxuan Zhang, Renzhe Xu, Han Yu, Hao Zou, and Peng Cui. Gradient norm aware minimization seeks first-order flatness and improves generalization. In CVPR 2023 , pages 20247-20257, 2023.

Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In CVPR , 2020.

Qihuang Zhong, Liang Ding, Li Shen, Peng Mi, Juhua Liu, Bo Du, and Dacheng Tao. Improving sharpness-aware minimization with fisher mask for better generalization on language models. arXiv preprint arXiv:2210.05497 , 2022.

Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, and De-Chuan Zhan. Pycil: A python toolbox for classincremental learning, 2023.

Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep classincremental learning: A survey. arXiv preprint arXiv:2302.03648 , 2023.

Da-Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory-efficient class-incremental learning. ICLR , 2023.

Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self-sustaining representation expansion for non-exemplar class-incremental learning. In CVPR , 2022.

Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.08065 , 2022.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Machine Learning, Proceedings of the Twentieth International Conference (ICML) , 2003.

A Appendix

A.1 Overview

In this supplementary material, we first present more intuitive visualizations of C-Flat, elucidating the loss landscape from local viewpoint (Appendix A.5.1) and each task during CL (Appendix A.5.2). Next, we provide more details on the accuracy and runtime trade-offs of other CL methods with our C-Flat (Appendix A.6)

A.2 C-Flat

We summarize the pseudo code of C-Flat in algorithm 1.

Algorithm 1 C-Flat Optimization

Input: Training phase T , training data S T , model f T -1 with parameter θ T -1 from last phase if T > 1 , batch size b , oracle loss function l , learning rate η > 0 , neighborhood size ρ , trade-off coefficient λ , small constant ϵ .

Output: Model trained at the current time T with C-Flat.

Initialization:

if T=1: then Randomly Initialize parameter θ T =1 , η T =1 = η, ρ T =1 = ρ . else

Reconstruct the model and training set if necessary,

Initialize model parameter θ T according to the training strategy, like randomly initialization or θ T = θ T -1 in pre-trained model based approaches, η T = η, ρ T = ρ ,

Frozen part of the parameter if required.

end if

Optimization:

while θ T not converge, do

Sample batch B T of b random instances from S T

Compute batch's loss gradient g B T = ▽ l B T ( f T ( θ T ))

Approximate zeroth-order gradient: g 0 = ▽ l B T ( f T ( θ T + ϵ 0 ))

Compute hessian vector product: h B T = ▽ 2 l BT ( θ ) · ▽ l BT ( f T ( θ T )) ∥▽ l BT ( f T ( θ T )) ∥ 2 + ϵ

Update: Model parameter: θ T = θ T -η T ( g 0 + λg 1 ) ; Update training parameters η T , ρ T according to a scheduler that the values drop with iterations;

end while

Post-Processing on Model and Training data if required.

return Model f T with parameter θ T

A.3 C-Flat-GPM

We summarize the pseudo code of C-Flat for GPM family in algorithm 2.

A.4 Convergency Proof

Assumptions 1: the loss function is twice differentiable, bounded by M , with bounded variance σ 2 , and obeys the triangle inequality. Both the loss function and its second-order gradient are β -Lipschitz smooth. η ≤ 1 /β, ρ ≤ 1 / 4 β , and η T i = η/ √ i , ρ T i = ρ/ 4 √ i for epoch i in any task T .

Input: Training set ˆ S T , parameter θ T = θ T -1 , loss l , learning rate η 1 , η 2 , basis matrix M and significance Λ from replay buffer.

while θ T not converge, do

Sample batch B T

Compute perturbation ϵ c using C-Flat optimization

Update basis significance: Λ = Λ -η 1 · ▽ Λ l B T ( θ T + ϵ c )

Update model parameter:

θ

T

Update M and replay buffer.

end while

return Model parameter θ T

Claim 1: with Assumptions 1 , the convergency of zeroth-sharpness with batch size b is guaranteed [3] by

hence, the zeroth-order part of C-Flat is bounded:

Lemma 1: let ξ tr ( θ ) = l tr ( f T ( θ ) , f T ( θ ∗ )) , with Assumptions 1 , the first-order part is bounded by

Theorem 1: with Assumptions 1 , by combining the zeroth- and first-order parts, we can prove C-Flat converges in all tasks that,

A.5 More Visualizations of C-Flat

In this section, we present additional visualization of the loss landscape involving two cases using PyHessian [59]: (i) Changes in the loss landscape from localized viewpoints; (ii) Changes in the loss landscape across each task during CL.

A.5.1 Landscapes in a Local Viewpoint

First, we present a more detailed visualization through changes in the local region of the loss landscape. We set a minimal radius threshold. At this threshold, more detailed changes are displayed. Similarly, we choose three typical method (Replay [44], WA [64], MEMO [68]) from each category of CL methods for visualization. As shown in Fig. 9, in a tiny view, C-Flat contributes to a flatter surface, a change that more intuitively reveals the mechanism of C-Flat.

=

θ

-

η

2

·

(

I

-

M

Λ

M

)

l

T

B

T

(

θ

T

+

ϵ

c

)

(a) Replay

(b) WA

(c) MEMO

UT5

N

M5

4/P08

4V80

-NK

=,149

IL12

LILTI

SH

SH

400*

JHT

=l30

U

4rh

AH

=15

A.5.2 Landscapes Across Each Task

Second, we further visualize the loss landscape of more tasks using PyHessian for more intuitive explanations during CL. To simplify, we choose one CL method (Replay [44]) for visualization on task 2, 7, 12 and 17 with 5 task intervals. As shown in Fig. 10 (a) to (d), the loss landscape all becomes much flatter than that of the vanilla method across each task. This trend provides strong empirical support for C-Flat.

(a) Task 2

(b) Task 7

(c) Task 12

(d) Task 17

1.11

LIX

1.hi

t

th

EU00

=U75

=4190

,1

400

=L5

0JR

t

=LA

=0,194

3,14

MTS

8L008

0.308

A.6 Overhead of C-Flat

To enhance the computing efficiency, we apply C-Flat in a limited number of iterations within each epoch. Remarkably, we observe that without executing C-Flat in every iteration can also significantly boost the performance of CL (All cases derived from C-Flat improves CL performance). As illustrated in Table 4, 10% C-Flat iterations is enough to improve CL performances, and around 50% C-Flat iterations is enough to approach and even exceed the impact of a full C-Flat training. As a consequence, the overhead of 50% C-Flat is at least 30% shorter compared with the full C-Flat training. These observations holds potential for light C-Flat boosted CL applications.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly stated the contributions and scope in the abstract and introduction. And our claims match experimental results conducted in various settings.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of our work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: There is no new theoretical result in our paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have detailed the settings of our proposed method and experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided details for reproduction in Appendix. Code will be publicly available upon publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have detailed all key experimental settings in the main paper, and hyperparameters of all employed CL methods adhere to the settings in the open-source libraries unless specifically stated.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We do not use error bars, but present extensive experiment results across CL methods, datasets, incremental settings, and architectures.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have details this information in the main paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: There are no ethical issues involved in this paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is just a basic study on continual learning, and does not directly address societal impacts to the best of our knowledge.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We have not released any data or models that have risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: we have cited all original papers that produced the code packages and datasets used in our work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We have not introduced any new asset.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper points out that the current sharpness in continual learning (CL) tends to optimize towards a suboptimal space rather than achieving a global solution for continuous tasks. The proposed C-Flat, smoothly migrates to the global optimal of the joint knowledge space of the current and next tasks, thereby promoting CL. The idea is an intriguing one and has the potential to make a significant contribution to the CL community.\n\nSTRENGTHS:\n1) the paper is well-structured and presented, compelling narrative. The benefits and motivation generally flowed well. To their credit, the authors provide a theoretical analysis. \n\n2) This paper investigates the relationship between flatness, loss landscape, and CL from an optimization view, and studies the limitation of sharpness applied in current works (e.g., FS-DGPM/F2M etc.), effectively complementing the current body of research. \n\n3) The authors unify the three preceding aspects, and demonstrate the mechanism of loss landscape flatness on several categories of CL methods. The authors show the necessity of a flatter loss landscape for CL cuz favors global optimal during CL. Then, their optimization framework can be plugged into any CL method. Such an approach seems to be non-trivial. Third, this work may be a basic toolkit for the CL community.\n\nWEAKNESSES:\n1) The authors seem made a non-trivial effort in the exploration of the sharpness-aware approach in CL. However, they should elaborate more on the differences with the current CL methods in INTRODUCTION to highlight the contribution of the manuscript, although I note that they did so in RELATED WORK.\n\n2) In Equation 3 and Equation 4, I noticed that C-Flat additionally decreases the curvature around local minima. Does this mean lower loss or less forgetting for old tasks in the next stage? \n\n3) Their method shows an outstanding convergence speed (Figure 5), thus, in practice, the overhead is acceptable (close to SGD, SAM in some cases). Perhaps the authors could put more emphasis on practicality in SECTION 4.6, which could be meaningful.\n\n4) A few terms (e.g., SAM, Figure 1, etc.) lack citations, which may hinder reading.\n\nQUESTIONS:\nAll experiments are conducted on the complicated Class IL scenarios and consistently achieve improvements. However, still wondering how much the larger domain gap affects the stability of C-Flat. Although Class IL already has fairly complex data distribution changes for CL.\n\nIn addition, please answer WEAKNESS i, and ii.\n\nLIMITATIONS:\nPlease see my comments in WEAKNESS and QUESTIONS.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method named C-Flat to mitigate catastrophic forgetting by optimizing for a flatter loss landscape. This method is described as a plug-and-play solution that can be easily integrated into a wide range of existing CL approaches. The paper argues that this approach not only stabilizes the learning process but also enhances the model's generalization across tasks by enabling it to find and utilize flatter minima. The authors demonstrate the effectiveness of their method when applied to different CL solutions on 3 different datasets.\n\nSTRENGTHS:\n1. **Novelty and Effectiveness of C-Flat**: The introduction of C-Flat as a method that emphasizes flatness in the loss landscape to address catastrophic forgetting is innovative. The paper provides extensive experimental results showing that C-Flat improves performance across a variety of continual learning settings and benchmarks.\n\n2. **Theoretical Analysis**: The theoretical grounding provided for the convergence of the proposed loss is valuable and adds depth to the paper.\n\n3. **Visualizations**: The visualization of the loss landscape in Figure 4 effectively illustrates the impact of the proposed method, providing a clear comparative insight into how C-Flat modifies the learning dynamics.\n\nWEAKNESSES:\n1. **Generalization Claims Overstated**: The paper's assertion that it resolves the sensitivity/stability dilemma in CL is overly strong. While the proposed method shows promising results, it would be more accurate to state that it addresses rather than resolves these issues. (see line 96)\n\n2. **Theoretical Section Structure**: The structure of the theoretical analysis section is confusing. It would benefit from a more organized presentation, starting with assumptions, followed by theorems, and a detailed proof. This would enhance the readability and academic rigor of the paper. The assumptions (such as the differentiability of the loss) should be stated first in an assumption environment. Then the theorem or proposition should be stated (for example: Given the assumption above, when <conditions>, the convergence of <loss> is guaranteed) followed by a proof environment. clumping several steps in one equation (Eq. 8) is confusing and should be avoided for clarity. \n\n3.**Oragnization**: From the beginning of line 141 to the end of Eq. 10 seems disconnected from the rest of the paper. The authors should provide context for what they are proposing and explain why the proposed content matters.\n\nQUESTIONS:\n1. **Revision of Language**: The authors are encouraged to thoroughly revise the language of the manuscript to enhance readability and clarity. \n\n2. **Typo**: I believe there is a typo in line 98.\n\n3. **Average Boost**: I would like to ask the authors to report the average of the boost alongside the maximum boost in the results section. This could give a better understanding of the effectiveness of the proposed method.\n\nLIMITATIONS:\nThe paper lacks a discrete limitation section or a discussion of the limitations in the conclusion section. I would like to ask the authors to include this in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nContinual learning seeks to learn a series of new tasks without forgetting old ones. This paper explores the impact of a flat loss landscape on catastrophic forgetting.\n\nSTRENGTHS:\n- This paper applies loss landscape optimization to multiple categories of continual learning methods.\n- The method proposed in this paper is easy to implement and the article structure is well organized.\n\nWEAKNESSES:\n- The contribution of this paper is unclear. The authors claim that they are the first to compare CL methods with loss landscapes. However, there are many works that have discussed the impact of flatness on continual learning (or catastrophic forgetting) [1-5], which are not discussed.\n- The discussion of related work is not comprehensive. First, the connection and difference between the proposed method and the existing loss landscape-based continual learning methods [1-5] are not fully discussed in related work. In addition, there is also a lack of discussion on the difference and connection between the proposed C-Flat and various existing sharpness-aware minimization methods.\n- There is a lack of performance comparison with related work, so it is unclear how the gains compare to other works that improve CL based on flatness [1, 2, 3].\n- The proposed C-Flat has significantly larger computational cost. As mentioned in Section 3, it requires 2 forward and 4 backward propagations.\n- The results in Tab.2 show that C-Flat has only a slight performance improvement.\n\nReferences:\n\n[1] Cha, S., Hsu, H., Hwang, T., Calmon, F. P., & Moon, T. CPR: classifier-projection regularization for continual learning. In ICLR, 2021.\n\n[2] Yang, E., Shen, L., Wang, Z., Liu, S., Guo, G., & Wang, X.. Data augmented flatness-aware gradient projection for continual learning. In ICCV, 2023.\n\n[3] Tran Tung, L., Nguyen Van, V., Nguyen Hoang, P., & Than, K.. Sharpness and gradient aware minimization for memory-based continual learning. In SOICT, 2023.\n\n[4] Chen, Runhang, et al. \"Sharpness-aware gradient guidance for few-shot class-incremental learning.\" Knowledge-Based Systems (2024): 112030.\n\n[5] Mehta, S. V., Patil, D., Chandar, S., & Strubell, E. (2023). An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research, 24(214), 1-50.\n\nQUESTIONS:\n- In Section 2, why should “Gradient-based solutions” be a separate paragraph?\n- In Figure 5, why does the loss become smooth after the 80th epoch, but it is very drastic before?\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new algorithm agnostic optimisation method, which is tailored specifically for continual learning. This method takes advantage of zeroth order landscape sharpness-aware optimisation and proposes a new method, which improves upon SGD.\n\nSTRENGTHS:\nOriginality: the method raises a question which is often overlooked, which is the role of stochastic optimisers in continual learning. While many of the existing methods focus on the continual learning algorithms, this submission considers orthogonal aspect, which is tailoring the optimisation process in accordance with the needs of continual learning. \n\nQuality: the ablation studies show the parameter choice trade-off, time expenditure and the performance of the method; however the lack of confidence intervals does not allow to find out how significant this performance improvement is. The presentation and the derivation of the method look clear.\n\n Clarity: the paper is clear and well-structured in general, however, please refer to the weaknesses section for more detail on omissions.\n\nWEAKNESSES:\nClarity: there has been a number of omissions in the paper which need to be fixed and which stay in a way of understanding of the paper. This includes, for example, \n\nLine 38: \"Another group of works seeks to preserve model generalization with **regulations** onto the training procedure itself\" regularisation (instead of regulations?) of the training procedure?\n\nLine 40: \"are designed to encourage the training to efficiently extracting **feathers** for the current data space without forgetting\" Features?\n\nLine 188: 'Foster uses KL-based loss function to regularize the three combinations of old and new blocks for a stable performance on the previous data. It also introduces an effective redundant parameters and feature pruning strategy to maintain the single backbone model using knowledge distillation. DER follows the same framework, and introduces an auxiliary classifier and related loss item to encourage the model to learn diverse and discriminate features for novel concepts'\nWhile I understand it is related to the methods described in Table 1, they are only discussed in the next section, it would be easier if the authors cited the FOSTER paper for clarity in  Section 3.1; furthermore, it is capitalised in the table but not in the description.\n\nQuality: The proposed experimental analysis of the method does not come across as fully backing up the claims: from table 1, the accuracy improvement does not looks too big, especially given that there are no confidence intervals given which does not allow us to make the conclusions on whether the advantage is significant.\n\nQUESTIONS:\n1) the confidence intervals for the experimental  analysis is crucial for the understanding of the performance improvements. \n2) The comparison is only done in relation to the standard SGD; would it make sense to compare such results with other optimisers such, e.g., Adam? Does this optimiser improve upon the SGD performance in the standard, offline learning scenarios? \n3) The paper only evaluates the optimiser on class-incremental learning benchmarks. I wonder how does the method perform on domain-incremental settings using, for example, the protocol from van de Ven et al (2022)\n4) Should the authors also compare the proposed method with other methods taking advantage of sharpness-aware optimisation?  If not, why?  The concern is that currently the model only compares with a weak baseline of standard SGD, while existing similar methods might provide better performance. \n\nvan de Ven, G.M., Tuytelaars, T. & Tolias, A.S. Three types of incremental learning. Nat Mach Intell 4, 1185–1197 (2022). https://doi.org/10.1038/s42256-022-00568-3\n\n[21] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. NeurIPS, 32, 2019.\n\nLIMITATIONS:\n1) It would be important to hear more about the choice of parameters: Section 4.1 states that for all methods, the hyperparameters have been chosen to be the same. At the same time, it would be interesting to know what procedure did the authors use to choose these hyperparameters.\n\n2) It might be also good to hear if the authors have any details on failure modes for the method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}