Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning
Year: 2022

Abstract:
The backpropagation networks are notably susceptible to catastrophic forgetting, where networks tend to forget previously learned skills upon learning new ones. To address such the ’sensitivity-stability’ dilemma, most previous efforts have been contributed to minimizing the empirical risk with different parameter regularization terms and episodic memory, but rarely exploring the usages of the weight loss landscape. In this paper, we investigate the relationship between the weight loss landscape and sensitivity-stability in the continual learning scenario, based on which, we propose a novel method, Flattening Sharpness for Dynamic Gradient Projection Memory (FS-DGPM). In particular, we introduce a soft weight to represent the importance of each basis representing past tasks in GPM, which can be adaptively learned during the learning process, so that less important bases can be dynamically released to improve the sensitivity of new skill learning. We further introduce Flattening Sharpness (FS) to reduce the generalization gap by explicitly regulating the flatness of the weight loss landscape of all seen tasks. As demonstrated empirically, our proposed method consistently outperforms baselines with the superior ability to learn new skills while alleviating forgetting effectively.2.

heading: 1 Introduction
text: Humans have the ability to continually learn new knowledge without forgetting their previously learned ones through mediating a rich set of neurocognitive mechanisms [41, 15, 39]. This ability, often known as continual learning or lifelong learning [29], is crucial for computational systems, such as deep neural networks (DNNs), which are required to sequentially learn and deal with multiple tasks when implemented in the dynamically changing environment. Continual learning remains a long-standing challenge for DNNs since these networks are typically trained with stationary training batches by stochastic gradient descent methods [19], which generally leads to an abrupt performance decrease on previously learned tasks as new tasks are learned. To address such catastrophic forgetting, we can brutally retrain an oracle network on the entire dataset containing all tasks to capture dynamic changes in the data distribution, but this methodology is obviously too inefficient to hinder the learning of novel data in real time.
During the last few years, lots of research efforts have been devoted to improving the stability of DNNs on old tasks while keeping sensitive to new information. The first intuitive idea is to introduce
∗Corresponding author: Guangyong Chen. <gy.chen@siat.ac.cn> 2The code is available at: https://github.com/danruod/FS-DGPM
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
an independent branch for each new task while freezing the old task parameters to preserve the old knowledge [33, 45, 43, 25, 35, 22]. However, in this way, the network will inevitably become redundant as the task number continually increases. As presented in the neurocognitive works [41, 15], the reactivation of neuronal activity patterns, representing old memories, plays an important role in the continual learning of humans [39]. Thus, forgetness can be effectively mitigated by training a single network for new tasks by considering diverse information stored in the memory, including the original training samples of old tasks [31, 5, 13], the gradients induced from old tasks [9] and the feature subspace representing old tasks [34]. However, their continual learning performance is still limited because DNNs can easily overfit the limited information stored in the small-size memory.
The overfitting problem of DNNs is often attributed to the complex loss landscape containing multiple local optima, and the sharpness of the loss landscape has been widely used to characterize the generalization gap in standard training scenarios from both theoretical and empirical perspectives [27, 21, 42, 10, 6]. While this characterization has inspired new approaches for model training with better generalization, practical algorithms that especially seek out flatter minima to effectively address the ’sensitivity-stability’ dilemma for continual learning have thus far been elusive. In this paper, our first contribution is to characterize the weight loss landscape for the continual learning scenario and identify that a flatter loss landscape with lower loss value often leads to better continual learning performance, as shown in Figure 1 and Figure 3.
Further, based on our characterization of the weight loss landscape, we find that the recently proposed Gradient Projection Memory (GPM) method [34] maintains the lowest loss value on old tasks among the previously proposed methods by taking gradient steps orthogonal to the subspace representing old tasks. However, its loss landscape on newly learned tasks is the sharpest due to the lack of sufficient subspace left for new task learning. To improve the network’s sensitivity, our second contribution is to predict the importance of bases spanning the subspace for old tasks, so that less important bases can be dynamically released. In particular, we introduce a soft weight to indicate the bases importance, which can be dynamically adjusted by combining the Flattening Sharpness (FS) to minimize the loss value and loss sharpness simultaneously. Intuitively, a basis will be regarded as important for preserving old knowledge if the gradients induced by new tasks and old ones are aligned in the opposite direction on that basis. As demonstrated through extensive experiments, our proposed method can consistently outperform the state-of-the-art methods [17, 30, 24, 31, 5, 13, 34] by a notable margin across a range of widely used benchmark datasets.

heading: 2 Related Work
text: In this section, we briefly survey the representative works of continual learning by highlighting their contributions. To simplify our presentation, this section is organized by dividing the representative works into three categories, parameter isolation, regularization-based, memory-based methods.
Parameter isolation methods address forgetting by assigning a different subset of network parameters to each task. Without restrictions on network architecture, new neurons or layers or modules can be added for new tasks, while the previous task parameters can be frozen or copied to preserve old knowledge. For instance, Progressive Neural Network (PGN) [33] freezes the parameters trained with previous knowledge while expands the architecture by allocating new sub-networks with fixed capacity for new tasks. Dynamically Expandable Networks (DEN) [45] selectively retrains or expands network capacity by splitting/duplicating important units on new tasks. Reinforced Continual Learning (RCL) [43] uses reinforcement learning strategy to adaptively expand the network of each layer, while [22] use neural architecture search to find optimal network structures for each sequential task. Alternatively, with the architecture remaining static, a fixed part is allocated to each task. During the training of a new task, previous task parts are masked out to prevent interference. The mask sets are imposed at parameter level [25], or unit level [35]. PackNet [25] uses iterative pruning to fully restrict gradient updates on important weights via a binary mask, whereas HAT [35] limits the update of important units recognized by the hard attention mask through gradient backpropagation.
Regularization-based methods introduce an additional regularization term in the loss function to consolidate previous knowledge without using replay. This involves using knowledge distillation [23, 14] or penalizing changes in weights deemed important for previous tasks [17, 46, 28, 2, 3] to reduce forgetting. There are many ways to measure the importance. Elastic Weight Consolidation (EWC) [17] identifies important weights based on the diagonal values of Fisher information matrix
after training, while Synaptic Intelligence (SI) [46] calculates them online and over the entire learning trajectory in parameter space. Memory Aware Synapses (MAS) [2] estimates importance weights in an unsupervised manner, while Variational Continual Learning (VCL) [28] introduces a variational framework that spawned some Bayesian-based works [32, 8, 1, 7]. For example, [32] recursively uses a Gaussian Laplace approximation of the Hessian to approximate the posterior after every task, [8] adjusts the learning rate according to the uncertainty defined by the probability distribution of the network weights. [1] introduces an interpretation of node-wise uncertainty on the Kullback-Leibler (KL) divergence term of the variational lower bound for Gaussian mean-field approximation.
Our method mainly follows memory-based methods, which mitigate forgetting based on information extracted from old tasks or based on a generative model to generate pseudo samples. For example, iCaRL [30] selects and stores samples closest to the feature mean of each class. ER [5, 31] suggests reservoir sampling under the limited and fixed budget for replay buffer. Deep Generative Replay (DGR) [36] trains a deep generative model in the Generative Adversarial Network (GAN) framework [11] to simulate past data. These previous task samples are mainly reused as model inputs for replay in the above methods. However, replay might be prone to overfitting the subset of stored samples. Alternatively, samples stored in memory can also be used to constrain the optimization of the new task loss to prevent previous task interference, thereby leaving more leeway for backward and forward transfer. Gradient Episodic Memory (GEM) [24] projects the estimated gradients in the feasible region, which is outlined by previous task gradients calculated from the episodic memory samples. Averaged-GEM (A-GEM) [4] relaxes the projection to a direction that is estimated from samples randomly selected from memory. [12] proposes a unified view of episodic memory-based continual learning methods, including GEM and A-GEM, and improves performance over these methods by using a loss-balancing update scheme. A few other works have utilized gradient information to protect previous knowledge. [31, 13] adopt optimization-based meta-learning to enforce gradient alignment between samples from different tasks. GPM [34] minimizes interference between sequential tasks by ensuring that gradient updates only occur in directions orthogonal to the input of previous tasks.

heading: 3 The Weight Loss Landscape of Continual Learning
text: In this section, we first introduce our formulation of continual learning, and then characterize the weight loss landscape for the continual learning scenario from stability and sensitivity. Finally, some insights combining the weight loss landscape and continual learning are provided.

heading: 3.1 Problem Formulation
text: Throughout the paper, we denote scalars as a, vectors as a, matrices as A, and sets asA. We consider a supervised learning setup where T tasks are sequentially learned from their training data. Each task has an identical task descriptor, τ ∈ {1, 2, . . . , T}, with its dataset Dτ = {xi,τ , yi,τ}nτi=1 containing nτ samples randomly generated from a latent distribution Dτ . At any time-step during the learning process, we minimize the empirical risk of the model on all t tasks seen so far, with just limited size of memoryM to summarize the training data of previous tasks {Dτ}t−1τ=1. To simplify the notation, we denote LA(w) = 1|A| ∑ (x,y)∈A [` (fw (x) , y)] as the average empirical loss for the set A, where `(·, ·) is an arbitrary loss function (e.g. the cross-entropy (CE) loss), |A| is the sample size of the set A, and fw is the DNN with weight w. Our final goal is to find an optimal parameter w, which minimizes the overall risk ∑T τ=1 LDτ (w) for all tasks.

heading: 3.2 Connection of Weight Loss Landscape and Continual Learning
text: After learning a new task, we visualize the weight loss landscape of each task seen so far by plotting changes in its training loss when moving the weights w in a random direction d with magnitude α following [21]:
gt(α) = LDt(w + αd) = 1 |Dt| ∑
(x,y)∈Dt
` (fw+αd (x) , y) ,
where Dt is the training set for the t-th task previously learned. To eliminate the scaling invariance of DNNs, d is sampled from a Gaussian distribution and further normalized by dl,j ← dl,j‖dl,j‖F ‖wl,j‖F ,
where dl,j represents the j-th filter at l-th layer of d, and ‖·‖F denotes the Frobenius norm. Compared with our visualization, [26] only consider one task and plot the loss landscape along the directions of its Hessian eigenvectors, which only reflects some of the relationship between forgetting and landscape. Considering d is randomly selected, we repeat the visualization 10 times with different d.
We first study the stability of the network by plotting changes of the weight loss landscape for the first task after new task learning. In particular, We use the previously proposed ER [5], La-MAML [13], and GPM [34] to train a MLP network with two hidden layers on the Permuted MNIST (PMNIST) [20] dataset that contains 10 tasks. We also retrain the network on the entire dataset contain all passed tasks as an Oracle network. Early stopping is used to halt the training with up to 10 epochs for each task based on the validation loss as proposed in [35]. As shown in Figure 1(a), all three continual learning methods lose their stability as learning new tasks. It can be observed from Figure 1(b)-(c) that the weight loss landscape becomes sharper and loss value increases simultaneously, when the testing accuracy of the first task continually decreases. We further evaluate the sensitivity of the network by observing the performance of the fifth task just after it has just been learned. As shown in Figure 1(d)-(e), ER shows the best learning capability compared with other methods with the lowest loss value and the flattest loss landscape. Thus, based on these empirical findings, we assume that lower loss value with a flatter neighbor may lead to better continual learning performance.
3.3 A Case Study of Flattening Sharpness for Vanilla ER
In this part, we further validate our above assumption by Flattening Sharpness for vanilla ER (FS-ER). Compared with ER that looks for a solution w that jointly minimizes the training loss of current task data and memory data, FS-ER seeks out a solution with both low loss and flat neighbor by minimizing the maximal loss in the neighbor around the parameter value. The schematic of the Flattening Sharpness (FS) is shown in Figure 2. We introduce the adversarial weight perturbation (orange dashed line) to explicitly flatten the weight loss landscape via injecting the worst-case weight perturbation, which is calculated from the current task data and past task data sampled from the replay buffer (Refer to Appendix C.1 and the next section for more details). Figure 3 respectively shows the weight loss landscapes of the fifth task after just learning the fifth task and all tasks. We find that FS-ER successfully gets a solution with a lower loss value and flatter landscape, either after the fifth task has just been learned or all ten tasks have been learned. The average testing accuracy of all tasks using FS-ER is 90.44%, significantly higher than ER (86.16%), which means that flattening sharpness does benefit continual learning.

heading: 4 Flattening Sharpness for Dynamic Gradient Projection Memory
text: As shown in Figure 1, GPM achieves the highest testing accuracy on old tasks among all three practical continual learning methods, but shows less sensitivity to new task learning. To address this issue, we propose Flattening Sharpness for Dynamic GPM (FS-DGPM), which dynamically adjusts the gradient subspace representing the past tasks to improve the sensitivity to the new task, while ensuring stability of the previous tasks. In particular, we let M = [u1,u2, · · · ,uk] denote the bases matrix that spans the gradient subspace of the previous task, Λ = diag[λ1, λ2, ..., λk] be the diagonal matrix with its i-th diagonal element λi ∈ [0, 1] indicating the importance of each basis, and k is the number of bases.

heading: 4.1 Sharpness Evaluation
text: Comparing with the classical strategy that perturbs weight in the entire space [40, 16, 42, 10], we focus on characterizing the weight loss landscape on the new task with respect to the important subspace representing old task. The important subspace can be effectively calculated based on the examples sampled from replay bufferM after each task training. Then, we can find the worst case by maximizing the training loss of the network on the new task in this subspace. Formally, the sharpness of the loss landscape around the solution w in the old parameter space can be predicted as,
max v∈V LD̂t (w + v) , (1)
where V denotes the subspace spanned by M and Λ, and D̂t denotes the batch samples of the current t-th task. As shown in Eq. (1), the high value can be obtained when the network fails to learn the new task (sensitivity) and the new task learning seriously interferes with the past tasks learning (stability). Based on the gradient method, the adversarial weight perturbation v can be solved as,
v ← v + η1MΛMT ( ∇(w+v)LD̂t(w + v) ) , (2)
where η1 is the update step size. Note that v is initialized as 0 and layer-wise updated. As shown in Appendix E, two-step for v (default settings) are enough to get good improvements.

heading: 4.2 Dynamic Gradient Projection Memory
text: After obtaining the adversarial weight perturbation v, we can further update the bases importance matrix Λ = diag[λ1, λ2, ..., λk] by jointly considering the current task batch D̂t and the batch M̂ sampled from the replay bufferM as following,
λi ← λi − η2 ( ∇λiLD̂t∪M̂(w + v) ) , (3)
where the sigmoid function is used at the end of gradient update to constrain the importance value λi between 0 and 1. In addition, the second term on the right side in Eq. (3) can be approximated by the first-order Taylor expansion as,
∇λiLD̂t∪M̂(w + v) ≈ η1 ( ∇wLD̂t (w) )T uiu T i ( ∇wLD̂t∪M̂ (w) ) . (4)
(a) Transfer
∇ 𝐿!(𝜔)
∇𝐿"(𝜔) 𝜇
(b) Interference
𝜇 ∇𝐿!(𝜔) ∇𝐿"(𝜔)
Figure 4: A depiction of transfer (a) and interference (b) in the basis u of gradient space.
The above equation characterizes the relationship between the gradients induced by the current task and the old tasks based on the basis ui. As illustrated in the Figure 4, this equation implies that when the projections of two gradients on the basis ui are aligned in the same direction, the gradient of λi will be positive, and when there is interference, the gradient will be negative. The positive (negative) gradient will decrease (increase) the importance λi, thereby releasing (tightening) the update limit of the new task on the corresponding
basis ui. We provide the full derivation in the Appendix A.2. Note that the initial value of all importance is set to 1 and dynamically adjusted from the second task.

heading: 4.3 Weight Updating
text: Finally, we update the model parameters by minimizing the worst performance of fw+v with respect to w, while adjusting the update magnitude of w on each basis based on its importance to alleviate forgetting. More concretely, the parameter w will be updated to:
w ← w − η3 ( I −MΛMT ) ∇wLD̂t∪M̂(w + v). (5)
Note that the optimization is performed over the model parameters w, whereas the objective is computed using perturbed model fw+v. In addition, we update the replay bufferM with reservoir sampling as in [31], and then use Singular Value Decomposition (SVD) to recalculate M based on the sampling data in the replay buffer after learning one task following GPM [34]. Comparing with [34], we calculate the important bases in the entire gradient space and use them to replace the bases calculated last time. Besides that, our method degenerates to GPM when η1 and η2 are set to 0. The complete pseudo-code of FS-DGPM is outlined in the Algorithm 1.
Algorithm 1 FS-DGPM (Flattening Sharpness for Dynamic Gradient Projection Memory) Input: Network weight w, loss function `, learning rate η3, FS step size η1, FS steps K, Soft weight step size η2, batch size b. InitializingM← {}, M ← I,Λ← I for t = 1, 2, · · · , T do
for ep = 1, 2, · · · , numepochs do for batch D̂t
b∼ Dt do M̂ b∼M for k = 1, · · · ,K do
v ← v + η1MΛMT ( ∇(w+v)LD̂(w + v) ) . Sharpness Evaluation
end for if t ≥ 2 then Λ← Λ− η2 ( ∇ΛLD̂t∪M̂(w + v) ) . Dynamic Gradient Projection Memory end if w ← w − η3 ( I −MΛMT ) ∇wLD̂t∪M̂(w + v) . Weight updating
Push D̂t toM with reservior sampling end for
end for M ← UpdateGPM (M) . see Appendix Alg. 2
end for

heading: 4.4 Theoretical Understanding
text: We further provide a theoretical view on why landscape can characterize the continual learning performance and why our proposed FS-DGPM works. To simplify our explanation, we only consider two tasks, which contains the training sets D1 and D2 sampled from the distributions D1 and D2, respectively. Based on the previous works on PAC-Bayes bound [27, 42, 10], given a "prior" distribution P (a common assumption is zero mean, σ2 variance Gaussian distribution) over the weights, the expected error of the classifier for the continual learning scenario can be bounded with probability at least 1− δ over the draw of n training data:
min ∆w Ev [LD1∪D2(w + ∆w + v)] ≤ min ∆w∈VC Ev [LD1(w + ∆w + v)] + LD2(w + ∆w)
+ max v∈V LD2(w + ∆w + v)− LD2(w + ∆w) + 4
√ 1
n
( KL(w + ∆w + v||P ) + ln 2n
δ
) .
where ∆w is the update based on the previously optimal solution w learned on the old task D1 when learning the new one D2, and v is often chosen as a zero mean spherical Gaussian perturbation with variance σ2 in every direction. Let ∆w ∈ VC , then ∆w lies in the complementary space of the important space representing the old task D1, so that Ev [LD1(w + ∆w + v)] does not increase
too much compared with the previously minimized Ev [LD1(w + v)]. The second term denotes the empirical loss on the second task and the third term represents the sharpness of the weight loss landscape around the w + ∆w. Since we have constrained ∆w ∈ VC , then it is natural to assume v ∈ V , so that ∆w + v will cover the full space. Thus, our FS-DGPM exactly optimizes the worstcase of the flatness of weight loss landscape to control the PAC-Bayes bound, which theoretically justifies both lower loss value and flatter landscape lead to better continual learning performance, and why our proposed FS-DGPM works.

heading: 5 Experiments
text: In this section, we conduct extensive experiments to compare the performance of our proposed FS-DGPM model with the state-of-the-art methods on widely used continual learning benchmark datasets. Additional results and more details about the datasets, experiment setup, baselines, and model architectures are presented in the Appendix D and E.

heading: 5.1 Experimental Setup
text: Datasets: We evaluate our algorithm on four image classification datasets: Permuted MNIST (PMNIST) [20], CIFAR-100 Split [18], CIFAR-100 Superclass [44] and TinyImageNet [37]. The PMNIST dataset is a variant of the MNIST dataset, in which each task applies a fixed random pixel permutation to the original dataset. The PMNIST benchmark dataset consists of 20 tasks, and each contains only 1000 samples from 10 different classes [13]. The CIFAR-100 Split is constructed by randomly dividing 100 classes of CIFAR-100 into 10 tasks with 10 classes per task. The CIFAR-100 Superclass is divided into 20 tasks according to the 20 superclasses of the CIFAR-100 dataset, and each superclass contains 5 different but semantically related classes. Whereas, TinyImageNet is constructed by splitting 200 classes into 40 5-way classification tasks. In our experiments, we do not use any data augmentation. The dataset statistics are given in Appendix D.1.
Network Architecture: For PMNIST, we use a fully connected network with two hidden layers of 100 units each following [24]. For experiments of CIFAR-100 Split and CIFAR-100 Superclass, we use a 5-layer AlexNet and LeNet architecture similar to [34] respectively. For TinyImageNet, we use the same network architecture as [13], which consists of 4 conv layers and 3 fully connected layers. In PMNIST, all tasks share the final classifier layer, while other experiments use a multi-head incremental setting, that is, each task has a separate classifier. Refer to Appendix D.2 for more details.
Baselines: We compare our method against multiple methods described below. (1) EWC [17], a regularization-based method that uses the diagonal of Fisher information to identify important weights; (2) ICARL [30], a memory-based method that uses knowledge-distillation and episodic memory to reduce forgetting; (3) GEM [24], another memory-based method that uses the gradient of episodic memory to constrain optimization to prevent forgetting; (4) ER [5], a simple and competitive method based on reservoir sampling; (5) La-MAML [13] and (6) GPM [34] are memory-based methods inspired by optimization-based meta-learning and based on gradient orthogonal constraints, respectively; (7) Multitask is an oracle baseline that all tasks are learned jointly using the entire dataset at once in a single network. Multitask is not a continual learning strategy but serves as an upper bound on average test accuracy on all tasks.
Training Details: All baselines and our method use stochastic gradient descent (SGD) for training. For each task in PMNIST and TinyImageNet, we train the network in 1 and 10 epochs, respectively, with the batch size as 10. These experimental settings are the same as La-MAML [13], so that we directly compare with their reported results. In the CIFAR-100 Split and CIFAR-100 Superclass experiments, we use the early termination strategy to train up to 50 epochs for each task, which is based on the validation loss as proposed in [35]. For both datasets, the batch size is set to 64. The replay buffer size of PMNIST, CIFAR-100 Split, CIFAR-100 Superclass, and TinyImageNet are 200, 1000, 1000, and 400, respectively. Details about the experimental setting and the hyperparameters considered for each baseline are provided in Appendix D.5 and D.6.
Metrics: We evaluate the continual learning performance by the average accuracy (ACC) and backward transfer (BWT) [24, 4, 5], formulated as following,
ACC = 1
T T∑ i=1 RT,i, BWT = 1 T − 1 T−1∑ i=1 RT,i −Ri,i,
where T is the total number of learned sequential tasks, Ri,j is the test classification accuracy of the model on j-th task after learning the last sample from i-th task. ACC is the average test classification accuracy of all tasks, bigger is better. BWT is the interference of new learning on the past knowledge. More specifically, negative BWT implies (catastrophic) forgetting whereas positive BWT indicates learning new task increases the performance on some preceding tasks.
5.2 Results and Discussion Table 2: Experimental results (mean ± std in 5 runs) on PMNIST in single-epoch.
PMNIST Method ACC(%) BWT(%) EWC 62.25 ± 1.44 -15.22 ± 1.25 GEM 61.82 ± 0.85 -15.58 ± 1.17 ER 68.31 ± 0.56 -13.91 ± 0.67 La-MAML 75.98 ± 0.60 -10.21 ± 0.90 GPM 74.54 ± 0.36 -7.17 ± 0.51 FS-DGPM 76.96 ± 0.51 -7.45 ± 0.30 Multitask 86.54 ± 1.74 - PMNIST: We first evaluate our algorithm for 20 sequential PMNIST tasks with only 1000 samples per task in a single-head setting. From the results, as shown in Table 2, we see that our method (FS-DGPM) achieves the best average accuracy (76.96% ± 0.77). Moreover, we achieve the least amount of forgetting except GPM, which is essentially a trade-off in accuracy to minimize forgetting. As shown in Figure 5(c), we only use about 31% of the final memory size of GPM and achieve ∼ 2.5% better accuracy.
CIFAR-100 and TinyImageNet: Next, we use a multi-head setting to evaluate our algorithm under the more challenging visual classification benchmarks. Table 1 reports all results of these experiments. We outperform all baselines on three datasets, with achieving the best average accuracy 74.33%, 58.81% and 70.41%. In these experiments, we observe that GPM is a strong baseline with the least forgetting. At the same time, we highlight that our method achieves the highest accuracy on all datasets and the second-lowest forgetting after GPM. Figure 5(a) shows the process of performance changing with the number of tasks on the CIFAR-100 Superclass. We consistently see the superior performance of our method at any stage of model evolution. It is also worth emphasizing that although our method requires more time for training than GPM, it has lower memory usage and better test accuracy (See Figure 5(b)-(c)). As noted by [4] and [38], EWC performs poorly without multiple passes over the datasets, and GEM is not very effective under the single-head variants. These situations have also been observed in our experiments.

heading: 5.3 Ablation studies on FS-DGPM
text: We further investigate our model performance with an ablation study and summarize it in Table 3. We respectively ablate the effects of flattening sharpness and dynamically adjusting the soft weight for bases. We refer to them as DGPM and FS-GPM. We also construct an ablation referred to as LaDGPM (Look-ahead DGPM), where the adversarial weight perturbation v in Eq. (2) is changed to the direction
of gradient descent. At the same time, we also change the sign in Eq. (3) to ensure that the soft weight of the basis is reduced when the gradients are aligned in the same direction. From the results, shown in Table 3, we observe that flattening sharpness does benefits GPM, with ∼ 1.0% improvement over GPM on both datasets. We can further observe through Figure 6 that all landscapes of FS-DGPM have lower loss values and flatter neighbors than DGPM and La-DGPM on the CIFAR-100 Split experiments. In addition, we see that DGPM performs well in learning new tasks, but it also leads to forgetting previous tasks. This situation can be efficiently alleviated by flattening sharpness. Hence, FS is indeed a powerful method worthy of being widely adopted for continual learning scenarios.

heading: 6 Conclusion
text: In this paper, we explore the weight loss landscape to characterize the well-known ’sensitivitystability’ dilemma faced by continual learning algorithms, and find that lower loss value with flatter neighbor often leads to better continual learning performance. Based on this finding, we propose our FS-DGPM algorithm, which introduces a soft weight to represent the importance of each basis representing past tasks in GPM, so that less important bases can be dynamically released to improve
the sensitivity of new skill learning. Flattening Sharpness (FS) is also introduced here to reduce the generalization gap by explicitly regulating the flatness of the weight loss landscape of all tasks seen so far. The evaluation of various image classification tasks with different network architectures and the comparison with some state-of-the-art algorithms show the effectiveness of our method in achieving high classification performance while alleviating forgetting. Although our method theoretically and empirically demonstrates the advantages of introducing FS into continual learning, whether there exists a closer upper bound for the continual learning performance remains an unresolved problem and left for our future exploration. Although our method theoretically and empirically demonstrates the advantages of introducing bases soft weight and FS into continual learning, whether there exists a better dynamic adjustment and a closer upper bound for the continual learning performance remains an unresolved problem and left for our future exploration.

heading: Acknowledgments
text: We would like to thank Dr. Yongzhe Deng and Zhuo Zhang for their helpful discussions, and anonymous reviewers for their valuable comments to improve this work. This work was supported by a grant from the National Key Research and Development Program of China (Project No. 2020YFB1313900), Hong Kong Research Grants Council under General Research Fund (Project No. 14201620), the National Natural Science Foundation of China (Project No. 62006219, 62072452) and Guangdong Provincial Basic and Applied Basic Research Fund-Regional Joint Fund (Project No. 2020B1515130004).

Review summaries:
The paper proposes a new continual learning method that promotes the loss shapes to be flattened. The main idea is to optimize the worst case perturbed loss function at every gradient step, and the empirical investigation of the loss values around the learned parameter corroborates their assertion. Also, experimentally, they showed promising results compared to other recent baselines. 

The paper studies catastrophic forgetting and continual learning from the loss landscape perspective. The primary motivation is that from the plasticity/stability view, stable networks tend to have flatter minima. While this has been observed before in [1], the paper studies the problem in more depth. Also, it proposes a solution to reduce the sharpness of minima for each task based on the gradient projection memory work.   Overall, I like the paper, and I think it is an influential continual learning paper. However, I think the paper can be even better if the authors can address a few shortcomings.

The article proposes a continual learning method that innovatively improves on an existing algorithm. Precisely, the current work takes GPM (Gradient Projection Memory) which they show is good at preserving performance on old tasks (good stability), but it's not that efficient at learning new ones (low sensitivity) and modifies it in two ways: by dynamically adjusting the importance of the various bases such that the less important ones can be dropped to increase sensitivity. In addition it improves the stability by employing a form of adversarial parameter perturbation to flatten the loss landscape. The contributions of the paper are: - an analysis that shows the loss landscape and its "flattness" / "sharpness"; this analysis is used to support the idea that flatter minima tend to forget less in continual learning setups; - changing the GPM such that it has an increased sensitivity while also improving the stability by flattening the loss landscape around the current solution 

In this paper, the authors propose a method called Flattening Sharpness for Dynamic Gradient Projection Memory (FS-DGPM) for continual learning. The main idea is to investigate the effect of weight loss landscape on overcoming catastrophic forgetting. With the understanding of the weight loss landscape, the authors further propose a method based on the recently proposed Gradient Projection Memory (GPM) method to predict the importance of bases which span the subspace of old tasks. Less important bases can be dynamically released based on the proposed soft weight. Extensive experiments show that the proposed method can consistently outperform the state-of-the-art methods across a range of widely used benchmark datasets.