Title: DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity
Venue: NeurIPS 2024
Accepted: yes
Keywords: loss of plasticity

warm-starting

incremental learning

generalization

Direction-Aware SHrinking
Abstract: Warm-starting neural network training by initializing networks with previously learned weights is appealing, as practical neural networks are often deployed under a continuous influx of new data. However, it often leads to *loss of plasticity*, where the network loses its ability to learn new information, resulting in worse generalization than training from scratch. This occurs even under stationary data distributions, and its underlying mechanism is poorly understood. We develop a framework emulating real-world neural network training and identify noise memorization as the primary cause of plasticity loss when warm-starting on stationary data. Motivated by this, we propose **Direction-Aware SHrinking (DASH)**, a method aiming to mitigate plasticity loss by selectively forgetting memorized noise while preserving learned features. We validate our approach on vision tasks, demonstrating improvements in test accuracy and training efficiency.

DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity

Abstract

Warm-starting neural network training by initializing networks with previously learned weights is appealing, as practical neural networks are often deployed under a continuous influx of new data. However, it often leads to loss of plasticity , where the network loses its ability to learn new information, resulting in worse generalization than training from scratch. This occurs even under stationary data distributions, and its underlying mechanism is poorly understood. We develop a framework emulating real-world neural network training and identify noise memorization as the primary cause of plasticity loss when warm-starting on stationary data. Motivated by this, we propose Direction-Aware SHrinking (DASH) , a method aiming to mitigate plasticity loss by selectively forgetting memorized noise while preserving learned features. We validate our approach on vision tasks, demonstrating improvements in test accuracy and training efficiency. 1 2

1 Introduction

When training a neural network on a gradually changing dataset, the model tends to lose its plasticity , which refers to the model's ability to adapt to new information (Dohare et al., 2021; Lyle et al., 2023b; Nikishin et al., 2022). This phenomenon is particularly relevant in scenarios with non-stationary data distributions, such as reinforcement learning (Igl et al., 2020; Nikishin et al., 2022) and continual learning (Chen et al., 2023; Kumar et al., 2023; Wu et al., 2021). While requiring to overwrite outdated knowledge as the environment changes, models overfitted to previously encountered environments often struggle to cumulate new information, which in turn leads to reduced generalization performance (Lyle et al., 2023b). Under this viewpoint, various efforts have been made to mitigate the loss of plasticity, such as resetting layers (Nikishin et al., 2022), regularizing weights (Kumar et al., 2023), and modifying architectures (Lee et al., 2023; Lyle et al., 2023a; Nikishin et al., 2023).

Perhaps surprisingly, a similar phenomenon occurs in supervised learning settings, even where new data points sampled from a stationary data distribution are added to the dataset during training. It is counterintuitive, as one would expect advantages in both generalization performance and computational efficiency when we warm-start from a model pre-trained on data points of the same distribution. For a particular example, when a model is pre-trained using a portion of a dataset and then we resume the training with the whole dataset, the generalization performance is often worse than a model trained from scratch (i.e., cold-start ), despite achieving similar training accuracy (Ash and Adams, 2020; Berariu et al., 2021; Igl et al., 2020). Liu et al. (2020) report a similar observation: training neural networks with random labels leads to a spurious local minimum which is challenging to escape from, even when retraining with a correctly labeled dataset. Interestingly, Igl et al. (2020) found that pre-training with random labels followed by the corrected dataset yields

0

10

20

30

40

50

Number of Experiments

10

20

30

40

50

Test Accuracy (%)

Cold

Warm

S&P

DASH

0

10

20

30

40

50

Number of Experiments

0

10000

20000

30000

Number of Steps

better generalization performance than pre-training with a small portion of the (correctly labeled) dataset and then training with the full, unaltered dataset. It is striking that warm-starting leads to such a severe loss of performance, even worse than that of a cold-started model or a model re-trained from parameters pre-trained with random labels, despite the stationarity of the data distribution.

These counterintuitive results prompt us to investigate the underlying reasons for them. While some studies have attempted to explain the loss of plasticity in deep neural networks (DNNs) under non-stationarity (Lewandowski et al., 2023; Lyle et al., 2023b; Sokar et al., 2023), their empirical explanations rely on various factors, such as model architecture, datasets, and other variables, making it difficult to generalize the findings (Lewandowski et al., 2023; Lyle et al., 2023a). Moreover, there is limited research that explores why warm-starting is problematic in stationary settings, highlighting the lack of a fundamental understanding of the loss of plasticity phenomenon in both stationary and non-stationary data distributions.

1.1 Our Contributions

In this work, we aim to explain why warm-starting leads to worse generalization compared to coldstarting, focusing on the stationary case. We propose an abstract framework that combines the popular feature learning framework initiated by Allen-Zhu and Li (2020) with a recent approach by Jiang et al. (2024) that studies feature learning in a combinatorial and abstract manner. Our analysis suggests that warm-starting leads to overfitting by memorizing noise present in the newly introduced data rather than learning new features.

Inspired by this finding, we propose Direction-Aware SHrinking (DASH), which aims to encourage the model to forget memorized noise without affecting previously learned features. This enables the model to learn features that cannot be acquired through warm-starting alone, enhancing the model's generalization ability. We validate DASH using an expanding dataset setting, similar to the approach in Ash and Adams (2020), employing various models, datasets, and optimizers. As an example, Figure 1 shows promising results in terms of both test accuracy and training time.

1.2 Related Works

Loss of Plasticity. Research has aimed to understand and mitigate loss of plasticity in non-stationary data distributions. Lewandowski et al. (2023) explain that loss of plasticity co-occurs with a reduction in the Hessian rank of the training objective, while Sokar et al. (2023) attribute it to an increasing number of inactive neurons during training. Lyle et al. (2023b) find that changes in the loss landscape curvature caused by non-stationarity lead to loss of plasticity. Methods addressing this issue in non-stationary settings include recycling dormant neurons (Sokar et al., 2023), regularizing weights towards initial values (Kumar et al., 2023), and combining techniques (Lee et al., 2023) like layer normalization (Ba et al., 2016), Sharpness-Aware Minimization (SAM) (Foret et al., 2020), resetting layers (Nikishin et al., 2022), and Concatenated ReLU activation (Shang et al., 2016).

However, these explanations and methods diverge from the behavior observed in stationary data distributions. Techniques aimed at mitigating loss of plasticity under non-stationarity are ineffective under stationary distributions, as shown in Appendix C.1, in line with the observations in Lee et al. (2023). While some works study the warm-starting problem in stationary settings, they rely on empirical observations without theoretical analysis (Achille et al., 2018; Ash and Adams, 2020; Berariu et al., 2021). The most relevant work by Ash and Adams (2020) introduces the Shrink & Perturb (S&P) method, which mitigates the loss of plasticity in stationary settings to some extent by shrinking all weight vectors by a constant factor and adding noise. However, they do not explain why this phenomenon occurs or why S&P is effective. We develop a theoretical framework explaining why warm-starting suffers even under stationary distribution. Based on findings, we propose a method that shrinks the weight vector in a direction-aware manner to maintain properly learned features.

Feature Learning in Neural Networks. Recent studies have investigated how training methods and network architectures influence generalization performance, focusing on data distributions with label-dependent features and label-independent noise (Allen-Zhu and Li, 2020; Cao et al., 2022; Deng et al., 2023; Jelassi and Li, 2022; Oh and Yun, 2024; Zou et al., 2023). In particular, Shen et al. (2022) examine a data distribution consisting of varying frequencies of features and large strengths of noise, emphasizing the significance of feature frequencies in learning dynamics. Jiang et al. (2024) propose a novel feature learning framework based on their observations in real-world scenarios, which also involves features with different frequencies but considers the learning process as a discrete sampling process. Our framework extends these ideas by incorporating features with varying frequencies, noise components, and the discrete learning process while introducing a more intricate learning process capturing the key aspects of feature learning dynamics in expanding datasets.

2 A Framework of Feature Learning

2.1 Motivation and Intuition

We present the motivation and intuition behind our framework before delving into the formal description. Our framework captures key characteristics of image data, where the input includes both label-relevant information (referred to as features , e.g., cat faces in cat images) and label-irrelevant information (referred to as noise , e.g., grass in cat images). A key intuition is that minimizing training loss involves two strategies: learning features and memorizing noise . This framework builds on insights from Shen et al. (2022) and integrates them into a discrete learning framework. We provide more detailed intuition on our framework, including the training process, in Appendix A.

Shen et al. (2022) consider a neural network trained on data with features of different frequencies and noise components stronger than the features. The negative gradient of the loss for each single data point aligns more with the noise than the features due to the larger scale of noise, making the model more likely to memorize noise rather than learn features. However, an identical feature appears in many data points, while noise appears only once and does not overlap across data points. Thus, if a feature appears at a sufficiently high frequency in the dataset, the model can learn the feature. Thus, the learning of features or noise depends on the frequency of features and the strength of the noise.

Inspired by Shen et al. (2022), we propose a novel discrete feature learning framework. This section introduces a framework describing a single experiment, while Section 3 analyzes expanding dataset scenarios. As our focus is on gradually expanding datasets, carrying out the (S)GD analysis over many experiments as in Shen et al. (2022) is highly challenging. Instead, we adopt a discrete learning process similar to Jiang et al. (2024) but propose a more intricate process reflecting key ideas from Shen et al. (2022). In doing so, we generalize the concept of plasticity loss and analyze it without assuming any particular hypothesis class for a more comprehensive understanding, whereas existing works are limited to specific architectures.

2.2 Training Process

We consider a classification problem with C classes, and data are represented as ( x , y ) ∈ X × [ C ] , where X denotes the input space. A data point is associated with a combination of class-dependent features V ( x ) ⊂ S y where S c = { v c, 1 , v c, 2 , . . . , v c,K } is the set of all features for each class c ∈ [ C ] . Also, every data point contains point-specific noise which is class-independent.

The model f : X → [ C ] sequentially learns features based on their frequency. The training process is described by the set of learned features L ⊂ S ≜ ⋃ c ∈ [ C ] S c and the set of data points with non-zero

gradients N ⊂ T , where T = { ( x i , y i ) } i ∈ [ m ] denotes a training set. The set N , representing the data points with non-zero gradients, will be defined below. The frequency of a feature v in data points belonging to N is denoted by

where ✶ ( · ) is the indicator function, which equals 1 if the condition inside the parentheses is true and 0 otherwise. At each step of training, if L and N are given, the model chooses the most frequent feature among the features not yet learned, i.e., arbitrarily choose v ∈ arg max u ∈S\L g ( u ; T , N ) .

The model decides whether to learn a selected feature v by comparing its signal strength, represented by |T | · g ( v ; T , N ) , with the signal strength of noise, given by γ , which reflects the key ideas of Shen et al. (2022). If the frequency of the selected feature v is no less than the threshold γ/ |T | , i.e., g ( v ; T , N ) ≥ γ/ |T | , the model learns v and adds it to its set of learned features L . The feature learning process continues until the model reaches a point where the selected feature v has g ( v ; T , N ) < γ/ |T | , indicating that the signal strength of every remaining feature is weaker than that of noise. At this point, the feature learning process ends.

We consider a data point x to be well-classified if the model f has learned at least τ features from V ( x ) , i.e., |L ∩ V ( x ) | ≥ τ , where τ < K . In this case, we consider x to have a zero gradient, meaning it cannot further contribute to the learning process. Throughout the feature learning process, the set N of data points with non-zero gradients is dynamically updated as new features are learned. At each step, when the model successfully learns a new feature, we update N by removing the data points that satisfy |L ∩ V ( x ) | ≥ τ , as they become well-classified due to the newly learned feature.

If the feature learning process ends and the model has learned as many features as it can, the remaining data points that have non-zero gradients will be memorized by fitting the random noise present in them and will be considered to have zero gradients. This step concludes the training process. Pictorial illustration can be found in Appendix A and a detailed algorithm of the learning process can be found in Algorithm 2 in Appendix E.

2.3 Discussion on Training Process

In our framework, the model selects features based on their frequency in the set of unclassified data points N . The intuition behind this approach is that features appearing more frequently in the set of data points will have larger gradients, leading to larger updates, and we treat g ( v ; T , N ) as a proxy of the gradient for a particular feature v . As a result, the model prioritizes sequentially learning these high-frequency features. However, if the frequency g ( v ; T , N ) of a particular feature v is not sufficiently large, such that the total occurrence of v is less than the strength of the noise, i.e., |T | · g ( v ; T , N ) < γ , the model will struggle to learn that feature. Consequently, the model will prioritize learning the noise over the informative features. When this situation arises, the learning procedure becomes sub-optimal because the model fails to capture the true underlying features of the data and instead memorizes the noise to achieve high training accuracy.

The threshold τ determines when a data point is considered well-classified and acts as a proxy for the dataset's complexity. A higher τ requires the model to learn more features for correct predictions, while a lower τ allows accurate predictions with fewer learned features. Experiments in Appendix B Figure 11 and 12 support this interpretation.

Remark 2.1. We believe our analysis can be extended to scenarios where feature strength varies across data by treating the set of features as a multiset, where multiple instances of the same element are allowed. The analyses in these cases are nearly identical to ours; therefore, we assume all features have identical strengths for notational simplicity.

3 Warm-Starting versus Cold-Starting, and a New Ideal Method

3.1 Experiments with Expanding Dataset

In this section, we set up the scenario where the dataset grows after each experiment in our learning framework, allowing us to compare warm-start, cold-start, and a new ideal method, which will be defined later in Sections 3.3 and 3.4.

To better understand the loss of plasticity under stationary data distribution, we consider an extreme form of stationarity where the frequency of each feature combination remains constant in each

chunk of additional data. We investigate if the loss of plasticity can manifest even under this strong stationarity. The detailed description of the dataset across the entire experiment is as follows:

Assumption 3.1. In each j -th experiment, we are provided with a training dataset T j := { ( x i,j , y i,j ) } i ∈ [ n ] with n samples. For each class c ∈ [ C ] and each possible feature combination A ⊂ S c , we assume that T j contains exactly n A ≥ 1 data points with associated feature set A , where the values of n A are independent of j . Note that ∑ c ∈ [ C ] , A⊂S c n A = n . In the j -th experiment, we use the cumulative dataset T 1: j := ⋃ l ∈ [ j ] T l , the set of all training data up to the j -th experiment.

Remark 3.2. In each experiment, the feature combinations remain the same across the dataset, but the individual data points differ. This is because each data point is associated with its specific noise, which varies across samples. Although the underlying features are the same, the noise component of each data point is unique. This approach ensures that the model is exposed to a diverse set of samples.

We define a technical term h ( v ; L ) ≜ 1 n ∑ c ∈ [ C ] , A⊂S c n A · ✶ ( v ∈ A ∧ |A ∩ L| < τ ) to denote the portion of data points containing v not-well-classified by feature set L . This leads to assumption:

Assumption 3.3. For any learned feature set L ⊂ S , if v 1 , v 2 ∈ S c for some class c ∈ [ C ] and h ( v 1 ; L ) = h ( v 2 ; L ) , then v 1 = v 2 . Also, for any class c ∈ [ C ] , there exists some τ distinct features v 1 , . . . , v τ ∈ S c such that g ( v 1 ; T j , T j ) , . . . , g ( v τ -1 ; T j , T j ) ≥ γ/n and g ( v τ ; T j , T j ) < γ/n .

This assumption leads to Lemma D.2, stating that the order in which features are learned within a class is deterministic. This is just for simplicity of presentation and can be relaxed. The last assumption is justified by the moderate number of data points in each chunk T j , ensuring the existence of both τ -1 learnable features and a non-learnable feature within a class. Throughout the following discussion, we will proceed under the above assumptions unless otherwise specified.

Notation. We denote a model at step s of the j -th experiment as f ( j,s ) . We denote the set of learned features and the set of memorized data for the model f ( j,s ) as L ( j,s ) and M ( j,s ) , respectively. We also define the set of data points with non-zero gradients at step s of the j -th experiment as N ( j,s ) . We define respective versions of these sets and the model, with different initialization methods, denoted by the subscripts (e.g., f ( j,s ) warm , f ( j,s ) cold , and f ( j,s ) ideal ). We emphasize that each method initializes f ( j, 0) , L ( j, 0) , M ( j, 0) , and N ( j, 0) differently at the start of the j -th experiment.

3.2 Prediction Process and Training Time

We provide a comparison of three initialization methods based on test accuracy and training time. To evaluate these metrics within our framework, we define the prediction process and training time.

Prediction Process. The model predicts unseen data points by comparing the learned features with features present in a given data point x . If the overlap between the learned feature set L and the features in x , denoted as V ( x ) , is at least τ , i.e., |V ( x ) ∩ L| ≥ τ , the model correctly classifies the data point. Otherwise, the model resorts to random guessing.

Training Time. Accurately measuring training time within our discrete learning framework is challenging. To address this, we introduce an alternative for training time of j -th experiment: the number of training data points with non-zero gradients at the start of j -th experiment, |N ( j, 0) | . This represents the amount of 'learning' required for the model to classify all data points correctly. We empirically validated this proxy in practical scenarios, as shown in Figures 6 and 7 in Appendix B.1. Additionally, Nakkiran et al. (2021) observe that in real-world neural network training, when other components are fixed, the training time increases with the number of data points to learn.

3.3 Comparison Between Warm-Starting and Cold-Starting in Our Framework

Now we analyze the warm-start and cold-start initialization methods within our framework, focusing on test accuracy and training time. We note that, by definition, L ( j, 0) cold and M ( j, 0) cold are both empty sets, while L ( j, 0) warm = L ( j -1 ,s j -1 ) warm and M ( j, 0) warm = M ( j -1 ,s j -1 ) warm , where s j denotes the last step of j -th experiment. Besides, we use a shorthand notation for step s j of the experiment j that we drop s if s = s j (e.g., L ( j ) := L ( j,s j ) ). For the detailed algorithms based on our learning framework, see Algorithms 3 and 4 in Appendix E.

In the test data, a feature combination A ⊂ S c of data point with class c ∈ [ C ] appears with probability n A /n along with data-specific noise. By Section 3.2, test accuracy for a learned set L and training time are defined as:

Based on these definitions, the following theorem holds:

Theorem 3.4. There exists nonempty G ⊊ S such that we always obtain L (1) warm = L (1) cold = G . For all J ≥ 2 , the following inequalities hold:

Proof Idea. After the first experiment, the data points in T 1 cannot further contribute to the learning process of the warm-started model. Consequently, even when a new data chunk is provided in subsequent experiments, the feature frequencies are too small, resulting in a weak signal strength of features that cannot overcome the noise signal strength. As a result, the model memorizes individual noise components of the new data points. This procedure is repeated with every experiment, causing the learned feature set to remain the same as at the end of the first experiment. In contrast, when receiving T 1: j at once (cold-starting), the signal strength of features is large enough to overcome the noise signal strength, allowing the model to learn many more features.

Theorem 3.4 highlights a trade-off between cold-starting and warm-starting. Regarding test accuracy, the theorem concludes that cold-starting can achieve strictly higher accuracy than warm-starting. However, warm-starting requires a strictly shorter training time compared to cold-starting.

Detailed proof is provided in Appendix D. Theorem 3.4 suggests that the loss of plasticity in the incremental setting under the stationary assumption can be attributed to the noise memorization process. A similar observation is made in real-world neural network training. It is widely believed that during the early stages of training, neural networks primarily focus on learning features from the dataset, and after learning these features, the model starts to memorize data points that it fails to classify correctly using the learned features. To investigate this phenomenon, we conducted an experiment where CIFAR-10 was divided into two chunks, each containing 50% of the training dataset. The model was pre-trained on one chunk and then further trained on the full dataset for 300 epochs. We used three-layer MLP and ResNet-18 with SGD optimizer across 10 random seeds.

Figure 2 shows the change in the model's performance based on the duration of pre-training. When pre-training is stopped at a certain epoch and the model is then trained on the full dataset, test accuracy is maintained. However, if pre-training continues beyond a specific threshold (approximately 50% pre-training accuracy in this case), warm-starting significantly impairs the model's performance as it increasingly memorizes training data points. We attribute this phenomenon to the neural network's memorization process after learning features. This is consistent with reports of a critical learning period where neural networks learn useful features in the early phase of learning (Achille et al., 2018; Frankle et al., 2020; Kleinman et al., 2024), and with findings that neural networks tend to learn features followed by memorizing noises (Arpit et al., 2017; Jiang et al., 2020). Using the same experimental settings as in Figure 2, we tested with a large-scale dataset, ImageNet-1k, and observed similar trends (see Figure 8 in Appendix B).

Remark 3.5. Igl et al. (2020) find that training a model on random labels followed by corrected labels results in better generalization compared to pre-training on a subset of correctly labeled data and then further training on the full dataset with the same distribution. Achille et al. (2018) also observe that pre-training with slightly blurred images followed by original images yields worse test accuracy than pre-training with random label or random noise images. These findings align with our observations: re-training with corrected labels after random label learning 'revives' gradients for most memorized data points, enabling new feature learning. Conversely, with static distributions, gradients for memorized data points remain suppressed, leading to learning from only a few data points with active gradients, causing memorization.

1

3

5

10

30

50

100

Pre-Train Epoch

56.5

57.0

57.5

58.0

58.5

Test Accuracy (%)

MLP

1

2

3

4

5

6

7

8

9

10

Pre-Train Epoch

62

63

64

65

66

67

68

69

70

ResNet-18

30

40

50

60

70

80

90

100

30

40

50

60

70

80

90

100

Pre-Train Accuracy (%)

Random (Test)

Warm (Test)

Warm (Pre-Train)

3.4 An Ideal Method: Retaining Features and Forgetting Noise

In Section 3.3, we observed a trade-off between warm-starting and cold-starting. Cold-starting often achieves better test accuracy compared to warm-starting, while warm-starting requires less time to converge. The results suggest that neither retaining all learned information nor discarding all learned information is ideal. To address this trade-off and get the best of both worlds, we consider an ideal algorithm where we retain all learned features while forgetting all memorized data points. For any experiment J ≥ 2 , if we consider the ideal initialization, learned features L ( J -1) ideal are retained, and memorized data points M ( J -1) ideal are reset to an empty set. Pseudo-code for this method is given in Algorithm 5, which can be found in Appendix E. We define T ( J ) ideal ≜ ∑ j ∈ [ J ] ∣ ∣ ∣ N ( j, 0) ideal ∣ ∣ ∣ as the training time with the ideal method, where N ( j, 0) ideal represents the set of data points having a non-zero gradient at the initial step of the j -th experiment. Then, we have the following theorem:

Theorem 3.6. For any experiment J ≥ 2 , the following holds:

The detailed proof is provided in Appendix D. The ideal algorithm addresses the trade-off between cold-starting and warm-starting. We conducted an experiment to investigate the performance gap between these initialization methods.

Synthetic Experiment. To verify our theoretical findings in more realistic scenarios, we conducted an experiment that more closely resembles real-world settings. Instead of fixing the frequency of each feature set, we sampled each feature's existence from a Bernoulli distribution to construct V ( x ) . This ensures that the experiment is more representative of real-world scenarios. Specifically, for each data point ( x , y ) , we uniformly sampled y ∈ { 0 , 1 } . From the feature set S y corresponding to the sampled class y , we sampled features where each feature's existence follows a Bernoulli distribution, ✶ ( v y,k ∈ V ( x )) ∼ Ber( p k ) , for all v y,k ∈ S y . This approach allows us to model the variability in feature occurrence that is commonly observed in real-world datasets while still maintaining the core principles of our learning framework. We set the number of features, K = 50 , with p k sampled from a uniform distribution, U(0 , 0 . 2) . Each chunk contained 1000 data points with total 50 experiments, with γ = 50 , τ = 3 . We sampled 10000 test data from the same distribution.

As shown in Figure 3, the results align with the above theorems. Random initialization, i.e. coldstarting, and ideal initialization achieve almost identical generalization performance, outperforming warm initialization. However, with warm initialization, the model converges faster, as evidenced by the number of non-zero gradient data points, which serves as a proxy for training time. Ideal initialization requires less time compared to cold-starting, which is also consistent with Theorem 3.6. Due to the sampling process in our experiment, we observe a gradual increase in the number of learned features and test accuracy in warm-starting, mirroring real-world observations. These findings remained robust across diverse hyperparameter settings (see Figures 9-11 in the Appendix B).

0

10

20

30

40

50

Number of Experiments

70

75

80

85

90

95

Test Accuracy (%)

Random

Warm

Ideal

0

10

20

30

40

50

Number of Experiments

20

30

40

50

60

70

80

90

Number of Learned Features

0

10

20

30

40

50

Number of Experiments

0

10000

20000

30000

40000

50000

Number of Non-Zero Gradient Data

4 DASH: Direction-Aware SHrinking

The ideal method recycles memorized training samples by forgetting noise while retaining learned features. From now on, we shift our focus to a practical scenario: training neural networks with real-world data. This brings up the question of whether such an ideal approach can be applied in real-world settings. To address this, we propose our algorithm, Direction-Aware SHrinking (DASH) , which intuitively captures this idea in practical training scenarios. The outlined behavior is illustrated in Figure 4. When new data is introduced, DASH shrinks each weight based on its alignment with the negative gradient of the loss calculated from the training data, placing more emphasis on recent data.

If the degree of alignment is small (i.e., the cosine similarity is close to or below 0), we consider that the weight has not learned a proper feature and shrinks it significantly to make it 'forget' learned information. This allows weights to forget memorized noises and easily change their direction. On the other hand, if the weight and negative gradient are well-aligned (i.e., the cosine similarity is close to 1), we consider it learned features and we shrink the weight to a lesser degree to maintain the learned information. This method aligns with the intuition of the ideal method, as it allows us to shrink weights that have not learned proper information while retaining weights that have learned commonly observed features.

-∇

Feature

Noise

θ

1

θ

2

˜

θ

1

˜

θ

2

θ

1

L

1:

j

-∇

θ

2

L

(Θ; /u1D4AF

1:

j

)

(Θ; /u1D4AF

)

Algorithm 1 Direction-Aware SHrinking (DASH)

Require:

Model f Θ with list of parameters Θ after the ( j -1) -th experiment

Training data points T 1: j

Averaging coefficient 0 < α ≤ 1

Threshold λ > 0

1: Initialize: G (0) θ ← 0 , ∀ θ in Θ 2: for i in 1 : j do 3: ℓ ← Loss( f Θ , T i ) 4: U Θ ← Gradient of loss ℓ 5: for θ in Θ do 6: G ( i ) θ ← (1 -α ) × G ( i -1) θ + α × U θ 7: end for 8: end for 9: for θ in Θ do 10: s θ ← CosineSimilarity ( -G ( j ) θ , θ ) 11: θ ← θ ⊙ max { λ, s θ } 12: end for 13: return model f Θ , initialized for the j -th experiment

The shrinking is done per neuron , where the incoming weights are grouped into a weight vector denoted as θ . For convolutional filters, the height and width of the kernel are flattened to form a single weight vector θ for each pair of input and output filters. DASH has two hyperparameters: λ and α . Hyperparameter λ is the minimum shrinkage threshold, as each weight vector is shrunk by max { λ, cos _ sim } , while α denotes the coefficient of exponential moving average of per-chunk loss gradients. Lower α value gives more weight to previous gradients, resulting in less shrinkage. This

is because gradients of the previously seen data usually have high cosine similarity with learned weights; low α is advantageous for simpler datasets where preserving learned features helps. Note that DASH is an initialization method applied only once when new data is introduced. The detailed algorithm is presented in Algorithm 1.

We discuss our intuition regarding the connection between DASH and the spirit of the ideal method. Generally, features from previous data are likely to reappear frequently in new data, as they are relevant to the data class. In contrast, noise from previous data rarely reappears as it is not classspecific and varies with each data point. As a result, the negative gradient of the loss naturally aligns more closely with learning features rather than memorizing noise from older data. This suggests that when neurons are aligned with its negative gradient of the loss, we can assume they have learned important features and should not be shrunk.

To validate our intuition, we plotted the accuracy on previously learned data points a few epochs after applying DASH in Figure 13, Appendix B. Our experiments show that DASH recovers training accuracy on previous datasets more quickly than other methods, likely because it preserves learned features while discarding memorized noise. As experiments progress, the growing number of learned features allows DASH to retain more information, leading to improved training accuracy across successive experiments. We further validate our intuition behind DASH in Figure 14 in Appendix B.

5 Experiments

5.1 Experimental Details

Our setup is similar to the one described in Ash and Adams (2020). We divided the training dataset into 50 chunks, and at the beginning of each experiment, a chunk is added to the existing training data. Models were considered converged and each experiment was terminated when training accuracy reached 99.9%, aligning with our learning framework. We conducted experiments with vanilla training i.e. without data augmentations, weight decay, learning rate schedule, etc. Appendix C.4 presents additional results on other settings, including the state-of-the-art (SoTA) settings that include the techniques mentioned above. We evaluated DASH on Tiny-ImageNet, CIFAR-10, CIFAR-100, and SVHN using ResNet-18, VGG-16, and three-layer MLP architectures with batch normalization layer. Models were trained using Stochastic Gradient Descent (SGD) and Sharpness-Aware Minimization (SAM) (Foret et al., 2020), both with momentum.

DASH was compared against baselines (cold-starting, warm-starting, and S&P (Ash and Adams, 2020)) and methods addressing plasticity loss under non-stationarity (L2 INIT (Kumar et al., 2023) and Reset (Nakkiran et al., 2021)). Layer normalization (Ba et al., 2016) and SAM (Foret et al., 2020), known to mitigate plasticity loss in reinforcement learning (Lee et al., 2023), were applied to both warm and cold-starting. Consistent hyperparameters were used across all methods, with details provided in Appendix C.3. S&P, Reset, and DASH were applied whenever new data was introduced. We report two metrics for both test accuracy and number of steps required for convergence: the value from the final experiment and the average across all experiments.

5.2 Experimental Results

We first experimented with CIFAR-10 on ResNet-18 to determine if methods from previous works for mitigating plasticity based on non-stationarity can be a solution to our incremental setting with stationarity. Appendix C.1 shows that L2 INIT, Reset, layer normalization, and reviving dead neurons, are not effective in our setting. Thus, we conducted the remaining experiments without these methods. Additionally, Table 1 shows that warm-starting with SAM does not outperform cold-starting with SAM, indicating that SAM alone is not an effective method in our case. Table 1 shows that DASH surpasses cold-starting (Random Init) and S&P in most cases. Training times were often shorter compared to training from scratch, and when longer, the performance gap in test accuracy was more pronounced. Omitted results are in Tables 3-6 located in Appendix C.2. Additionally, we confirm that DASH is computationally efficient, with details on the computation and memory overhead comparisons provided in Appendix C.5.

We argue that S&P can cause the model to forget learned information, including important features, due to shrinking every weight uniformly and perturbing weights. This leads to increased training time and relatively lower test accuracy, especially in SoTA settings (see Appendix C.4.1). In contrast, DASH addresses these issues by preserving learned features with direction-aware weight shrinkage.

Theorem 3.6 shows that ideal initialization can achieve the same test accuracy as cold-starting. Yet in practice, DASH surpasses cold-starting in test accuracy. This could be due to the difference between the discrete learning process in our framework and the continuous learning process in real-world neural network training. Even if features have already been learned, DASH can learn them in greater strength compared to learning from scratch by preserving previously learned features during training.

Further insights into the applicability of DASH can be found in Appendix C.4. We evaluate DASH in various settings beyond our original setup. In the SoTA setting, different observations are made: DASH achieves test accuracy close to (but does not outperform) cold-starting, without requiring additional hyperparameter tuning, which aligns more closely with our theoretical analysis. We demonstrate DASH's scalability on large-scale datasets such as ImageNet-1k. We also examine two additional practical scenarios: a data-discarding setting and a situation where new data are continuously added. In such cases, applying DASH with an interval (rather than upon every arrival of new data) proves effective. Finally, we explore DASH's behavior in non-stationary environments, specifically in Class Incremental Learning (CIL) with data accumulation settings.

6 Discussion and Conclusion

In this work, we defined an abstract framework for feature learning and discovered that warm-starting benefits from reduced training time compared to random initialization but can hurt the generalization performance of neural networks due to the memorization of noise. Motivated by these observations, we proposed Direction-Aware SHrinking (DASH), which shrinks weights that learned data-specific noise while retaining weights that learned commonly appearing features. We validated DASH in real-world model training, achieving promising results for both test accuracy and training time.

Loss of plasticity is problematic in situations where new data is continuously added daily, which is the case in many real-world application scenarios. Our research aimed to interpret and resolve this issue, preventing substantial waste of energy, time, and the environment. By elucidating the loss of plasticity phenomenon in stationary data distributions, we have taken a crucial step towards addressing challenges that may emerge in real-world AI, where the continuous influx of additional data is inevitable.

We hope our fundamental analysis of the loss of plasticity phenomenon sheds light on understanding this issue as well as providing a remedy. To generalize our findings to any neural network architecture, we treated the learning process as a discrete abstract procedure and did not assume any hypothesis class. Future research could focus on understanding the loss of plasticity phenomenon via optimization or theoretically analyzing it in non-stationary data distributions, such as in reinforcement learning.

References

Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations , 2018.

Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 , 2020.

Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning , pages 233-242. PMLR, 2017.

Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in neural information processing systems , 33:3884-3894, 2020.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks. arXiv preprint arXiv:2106.00042 , 2021.

Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems , 35:25237-25250, 2022.

Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Adelani, Pontus Lars Erik Saito Stenetorp, Sebastian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active forgetting. Advances in Neural Information Processing Systems , 36:31543-31557, 2023.

Yihe Deng, Yu Yang, Baharan Mirzasoleiman, and Quanquan Gu. Robust learning with progressive data expansion against spurious correlation. Advances in Neural Information Processing Systems , 36, 2023.

Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325 , 2021.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 , 2020.

Jonathan Frankle, David J Schwab, and Ari S Morcos. The early phase of neural network training. arXiv preprint arXiv:2002.10365 , 2020.

Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. arXiv preprint arXiv:2006.05826 , 2020.

Samy Jelassi and Yuanzhi Li. Towards understanding how momentum improves generalization in deep learning. In International Conference on Machine Learning , pages 9965-10040. PMLR, 2022.

Yiding Jiang, Christina Baek, and J Zico Kolter. On the joint interaction of models, data, and features. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=ze7DOLi394 .

Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206 , 2020.

Michael Kleinman, Alessandro Achille, and Stefano Soatto. Critical learning periods emerge even in deep linear networks. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=Aq35gl2c1k .

Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308.11958 , 2023.

Hojoon Lee, Hanseul Cho, Hyunseung Kim, Daehoon Gwak, Joonkee Kim, Jaegul Choo, SeYoung Yun, and Chulhee Yun. Plastic: Improving input and label plasticity for sample efficient reinforcement learning. Advances in Neural Information Processing Systems , 36, 2023.

Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C Machado. Curvature explains loss of plasticity. arXiv preprint arXiv:2312.00246 , 2023.

Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and sgd can reach them. Advances in Neural Information Processing Systems , 33:8543-8552, 2020.

Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Razvan Pascanu, James Martens, Hado van Hasselt, and Will Dabney. Towards perpetually trainable neural networks, 2023a.

Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning , pages 23190-23211. PMLR, 2023b.

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124003, 2021.

Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning , pages 16828-16847. PMLR, 2022.

Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems , 36, 2023.

Junsoo Oh and Chulhee Yun. Provable benefit of cutout and cutmix for feature learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview.net/forum?id=8on9dIUh5v .

Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference on machine learning , pages 2217-2225. PMLR, 2016.

Ruoqi Shen, Sébastien Bubeck, and Suriya Gunasekar. Data augmentation as feature manipulation. In International conference on machine learning , pages 19773-19808. PMLR, 2022.

Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning , pages 32145-32168. PMLR, 2023.

Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. Pretrained language model in continual learning: A comparative study. In International conference on learning representations , 2021.

Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. The benefits of mixup for feature learning. arXiv preprint arXiv:2303.08433 , 2023.

Contents

A Detailed Intuition and Example of Our Framework

In this section, we provide a detailed explanation of how our theoretical framework reflects the intuitive process of learning from image data. Our framework is designed to capture the characteristics of image data, where the input includes both relevant information for the image labels (which we refer to as ' features ' such as a cow's ears, eyes, tail, and mouth in Figure 5a) and irrelevant information (which we refer to as ' noise ' such as the sky or grass circled in red in Figure 5a). Our framework builds on the insights from Shen et al. (2022) and incorporates them into a discrete learning model.

Each data point is associated with

a combination of

features

x

1

x

2

x

3

x

4

x

5

x

6

x

6

x

1

If all features are not frequent enough,

the model

memorizes

remaining data

Data points with su

ffi

ciently many

learned features are

well-classified

At each step, the model learns

the most frequent feature

: Learned feature

Frequency of features

evaluated

solely on not-well-classified data points

: Well-classified data

: Memorized data

Before explaining the intuition of our learning framework, let us connect notation from Section 2 with Figure 5. Figure 5 shows six data points from a single cow class, where training set T = { ( x i , y ) } i ∈ [6] . Features are denoted by their color's first letter: violet as v , blue as b , green as g , mint as m , yellow as y , and pink as p . The complete feature set S is therefore { v , b , g , m , y , p } . The learned feature set L is empty at initialization, and the set of data points with non-zero gradients N ⊂ T is equal to T at initialization. We set the data learning threshold τ = 2 and noise strength γ = 2 .

Our training process is based on the idea that features which occur more frequently in the training data are easier to learn, as the gradients tend to align more strongly with these frequent features. Consequently, our framework is designed to learn the most frequent features in a sequential manner. For example, a cow's ears, shown in violet in Figure 5 would be selected first since it is the most frequent feature:

where L ( s ) and N ( s ) represent the learned feature set and the set of data points with non-zero gradients, respectively, at step s . This holds because:

for all c ∈ { b , g , m , y , p } . Since we have set γ to 2, the feature v is learned because its frequency is sufficiently high compared to the noise, i.e., g (v; T , N (0) ) ≥ γ/ |T | = 2 6 . Also, since τ set to 2, no data point is considered well-classified at this stage, i.e., |{ v } ∩ V ( x i ) | < τ for all i ∈ [6] . As a result, L (1) = { v } , N (1) = T .

In the next step, the model learns feature b in the same way. Therefore, x 1 , x 3 , and x 4 are now considered well-classified since they share two overlapping features with the learned feature set L (2) = { v , b } . This indicates that the model accumulates enough information to accurately classify these cow images as cows. Once the model correctly classifies data based on these features, the influence of those data points on the learning process diminishes, as the loss gradients become smaller as the predictions become more confident. To account for this, our framework evaluates the frequency of features only on data points that are not yet well-classified. Therefore, the set of data points with non-zero gradients is now updated to N (2) = { ( x i , y ) } i ∈{ 2 , 5 , 6 } . Subsequently, we compute the gradient proxy g ( c ; T , N (2) ) using only the data points in N (2) . In the third step, considering the features included in { x 2 , x 5 , x 6 } , we get that the feature g is the most frequent among those in S \ L (2) , and the feature meets the threshold for learning. Hence, the feature g is newly learned, leading us to L (3) = { v , b , g } and N (3) = { ( x 6 , y ) } .

As training progresses, the algorithm may reach a point where the remaining features in the not-wellclassified data points are too infrequent to be learned effectively. At this point, the noise in each data point has a faster learning speed, which leads the model to memorize noise instead of learning the features to achieve 100% training accuracy. In the example described in Figure 5, at the fourth step, the remaining feature m in x 6 does not meet the threshold: g (m; T , N (3) ) = 1 / 6 < γ/ |T | , so the model is unable to learn feature m . Consequently, the model shifts to memorizing the noise in the remaining data point x 6 and completes training by memorizing the noise.

B Connection Between Our Framework and Practical Scenarios

This section demonstrates how our theoretical framework relates to real-world scenarios. Section B.1 presents real-world experimental evidence supporting our theoretical framework. Section B.2 explores the robustness of our synthetic experiments across various hyperparameters. Finally, Section B.3 provides empirical validation of the key intuitions behind DASH.

B.1 Justification of Our Framework

Figure 6 shows that the initial gradient norm of training data, |N ( j, 0) | , can be a proxy for the training time until convergence. As the initial gradient norm increases, the number of steps required for convergence also increases. While this figure uses the gradient norm instead of the number of non-zero gradient data points due to the continuous nature of real-world neural network training, we believe it resembles the behavior of non-zero gradient data points. Additionally, Figure 7 demonstrates that the number of steps required for convergence increases as the number of data points increases in various datasets.

40

60

80

100

120

140

Initial Gradient Norm of Training Data

0

500

1000

1500

2000

2500

3000

Number of Steps

Warm

0

250

500

750

1000 1250 1500 1750

Initial Gradient Norm of Training Data

0

2000

4000

6000

8000

10000

12000

Random

10

20

30

40

50

Number of Experiments

0

10

20

30

40

50

Number of Experiments

0

5000

10000

15000

20000

25000

30000

Number of Steps

F-MNIST

0

10

20

30

40

50

Number of Experiments

CIFAR-10

0

10

20

30

40

50

Number of Experiments

CIFAR-100

0

10

20

30

40

50

Number of Experiments

T-ImageNet

We also investigate the effect of the pretrain epoch on warm-starting in ImageNet-1k classification with ResNet18, similar to the setup in Figure 2 in Section 3.3. We conducted experiments using different pretrain epochs: 30, 50, 70, and 150. Figure shows a declining trend in accuracy for warm-started models as pretrain epoch increases. Interestingly, we noted a similar phenomenon in Figure 2 in Section 3.3, where warm-starting does not negatively impact test accuracy when training resumes from earlier pretrained epochs, such as 30 in this case. This observation aligns with our

theoretical framework, which suggests that neural networks tend to learn meaningful features first and then begin to memorize noise later. We believe this phenomenon is persistent across different datasets, model architectures, and optimizers.

150

100

50

0

50

100

150

Epochs

0

20

40

60

80

100

150

100

50

0

50

100

150

Epochs

0

10

20

30

40

50

Test Accuracy (%)

Cold

30 Pretrain

50 Pretrain

70 Pretrain

150 Pretrain

B.2 Further Results on Warm-starting vs. Cold-starting

We conducted synthetic experiments across a wide range of hyperparameters. Figure '9 uses the same setup as Section 3.4 but varies numbers of classes, C . Figure 10 investigates varying noise signal strengths, γ , while Figure 11 explores different values of τ . These results align with our findings from Theorems 3.4 and 3.6.

C

0

10

20

30

40

50

Number of Experiments

40

50

60

70

80

90

Test Accuracy (%)

Random

Warm

Ideal

0

10

20

30

40

50

Number of Experiments

20

40

60

80

100

120

Number of Total Learned Features

0

10

20

30

40

50

Number of Experiments

0

10000

20000

30000

40000

50000

Number of Non-Zero Gradient Data

0

10

20

30

40

50

Number of Experiments

20

30

40

50

60

70

80

90

Test Accuracy (%)

0

10

20

30

40

50

Number of Experiments

0

50

100

150

200

Number of Total Learned Features

C

=5

0

10

20

30

40

50

Number of Experiments

0

10000

20000

30000

40000

50000

Number of Non-Zero Gradient Data

0

10

20

30

40

50

Number of Experiments

10

20

30

40

50

60

70

80

90

Test Accuracy (%)

0

10

20

30

40

50

Number of Experiments

0

50

100

150

200

250

300

350

Number of Total Learned Features

C

=10

0

10

20

30

40

50

Number of Experiments

0

10000

20000

30000

40000

50000

Number of Non-Zero Gradient Data

C

=3

=100

20000

0 10 20 30 40 50 Number of Experiments 0 5 0 10 20 30 40 50 Number of Experiments 0 200 0 10 20 30 40 50 Number of Experiments 0 10000 As stated in Section 2, we posited that τ could serve as a proxy for dataset complexity. Figure 11 shows that as τ increases, the threshold for considering a data point well-classified also increases, making it more difficult to correctly predict unseen data points. This difficulty is particularly pronounced for warm-starting, leading to a widening gap between the random initialization and warm initialization methods. Additionally, this phenomenon is observed in real-world neural network training, as depicted in Figure 12. For datasets with higher complexity (from left to right), the gap between the two initialization methods widens, exhibiting the same trend as an increasing τ .

0

10

20

30

40

50

Number of Experiments

88

90

92

94

Test Accuracy (%)

Random

Warm

Ideal

0

10

20

30

40

50

Number of Experiments

60

70

80

90

100

Number of Total Learned Features

=10

0

10

20

30

40

50

Number of Experiments

0

10000

20000

30000

40000

50000

Number of Non-Zero Gradient Data

0

10

20

30

40

50

Number of Experiments

75.0

77.5

80.0

82.5

85.0

87.5

90.0

92.5

95.0

Test Accuracy (%)

0

10

20

30

40

50

Number of Experiments

30

40

50

60

70

80

90

100

Number of Total Learned Features

=30

0

10

20

30

40

50

Number of Experiments

0

10000

20000

30000

40000

50000

Number of Non-Zero Gradient Data

0

10

20

30

40

50

Number of Experiments

65

70

75

80

85

90

95

Test Accuracy (%)

0

10

20

30

40

50

Number of Experiments

20

30

40

50

60

70

80

90

Number of Total Learned Features

=50

0

10

20

30

40

50

Number of Experiments

0

10000

20000

30000

40000

50000

Number of Non-Zero Gradient Data

20000

0

10

20

30

40

50

Number of Experiments

0

10

20

30

40

50

Number of Experiments

0

10

20

30

40

50

Number of Experiments

0

10000

0

10

20

30

40

50

Number of Experiments

90

92

94

96

Test Accuracy (%)

Random

Warm

Ideal

0

10

20

30

40

50

Number of Experiments

10

20

30

40

Number of Total Learned Features

0

10

20

30

40

50

Number of Experiments

0

10000

20000

30000

40000

50000

Number of Non-Zero Gradient Data

0

10

20

30

40

50

Number of Experiments

70

75

80

85

90

95

Test Accuracy (%)

0

10

20

30

40

50

Number of Experiments

20

30

40

50

60

70

80

Number of Total Learned Features

=2

0

10

20

30

40

50

Number of Experiments

0

10000

20000

30000

40000

50000

Number of Non-Zero Gradient Data

0

10

20

30

40

50

Number of Experiments

60

65

70

75

80

85

90

95

Test Accuracy (%)

0

10

20

30

40

50

Number of Experiments

20

30

40

50

60

70

80

90

Number of Total Learned Features

=3

0

10

20

30

40

50

Number of Experiments

0

10000

20000

30000

40000

50000

Number of Non-Zero Gradient Data

=4

0

10

20

30

40

50

Number of Experiments

0

10

20

30

40

50

Number of Experiments

0

10

20

30

40

Number of Experiments

0

10000

0

10

20

30

40

50

Number of Experiments

80.0

82.5

85.0

87.5

90.0

Test Accuracy (%)

Random

Warm

0

10

20

30

40

50

Number of Experiments

30

40

50

60

0

10

20

30

40

50

Number of Experiments

5

10

15

20

25

30

35

CIFAR-100

0

10

20

30

40

50

Number of Experiments

5

10

15

20

25

T-ImageNet

50

50

B.3 Experiments Supporting the Intuition behind DASH

To validate whether previously learned/memorized data points do not have large gradients when further trained with a combined dataset (existing + newly introduced data) using warm-starting, we plotted the train accuracy on the previous dataset for the first few epochs. Using ResNet-18 on CIFAR-10 (Figure 13), we found that warm-starting preserves performance on previously learned data even when training continues with combined datasets, supporting the main idea of Theorem 3.4.

To verify whether DASH truly captures our intuitions from the ideal algorithms, we conducted an experiment using CIFAR-10 trained on ResNet-18, with the same experimental settings. Figure 13 demonstrates that when applying DASH, the train accuracy on previous datasets increases more rapidly after a few epochs compared to other methods. We argue that this behavior stems from our algorithm's ability to forget memorized noise while preserving learned features. As the number of experiments increases, the number of learned features also grows. For a fair comparison, we used λ = 0 . 05 for DASH, and when performing S&P and shrink, we shrank each weight by multiplying 0 . 05 . In the case of S&P, after shrinking, we added noise sampled from N (0 , 0 . 01 2 ) .

0

10

20

30

40

50

Number of Experiments

0.2

0.4

0.6

0.8

1.0

Train Accuracy of Previous Dataset

0

10

20

30

40

50

Number of Experiments

0

10

20

30

40

50

Number of Experiments

Random

Warm

DASH

S&P

Shrink

We further validate our intuition that cosine similarity between the negative loss gradient and model weights can indicate whether the model has indeed learned features. We trained a 3-layer CNN on CIFAR-10, varying the size of the training dataset. We observed that the cosine similarity between the negative gradient from the test data and the learned filters increases as the training dataset size grows. The model trained with more data appears to learn more features, as evidenced by the rising trend in test accuracy, shown by the dashed line in Figure 14. This suggests that cosine similarity can capture whether the weights have learned features, which aligns with our expectations.

1000

10000

25000

40000

50000

Train Data Size

0.2

0.4

0.6

0.8

1.0

Cosine Similarity

20

30

40

50

60

70

80

Test Accuracy (%)

C Omitted Experimental Results

This section presents additional experimental results and their corresponding hyperparameters. Appendix C.1 examines non-stationary solutions, demonstrating their limitations in stationary settings. The following section compares DASH against other baselines in various settings, excluding those methods from Appendix C.1 due to their poor performance in stationary conditions. Appendix C.3 provides a comprehensive list of hyperparameters used across all experiments from Section 5, as well as Appendix C.1 and C.2. Additionally, we validate DASH's applicability including state-of-the-art (SoTA) settings in Appendix C.4, and analyze computational and memory overhead in Appendix C.5.

We ran our experiments on two distinct hardware setups. The first server had an Intel Gaudi-v2 HPU with 96GB of VRAM with four Intel Xeon Platinum 8380 40-core CPUs. The second server had an NVIDIA A6000 GPU with 48GB of VRAM, paired with two AMD EPYC 7763 64-core CPUs. We compared the computing performance of these two hardware setups in Appendix C.6.

Unless specified otherwise, we follow the experimental settings outlined in Section 5.1. We conducted an experiment with an incremental training dataset comprised of 50 chunks. At the start of each experiment, a new chunk is provided and added to the existing training dataset. Before proceeding to the next experiment, the model is trained until achieving 99.9% train accuracy.

C.1 Methods for Non-Stationary Data Distribution Struggle in Stationary Settings

In this subsection, we describe solutions that aim to mitigate plasticity loss under non-stationarity, which cannot remedy the loss of plasticity in an incremental setting with a stationary data distribution. Table 2 shows L2 INIT (Kumar et al., 2023) and Reset (Nikishin et al., 2022) cannot be a solution in our setting.

Furthermore, applying layer normalization cannot close the gap between cold-starting and warmstarting; rather, the gap increases, as shown in Figure 15. Also, Nikishin et al. (2022) and Sokar et al. (2023) state that loss of plasticity in non-stationary data distributions arises from inactive neurons in the model. However, this is not the case in our setting, as demonstrated in Figure 16.

0

10

20

30

40

50

Number of Experiments

30

40

50

60

70

Test Accuracy (%)

Cold

Warm

BN

LN

0

10

20

30

40

50

Number of Experiments

0.050

0.025

0.000

0.025

0.050

Dead Neuron Ratio

Block 1

0

10

20

30

40

50

Number of Experiments

Block 2

0

10

20

30

40

50

Number of Experiments

Block 3

0

10

20

30

40

50

Number of Experiments

Block 4

C.2 Results of Experiments Across Different Datasets, Models, and Optimizers

This subsection presents our complete experimental results (Tables 3-6), following the settings detailed in Section 5.1. We evaluated multiple approaches, including 'Warm ReM' (warm-starting with momentum reset), which also proved ineffective. We conducted comprehensive comparisons across multiple datasets (Tiny-ImageNet, CIFAR-10, CIFAR-100, and SVHN), architectures (ResNet18, VGG-16, three-layer MLP), and optimizers (SGD and SGD-based SAM). The results demonstrate that DASH consistently outperforms baseline methods - warm-starting, cold-starting, and S&P while often requiring less training time. Results are averaged across five random seeds, except for Tiny-ImageNet which uses three random seeds due to its high computational cost.

C.3 Hyperparameters

In this section, we provide the details of the hyperparameters used in our experiments from Section 5, as well as Appendix C.1 and C.2. Additionally, we present heatmaps illustrating the results for a wide range of two hyperparameters, α and λ , in DASH. The heatmaps in Figure 17 suggest that DASH exhibits robustness to hyperparameter variations, indicating that its performance is less affected by the choice of hyperparameter values.

We fixed the momentum to 0.9 and the batch size to 128. The learning rate is set to 0.001 for training ResNet-18, and for other models, a learning rate of 0.01 is used. The value of ρ for SAM is chosen based on the performance of cold-starting. The default value of α = 0 . 3 is used, and we did not change this value frequently. The perturbation parameter σ used in the Shrink & Perturb (S&P) procedure is set to 0 . 01 , as this value is considered optimal for perturbation, as described in Ash and Adams (2020). Initially, we tested σ = 0 . 1 as the perturbation parameter, since Ash and Adams (2020) reported slightly better test accuracy compared to σ = 0 . 01 in some cases. However, we experienced significantly poorer generalization performance with σ = 0 . 1 compared to σ = 0 . 01 , as shown in Figure 18. The hyperparameters used in our experiments are described in Table 7.

0.05

0.1

0.3

0.1

0.3

0.7

1

83.77

83.89

83.42

84.28

83.69

83.42

83.98

83.79

83.17

83.72

83.88

83.21

(0.25)

(0.23)

(0.26)

(0.38)

(0.29)

(0.42)

(0.48)

(0.36)

(0.22)

(0.40)

(0.17)

(0.24)

Test Acc at Last Experiment

0.05

0.1

0.3

0.1

0.3

0.7

1

75.00

74.99

74.70

75.01

74.89

74.76

74.99

74.85

74.73

74.91

74.97

74.59

(0.06)

(0.08)

(0.17)

(0.10)

(0.03)

(0.04)

(0.05)

(0.10)

(0.05)

(0.10)

(0.05)

(0.06)

AVG Test Acc across All Experiments

0.05

0.1

0.3

0.1

0.3

0.7

1

3454

3325

2953

3450

3231

2954

3384

3290

2953

3368

3276

2884

(60)

(57)

(60)

(54)

(7)

(77)

(124)

(33)

(74)

(49)

(85)

(139)

83.2

83.4

83.6

83.8

84.0

84.2

74.60

74.65

74.70

74.75

74.80

74.85

74.90

74.95

75.00

2900

3000

3100

3200

3300

3400

0.1

0.01

0.05

0.1

0.3

59.16

79.65

59.13

80.12

58.52

81.27

(0.70)

(0.23)

(0.70)

(0.05)

(0.24)

(0.03)

Test Acc at Last Experiment

0.1

0.01

0.05

0.1

0.3

50.36

70.89

50.32

71.00

49.91

71.66

(0.08)

(0.04)

(0.16)

(0.03)

(0.13)

(0.06)

AVG Test Acc across All Experiments

0.1

0.01

0.05

0.1

0.3

7830

2884

7803

2864

8188

2777

(132)

(31)

(114)

(31)

(84)

(45)

60

65

70

75

80

50.0

52.5

55.0

57.5

60.0

62.5

65.0

67.5

70.0

3000

4000

5000

6000

7000

8000

C.4 Discussions on the Broader Applicability of DASH

In this subsection, we explore how DASH performs in a variety of settings, including state-of-the-art (SoTA) configurations, larger datasets, and scenarios where previous data cannot be stored. We also examine how DASH compares to other methods when data is continuously introduced throughout training, as well as its effectiveness in non-stationary data distribution environments, such as Class Incremental Learning (CIL) setups.

C.4.1 Performance in State-of-the-Art Settings

In the state-of-the-art (SoTA) setting, we employed weight decay and standard data augmentation techniques, such as horizontal flipping and random cropping. We also used a learning rate scheduler that reduces the learning rate step-wise by a factor of 0.2 at 60, 120, and 200 epochs. By applying the learning rate scheduler, there is no need to compare training time since training is completed at roughly the same epoch across all experiments. The weight decay was set to 0.0005, and the initial learning rate was set to 0.1. All other settings remain the same as mentioned above. We tested this setup on CIFAR-10 and CIFAR-100 using the ResNet-18 architecture.

The results in Table 8 show that DASH performs similarly to or slightly worse than starting from random initialization. It appears that this is partly because all hyperparameters are tuned to maximize

the performance of cold-starting, to achieve the (close-to-)SoTA test accuracy numbers. Due to the lack of computational resources, we were unable to tune hyperparameters specifically for DASH.

Furthermore, we believe this aligns more closely with our theoretical anylsis in Theorem 3.6, as the hyperparameters are tuned to allow the model to learn as many features as possible, making it difficult for DASH to outperform cold-starting.

Moreover, we observe that S&P cannot be used in these SoTA settings. We believe this is due to the nature of S&P, which shrinks all weights, while the SoTA setting is likely designed to avoid learning unuseful features, unlike the previous setting. Consequently, it is plausible that retaining learned features is more important than forgetting them, making S&P unsuitable for SoTA settings. Although DASH performs slightly worse than cold-starting, it is conceivable that it is better at retaining features compared to S&P and other warm-starting methods, resulting in better overall performance.

The gap between warm-starting and cold-starting has been significantly reduced, likely due to data augmentation techniques and the increase in learning rate when new data is introduced. Data augmentation techniques increase the amount of feature information, allowing warm-starting to learn features that vanilla training (without augmentation) cannot (Shen et al., 2022). Furthermore, as the learning rate is set to a higher value at the beginning of each new experiment, the model can forget previously memorized data points and escape spurious minima that were difficult to escape from, which is consistent with the findings of Berariu et al. (2021). Despite these improvements, a gap still exists between warm-starting and cold-starting.

C.4.2 Scalability on Large Datasets

We validate the scalability of DASH for larger datasets such as ImageNet-1k. However, conducting experiments for such datasets is challenging, as it would require repeating the training process 50 times until convergence-an extremely time-consuming process. As an alternative, we trained ImageNet-1K on ResNet18 using a setup similar to Figure 2 in Section 3.3. Our setup involved pretraining on 50% of the data before fine-tuning on the complete dataset. We used shrinkage parameter λ = 0 . 3 for both DASH and S&P. Results shown in Figure 19 demonstrate that DASH achieves superior performance compared to all baseline methods-including cold initialization, warm initialization, and S&P-both in terms of test accuracy and convergence speed. Notably, DASH achieves faster convergence and marginally better test accuracy than S&P, demonstrating its effectiveness even on challenging large-scale datasets.

C.4.3 Effectiveness in Data-discarding Setting

To explore how our algorithm can be applied to a broader range of scenarios further, we consider situations where storing all data is not feasible (e.g., due to memory constraints), rather than accumulating and retaining data. Specifically, we divided the CIFAR-10 dataset into 50 chunks and

150

100

50

0

50

100

150

Epochs

0

20

40

60

80

100

150

100

50

0

50

100

150

Epochs

0

10

20

30

40

50

Test Accuracy (%)

Cold

Warm

S&P

DASH

experimented with ResNet-18, running each experiment on a single chunk before moving on to the next. All other settings remained consistent with the details provided in Section 5.1.

We configured our experiments to apply DASH at specific intervals (e.g., every 10, 15, or 20 experiments) instead of after each one. Since there is no previous data available in this scenario, we set α = 0 . After the 40th experiment, when the model had sufficiently learned, we stopped applying the shrinking process. We also conducted similar experiments with the S&P method for comparison. It's important to note that this variant is feasible because both DASH and S&P focus on adjusting the model's initialization.

We tested DASH and S&P with intervals of 10, 15, and 20 epochs. For both methods, we explored shrinkage parameter ( λ ) of 0.05, 0.1, and 0.3. We plotted the results using the best hyperparameters for each method in Figure 20. Notably, DASH's test accuracy consistently exceeded that of S&P across all hyperparameter configurations, as shown in Figure 21. These findings demonstrate that DASH outperforms both the warm-starting baseline and the S&P method in terms of test accuracy. Based on this evidence, we can conclude that DASH is well-suited for data-discarding scenarios.

0

10

20

30

40

50

Number of Experiments

30

40

50

60

70

Test Accuracy (%)

Warm

S&P

DASH

0

10

20

30

40

50

Number of Experiments

50

100

150

Number of Steps

S&P

DASH

C.4.4 Performance with Continuously Added Data Setting

We designed an experiment mimicking real-world scenarios where new data arrives continuously during training. Following Igl et al. (2020)'s approach, we sampled new data randomly each epoch for the first 500 epochs, combining it with the existing dataset, then continued training for another 500 epochs. This setting assumes a scenario where data continuously arrives before the model has fully converged, as it often does in real-world situations. Cold-started models were trained for 1000 epochs using the entire dataset from the beginning. We used the CIFAR-10 dataset with ResNet18 across five random seeds. We applied both DASH and S&P every 50 epochs through the first 500 epochs with shrinkage parameter λ = 0 . 3 for both methods, following a similar setup to Figure 20.

0

200

400

600

800

1000

Number of Epochs

0

25

50

75

100

Cold

Warm

S&P

DASH

0

200

400

600

800

1000

Number of Epochs

0

20

40

60

80

Test Accuracy (%)

As shown in Figure 22, while there is a clear performance gap between warm-starting and cold-starting approaches, both DASH and S&P aim to address this difference. Despite using the same shrinkage value of 0.3, DASH successfully preserved learned features while S&P did not. This difference is reflected in the training accuracy: DASH showed steady improvement as training progressed, while S&P struggled during the shrinking phases. As a result, DASH not only converged faster but also achieved higher test accuracy, while S&P lost the convergence advantage that warm-starting provides.

C.4.5 Experiments on Class Incremental Learning Setting

Experiments so far focused exclusively on stationary data distributions. To broaden our understanding, we explored DASH's performance in non-stationary settings, particularly in Class Incremental Learning (CIL) - an important area in continual learning where data distributions shift over time.

We began with experiments on CIFAR-10, dividing the dataset into 10 chunks, each representing a distinct class. The model, based on ResNet-18, was introduced to one chunk at the start of each experiment and trained without access to previously encountered data. However, this setup proved challenging; even with warm-starting as a baseline, the model simply overfitted to each new class, resulting in only 10% test accuracy across all experiments, equivalent to random guessing.

Given these limitations, we adjusted our experimental setup while preserving the non-stationary nature of the task. Using the same configurations with above, instead of completely discarding previous data, we accumulated data over time. Thus, each new chunk was combined with previously seen data. During evaluation, we tested the model only on classes it had encountered during training.

This revised approach yielded more promising results, as shown in Figure 23. DASH surpasses other baselines in terms of test accuracy despite requiring longer convergence times. While these results demonstrate DASH's potential in certain non-stationary environments, we recognize that our modified setup simplifies true non-stationarity, such as in continual learning scenarios.

1

2

3

4

5

6

7

8

9

10

Number of Experiments

60

70

80

90

100

Test Accuracy (%)

Cold

Warm

S&P

DASH

1

2

3

4

5

6

7

8

9

10

Number of Experiments

0

2000

4000

6000

8000

Number of Steps

C.5 Computation and Memory Overhead Comparison

In this subsection, we will compare the computational and memory overhead of each method. It is important to note that DASH is applied only once when new data is introduced. Since DASH calculates the gradient of the whole dataset just once, its memory complexity is proportional to the batch size and model size. When comparing experiments with the same number of epochs, we can think of DASH as adding approximately one extra epoch to the total training time. Similarly, the computational overhead of DASH is roughly equivalent to running one additional epoch.

We provide the experimental results of the computation and memory overhead for each method in Table 9. The experiments were conducted under the same conditions described in Section 5.1 on CIFAR-10, using ResNet-18 with the SGD optimizer. We provide aggregated results for FLOPS, CPU/CUDA memory usage, and training time on an NVIDIA A6000. These measurements were obtained using the torch.profile library, with values summed across all operations. We only profiled two out of the 50 experiments until convergence (99.9% training accuracy) because profiling adds significant overhead to the training process, and running it continuously slows down the entire training pipeline. A comparison of the total training times of each method for all 50 experiments without profiling is provided in Section C.6. Table 9 indicates that S&P requires two more training epochs than DASH. Since DASH only adds approximately one extra epoch of overhead, the results show that DASH's total computational and memory overhead is about one epoch less than S&P's.

C.6 Comparison between NVIDIA A6000 and Intel Gaudi-v2

We conducted experiments on two different hardware platforms: Intel Gaudi-v2 HPUs and NVIDIA A6000 GPUs. In this section, we compare their implementation code and training times. Using CIFAR-100 and ResNet-18, we measured the training times on both platforms while maintaining the same experimental settings as described in Section 5.1 with SGD optimizer. It is important to note that no hardware-specific optimizations were performed on either platform, and all hyperparameters were kept constant.

To ensure a fair comparison, we ported our NVIDIA implementation to Intel Gaudi-v2, keeping the other settings consistent. The porting process was straightforward, requiring only about three extra lines of code, as shown in the example below. The full Gaudi-v2 implementation can be found in our GitHub repository at https://github.com/NAVER-INTEL-Co-Lab/gaudi-dash .

In Table 10, we present the results of the training time comparison until convergence. The table shows that the Intel Gaudi-v2 is slightly faster than the NVIDIA A6000 across all initialization methods. For instance, the Intel Gaudi-v2 achieves a speedup of 1 . 16 × over the NVIDIA A6000 with cold initialization.

D Proof of Theorems

This section provides the proof for Theorems 3.4 and 3.6, stated in Section 3, respectively. Before presenting the main proof, we state some technical lemmas.

Lemma D.1. For any learned feature set A , B ⊂ S that satisfies A ⊊ B and |A ∩ S c | ≥ τ -1 for any c ∈ [ C ] , then we have ACC( A ) < ACC( B ) .

Proof of Lemma D.1. Since A ⊊ B , it is trivial that for any c ∈ [ C ] and Λ ⊂ S c , we have

✶ ( | Λ ∩ A| < τ ) ≥ ✶ ( | Λ ∩ B| < τ ) . (1) From the given condition, we can choose c ∗ ∈ [ C ] such that there exists τ -1 distinct features v 1 , . . . , v τ -1 ∈ A ∩ S c ∗ and v τ ∈ ( B ∩ S c ∗ ) \ ( A∩S c ∗ ) . Our choice of Λ ∗ ≜ { v 1 , . . . , v τ } ⊂ S c ∗ satisfies

✶ ✶ From (1), (2), and the definition of ACC( · ) , we have

For ease of presentation, let us say 'a model learns NA ' if a model cannot learn any features. For example, if a model learns u 1 , · · · , u s ∈ S during s steps of training process and feature learning process ends in ( s +1) -th step, let us say that we learn u 1 , · · · , u s , NA , NA , · · · .

Using the notion above, we prove that our learning process uniquely determines the behavior within the same class regardless of the randomness of the training process, where the randomness may come from tie-breaking that can happen in the choice of the most frequent non-learned feature.

Lemma D.2. Suppose we train two models with different randomness on T 1: j for some j ∈ N starting from a learned set L and without any memorized data. We use u s and u ′ s to denote features learned in s -th step of training process by two models, respectively. The i -th learned feature within class c ∈ [ C ] is denoted as u c,i for the first model and u ′ c,i for the second model. Then, u c,i = u ′ c,i for all c ∈ [ C ] and i ∈ N .

̸

Proof of Lemma D.2. Suppose there exists some class c ∈ [ C ] and i ∈ N such that u c,i = u ′ c,i and choose one with the smallest i . Without loss of generality, we may assume u ′ c,i = NA . Then, we have

̸

̸

The first equality holds since { u c, 1 , . . . , u c,i -1 } = { u ′ c, 1 , . . . , u ′ c,i -1 } from our choice of c, i and the second equality holds since the second model learns u ′ c,i . The last inequality holds since u ′ c,i = NA . Hence, u c,i = NA and

̸

From our Assumption 3.3 and since { u c, 1 , . . . , u c,i -1 } = { u ′ c, 1 , . . . , u ′ c,i -1 } , we have u c,i = u ′ c,i . This is contradictory and we have our desired conclusion.

Lemma D.3. Suppose we train two models on T 1: j 1 and T 1: j 2 for some j 1 > j 2 starting from a learned set L and without any memorized data. We use u s and u ′ s to denote features learned in s -th step of the training process by two models trained on T 1: j 1 and T 1: j 2 , respectively. The i -th learned feature within class c ∈ [ C ] is denoted as u c,i for the first model and u ′ c,i for the second model. Then, u c,i = u ′ c,i or u ′ c,i = NA for all c ∈ [ C ] and i ∈ N .

̸

Proof of Lemma D.3. Suppose there exists some class c ∈ [ C ] and i ∈ N such that u c,i = u ′ c,i and u ′ c,i = NA . Choose one with the smallest i . Since u ′ c,i = NA and from our choice of c and i , we have

̸

̸

̸

Hence, u c,i = NA and

From our Assumption 3.3 and since { u c, 1 , . . . , u c,i -1 } = { u ′ c, 1 , . . . , u ′ c,i -1 } , we have u c,i = u ′ c,i . This is contradictory and we have our desired conclusion.

With above Lemma D.1, D.2 and D.3, we have the following theorems.

Theorem 3.4. There exists nonempty G ⊊ S such that we always obtain L (1) warm = L (1) cold = G . For all J ≥ 2 , the following inequalities hold:

Furthermore, ACC( L ( J ) warm ) < ACC( L ( J ) cold ) holds when J > γ δn where δ ≜ max v ∈S\G h ( v ; G ) > 0 .

Proof of Theorem 3.4. By Lemma D.2 for the case j = 1 , we immediately have our first conclusion by defining G as a learned feature set from the first experiment. Furthermore, we have G ⊊ S c and |G ∩ S c | ≥ τ -1 for any class c ∈ [ C ] from our feature learning framework and Assumption 3.3.

We want to show that for any J ≥ 2 , L ( J ) warm = G . Since we never forget the learned feature in warm training, it is clear that G = L (1) warm ⊂ L ( J ) warm . We may assume that the existence of J ∗ ≥ 2 such that L (1) warm ⊊ L ( J ∗ ) warm and choose the smallest J ∗ ≥ 2 . Then, in the first step of J ∗ -th experiment, a model learns some feature u . From our training process, u satisfies

and since J ∗ denotes the first experiment that can learn beyond G , L ( J ∗ -1) warm = G and

It means that u must have been already learned in the ( J ∗ -1) -th experiment and it is contradictory.

Thus, we have L ( J ) warm = L (1) warm = L (1) cold ⊂ L ( J ) cold for all J ≥ 2 and combining with Lemma D.1, we have

To show that strict inequality for J > γ δn , it suffices to show that L (1) cold ⊊ L ( J ) cold for J > γ δn since we already showed that G ⊊ S and |G ∩ S c | ≥ τ -1 for any class c ∈ [ C ] . In J -th experiment using cold-starting, by Lemma D.3, a model first learns features in G , say, in the first s steps. In the ( s +1) -th step, cold-starting model learns a new feature since

from the condition in the theorem statement. Hence, we have our conclusion for the test accuracy.

For the train time, since the following holds, we conclude T ( J ) warm < T ( J ) cold when J ≥ 2 :

Theorem 3.6. For any experiment J ≥ 2 , the following holds:

Proof of Theorem 3.6. Recall that the ideal algorithm works by forgetting memorized data points while retaining previously learned features. In other words, at the initial step of the ( j + 1) -th experiment, we have L ( j +1 , 0) ideal = L ( j ) ideal . Additionally, g ( v ; T 1: j +1 , N ( j +1 , 0) ideal ) = h ( v ; L ( j +1 , 0) ideal ) holds for all v ∈ S since M ( j +1 , 0) = ∅ .

We will show that L ( J ) cold = L ( J ) ideal holds for all J ≥ 1 by using induction.

When J = 1 , by applying Lemma D.2, it holds since L (1) cold = L (1) ideal . Suppose L ( J -1) cold = L ( J -1) ideal for some J ≥ 2 and we will prove that L ( J ) cold = L ( J ) ideal . We have the following at the first step of the J -th experiment for all v ∈ S :

For the cold-starting method in the J -th experiment, by Lemma D.2, let s be the step at which the model first finishes learning features in L ( J -1) cold . Then, at the ( s +1) -th step for all v ∈ S :

Since we assumed h ( v ; L ( J -1) cold ) = h ( v ; L ( J -1) ideal ) , the cold-starting method starts to behave identically to the ideal method from the ( s +1) -th time step onwards, by Lemma D.2, resulting in L ( J ) cold = L ( J ) ideal .

∣ ∣ ∣ N ( J, 0) ideal ∣ ∣ ∣ < |T 1: J | for J ≥ 1 since ∣ ∣ ∣ L ( J ) ideal ∩ S c ∣ ∣ ∣ ≥ τ for some class c ∈ [ C ] due to Assumption 3.3. Thus, the training time of the ideal method, T ( J ) ideal , is as follows:

E Omitted Algorithms

In this section, we provide detailed training algorithms for our proposed learning framework. Algorithm 2 outlines the standard training method within our learning framework. Subsequently, we compare the Cold-starting, Warm-starting, and Ideal methods using the given abstract algorithm in the following algorithms.

Algorithm 2 Training Process

Require:

L : Set of learned features

M : Set of memorized data points

T : Training dataset

γ : Threshold for learning features

τ : Threshold for the number of learned features a data point needs to be considered well-classified

1: function TRAININGPROCESS( L , M , T , γ , τ )

Algorithm 3 Cold-Starting until J -th Experiment

Require:

T 1: J : Training dataset

γ : Threshold for learning features

τ : Threshold for the number of learned features a data point needs to be considered well-classified

1: Initialize:

7: return L

Algorithm 4 Warm-Starting until J -th Experiment

Require:

T 1: J : Training dataset

γ : Threshold for learning features

τ : Threshold for the number of learned features a data point needs to be considered well-classified

1: Initialize:

(0)

M ←∅ 2: for j in 1 : J do

3: L ( j ) , M ( j ) ← TrainingProcess( L ( j -1) , M ( j -1) , T 1: j , γ , τ )

4: end for

5: return L ( j )

Algorithm 5 Ideal-Starting until J -th Experiment

Require:

T 1: J : Training dataset

γ : Threshold for learning features

τ : Threshold for the number of learned features a data point needs to be considered well-classified

1: Initialize:

M (0) ←∅ 2: for j in 1 : J do

4: M ( j ) ←∅

5: end for

6: return L ( j )

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the main claims and contributions of the paper, including the development of a learning framework to study loss of plasticity in stationary data distributions, identifying noise memorization as the primary cause, and proposing the DASH method to mitigate the issue. The claims match the theoretical and empirical results presented.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: This paper includes a "Discussion and Conclusion" section that reflects on the scope and limitations of the work. For example, it notes that the theoretical analysis treats the learning process as discrete and does not assume any specific hypothesis class in order to generalize the findings.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The main theoretical results (Theorems3.4 and 3.6) clearly state our assumptions and provide proof sketches in the main text with references to complete proofs in the Appendix D. The proofs rely on reasonable assumptions and appear to be correct stated in Section 3.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental setup and training details are described in sufficient detail in Section 5.1 to reproduce the main results, including dataset splits, model architectures, optimizers, and hyperparameters. Appendix C.3 provides additional specifics on the hyperparameter settings used.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provided open access to the code. Please refer to footnotes 1 and 2.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hypersparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 5.1 describes the key experimental settings including the dataset setup, model architectures, optimizers, and hyperparameters. Appendix C.3 provides additional details on the specific hyperparameter values used for each experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The experimental results include standard deviations over multiple random seeds to quantify the statistical significance and stability of the findings. The number of random seeds used is reported throughout the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The experimental compute resources are reported in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper does not violate the NeurIPS Code of Ethics in terms of data privacy, potential for misuse, or other ethical issues. The work is largely theoretical and the experiments use common open datasets.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Section 6 discusses the positive impact of the work in enabling more efficient and sustainable AI that can adapt to increasing real-world data. It notes this can help prevent wasted time, energy and environmental impact. Potential negative impacts are not explicitly discussed as the work is largely theoretical.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper focuses on theoretical analysis and experiments on common open datasets. It does not introduce any models or datasets with high risk of misuse that would require safeguards.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not rely on or adapt any existing code or model assets. The datasets used are common open datasets and cited appropriately.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not introduce any new datasets, code or models. It focuses on theoretical analysis and experiments using existing datasets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This research does not involve any human subjects or crowdsourcing. It is based on theoretical analysis and experiments on open datasets.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This research does not involve any human subjects, so IRB approval is not applicable. It focuses on theoretical work and experiments on open datasets.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines a setting in which a model is learned with an expanding dataset, such that new data is introduced after the model hits a training accuracy threshold. In particular, the new data comes from the same distribution as the existing data (or sometimes has the exact same statistics), making this a stationary learning problem (unlike the non-stationary problems commonly explored in plasticity loss settings). After demonstrating that plasticity loss occurs in this setting, the paper proposes a new method, DASH, to tackle the problem. DASH shrinks weights based on the cosine similarity between the weight vector and the negative gradient of the training loss. There are a series of experiments which generally show DASH outperforming other plasticity loss prevention approaches, with more in the appendix.\n\nSTRENGTHS:\n- Based on my knowledge and research, I think this is a novel approach to preventing plasticity loss. It is quite original in its idea, which the authors have tried to ground in theory.\n- This is a relatively underexplored area (plasticity loss under stationary settings), and thus is useful for understanding plasticity loss further.\n- DASH seems to offer genuine performance improvements over most methods, though it is worth noting that this is not always the case (i.e., in the appendix more results are included which are either significantly more marginal, or sometimes favouring other methods). In particular, the focus that test accuracy is not only improved, but we carry over some of the benefits of warm starting (i.e., quicker convergence) is notable.\n- The figures/graphs in the paper were generally good and useful for understanding what was being discussed.\n\nWEAKNESSES:\nBefore noting weaknesses of the paper, which I generally feel come down to clarity issues. I am open to increasing my score in light of discussion based on the author's response to my highlighted weaknesses and questions.\n\n- I found a lot of the theory quite confusing, and couldn't follow section 3 ('Comparison Between Warm-Start and Cold-Start'). Similarly, I found some explanations about the method, and the training process quite confusing. Perhaps demonstrating with examples could be useful, but it is currently quite hard to parse exactly what the problem setting is.\n- This work has not been studied in a particularly standard learning process. Instead, the focus is on a setting where features are learned based on their frequency of appearance in data, rather than their signal strength. This is quite an artificial setting, and it is not exactly clear to me that DASH scales to more conventional learning frameworks.\n- Appendix A.3 feels slightly burrowed away, but considers the SOTA setting for CIFAR-10 and CIFAR-100, in which DASH underperforms random initialisation. I think this is quite a crucial result, and arguably outweights the results from Table 1 in which a more naive approach is taken. While it is good that the authors include these results in the appendix, it would be good to try to include this in a more upfront way (perhaps including discussion about how DASH is helpful in a naive setting and helps reduce the barrier to entry with less studied datasets?)\n\nQUESTIONS:\n- Considering a different learning setting, where data is continuously added rather than in predictable batches, would DASH still be able to perform (given at the moment, it is applied whenever new data is provided)? Similarly, how is its performance when we are not adding data from a stationary distribution, given this is possibly more realistic?\n- How do the authors think DASH could work in the more generic learning problem (i.e., where we are not learning features based on their frequency)?\n- What is actually meant by the frequency of features - can you offer an example please? I find it hard to determine exactly what is meant by the frequency of these discrete features?\n\n- In table 8, the authors write CIF1R rather than CIFAR.\n- What is meant in line 325: 'This allows weights to easily change its direction.'?\n\nLIMITATIONS:\n- In the appendix, there are examples where DASH does not outperform other methods, or is within the error margin. However, there is clearly a strong signal that DASH on average provides benefit to the training in all but the SOTA setting. The authors are upfront about this.\n- I believe the authors are generally upfront about the limitations of their work, though I feel perhaps highlighting that they are working in a slightly artificial setting, and demonstrating how DASH could be implemented to a typical machine learning pipeline, would be useful.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examines warm-start and the loss of plasticity, identifying that noise memorization from warm-start impairs generalization in stationary data distributions. It proposes a method called DASH, which selectively forgets previously memorized noise while preserving learned features.\n\nSTRENGTHS:\n- **Feature Learning Framework**: The paper introduces a feature learning framework to investigate the warm-start issue. It thoroughly explores the properties of warm-start and cold-start training, the role of noise memorization in the loss of plasticity, and the selective retention (forgetting) of features (noise). These core findings are well-supported through theoretical grounding and experimentation.\n- **Performance:** In vanilla settings, DASH significantly outperforms other methods on several small datasets in terms of accuracy and training steps. And, in the SoTA setting, DASH rivals other methods.\n\nWEAKNESSES:\n- **Compute Overhead**: Gradient-based approaches typically incur significant compute overhead. While DASH is claimed to achieve faster convergence, it remains unclear if its shrinking process slows down the overall training. Quantitative comparisons in terms of FLOPs and training time would be insightful. Furthermore, DASH has not been compared with baselines regarding compute and memory overhead. The training convergence based on the gradient norm does not consider the compute and memory overhead associated with DASH’s gradient calculations.\n- **Assumption: Gradient norm as a proxy for the number of training steps**: The assumption that the number of training steps can be reliably approximated by the initial gradient norm is insufficiently supported by the data presented in Figure 10 of the paper. The correlation between training steps and the initial gradient norm does not appear to be linear, particularly for the Warm start. \n- **In-depth Analysis**: The experiments do not demonstrate a direct link between learned/memorized data and the cosine similarity of gradients and weights. Conducting ablation studies might provide further insights into this relationship. Additionally, the proposed method appears to be a variant of the shrink and perturb approach.\n- **Evaluation and Scalability**: Evaluation is performed exclusively on toy datasets, leaving DASH’s scalability to larger and more challenging datasets, such as ImageNet-1K, unknown.\n- **Generality**: In SOTA settings, as shown in Table 7, DASH does not consistently outperform the compared methods, particularly warm initialization. This raises questions about the wider applicability and robustness of DASH. Its efficacy seems to diminish when using augmentations and weight decay.\n\nQUESTIONS:\n- Does the apparent non-linearity in the correlation between initial gradient norms and training steps in Figure 10 change how Figure 1(b) should be interpreted?\n- In Figure 2, the test accuracy drops with longer pre-training when the model is pre-trained on 50% of the training data and then fine-tuned on the full dataset. Is this phenomenon universal or does it only occur with the CIFAR dataset? Could the authors provide similar results for high-dimensional datasets, such as ImageNet-1K (224x224)? If the network learns the same set of semantic categories during the pre-training and fine-tuning stages, the fine-tuning stage only differs in terms of the number of samples. Is this phenomenon related to the size of the training data? How much do pre-training and fine-tuning datasets differ in terms of distribution shifts? Is the pre-training data class balanced? How does DASH perform in a similar setting?\n- Does DASH need access to the entire dataset (all chunks), including current and previous datasets/chunks, for computing gradients and shrinking weights during every training step?\n- In *Line 116*, the set `S_y` is undefined, and may be corrected to `S_c`.\n- In *Line 134*, `Tau < K` may be corrected to `Tau <= K`, where the number of learned features can also equal the set of learnable class-dependent features, K.\n\nLIMITATIONS:\nThe paper provides little to no detail on the limitations of this work. While it mentions the assumptions of a discrete learning process and the lack of consideration for a hypothesis class, it does not address whether DASH can be extended to larger problem settings. If such limitations exist, they should be explicitly stated by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWarm-starting neural networks may lead to poor generalization performance or loss of plasticity, likely due to overfitting. This paper presents a framework that hypothesizes that noise memorization is the primary cause of loss of generalization in warm-starting settings. The authors then present an algorithm motivated by their framework that improves the generalization and plasticity of the neural networks. The algorithm works on a per-neuron level and measures the cosine similarity between the negative gradient of the input weight to each neuron and its weights. This cosine similarity is used as a utility measure to allow for more shrinking for the input weights with low cosine similarity. The authors showed the effectiveness of their method on a wide range of problems and compared it against other baselines demonstrating superior performance.\n\nSTRENGTHS:\nThe paper addresses an important problem: maintaining plasticity and generalizing neural networks. The proposed algorithm is novel and effective, and the experimental results seem comprehensive.\n\nWEAKNESSES:\n- The main problem of this work is that the link between the framework and the algorithm is not clear. For example, it’s unclear why different parts of the algorithm are designed this way, especially the cosine similarity part. Why is the momentum direction assumed to be the direction of features? \n- The evaluation may be unfair and statistically insignificant. In multiple experiments (e.g., figure 1),  the results are generated based on three seeds. This is a very low number to have statistically significant results. Additionally, the reported methods use the same hyperparameter across multiple methods, which may be unfair to the baselines since they are not well-tuned compared to DASH. The results should be based on the best hyperparameter of each method. This might explain why the authors found approaches such as L2 Init, resetting, layer normalization, SAM, or reviving dead neurons to be less effective. In contrast, the only effective one was S&P, which uses a hyperparameter (shrinking) similar to the DASH method.\n\nQUESTIONS:\n- In Figure 4, the authors presented an arrow with a feature direction. What does that mean exactly in a rigorous way?\n- What is the definition of a feature? The word is used in the paper many times without a proper definition, leading to further confusion. In neural networks, the learner always improves its set of features, so they are always changing.\n\nLIMITATIONS:\nNo. I think the main limitation of the method is that the cosine-similarity metric seems to be heuristic-based and not motivated by the introduced framework. It is unclear if such a metric is ideal or if it reflects what is suggested by the theoretical framework.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the reasons why warm-starting a neural network by pre-training it on a subset of the full training set leads to suboptimal performance when compared to training it on the full dataset from scratch. In particular, it proposes an abstract combinatorial model of feature learning that according to the authors captures the essence of this phenomenon. In this model, warm-starting leads to overfitting on feature noise that hinders learning new features when fine-tuning on the full distribution when compared to cold-starting. The authors then show that there exist an idealized learning algorithm in this abstract setting that can perform as well as cold-starting by forgeting the memorized noise prior to fine-tuning. Based on this theoretical intuition, they then propose Direction-Aware SHrinking (DASH) a technique that shrinks the learning step per-neuron based on its alignment with previous warm-starting epochs. DASH seems to outperform other techniques in a synthetic warm-starting benchmark.\n\nSTRENGTHS:\n1. **Intuitive and insightful theoretical model**: Personally, I find the discrete abstract model of learning of this paper to be very insightful. Theorems 3.4 and Theorems 4.1 are good examples of theoretical propositions that can help cement our empirical observations in deep learning. This model may not provide a rigorous path towards mathematically \"proving\" the observed warm-starting phenomenon, but it is useful to reason about it.\n2. **Convincing explanations of warm-starting**: The provided explanations for the observed behavior of deep models under warm starting and the evidence to support them are solid and convincing.\n3. **Good performance of DASH**: DASH seems to perform strongly against prior work on the studied warm-starting benchmarks.\n\nWEAKNESSES:\n1. **Unclear details of DASH**: The description of DASH in section 4.5 is not clear and the algorithm listing does not fully explain the algorithm (see questions). This makes it hard to assess the complexity of implementing DASH and its practical relevance.\n2. **Limited evaluation of DASH in online learning**: The authors argue that one of the main motivations of studying the decrease of performance under warm-starting is the fact that many networks are trained in an online setting with new data being collected sequentially. However, the only evaluation of DASH is a synthetic warm-starting benchmark that only tests a narrow and unrealistic setting. I am not against this particular benchmark, but I believe the paper would have benefited from further investigations of DASH in a more realistic online setting (including comparing it against strong baselines in that space).\n3. **Reasons for improved performance of DASH over cold-starting are not explained**: As the authors highlight, DASH seems to outperform cold-starting in their benchmarks. This phenomenon, however, cannot be explained by their theoretical model and thus it is a weakness of the model itself.\n\nQUESTIONS:\nI would appreciate a deeper description of how DASH works:\n- Is the running average of gradients computed at every step or every epoch? \n- Is the shrinking performed after every step, every epoch, every experiment?\n- Can DASH be combined with any off-the-shelf deep learning optimizer? If so, how?\n- What is the memory complexity of DASH?\n\nIf these questions are addressed and the explanations are clear and made part of the final manuscript I would be open to increase my score.\n\nLIMITATIONS:\nThe authors briefly mention mention some of the limitations of their work (such as the unexplained outperformance of DASH over cold-starting) or specificity of their results to the stationary setting.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}