DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity
Authors: Baekrok Shin
Junsoo Oh
Hanseul Cho
Chulhee Yun
Venue: NeurIPS 2024 poster (2024)
Keywords: loss of plasticity
warm-starting
incremental learning
generalization
Direction-Aware SHrinking
Primary area: optimization_for_deep_networks

Abstract: Warm-starting neural network training by initializing networks with previously learned weights is appealing, as practical neural networks are often deployed under a continuous influx of new data. However, it often leads to *loss of plasticity*, where the network loses its ability to learn new information, resulting in worse generalization than training from scratch. This occurs even under stationary data distributions, and its underlying mechanism is poorly understood. We develop a framework emulating real-world neural network training and identify noise memorization as the primary cause of plasticity loss when warm-starting on stationary data. Motivated by this, we propose **Direction-Aware SHrinking (DASH)**, a method aiming to mitigate plasticity loss by selectively forgetting memorized noise while preserving learned features. We validate our approach on vision tasks, demonstrating improvements in test accuracy and training efficiency.

DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity
Baekrok Shin
Junsoo Oh Hanseul Cho Chulhee Yun
Kim Jaechul Graduate School of AI, KAIST (bi.shin, junsoo. oh, jhs4015, chulhee, . yun) ackaist.ac.k.
Abstract
Warm-starting neural network training by initializing networks with previously Iearned weights is appealing, as practical neural networks are often deployed under continuous influx of new data. However, it often leads to loss of plasticity, where the network loses its ability to learn new information, resulting in worse generaliza- tion than training from scratch. This occurs under stationary data distributions, and its underlying mechanism is poorly understood. We develop a framework emu- lating real-world neural network training and identify noise memorization the primary cause of plasticity loss when warm-starting on stationary data. Motivated by this, we propose Direction-Aware SHrinking (DASH), a method aiming mitigate plasticity loss by selectively forgetting memorized noise while preserv- ing learned features. We validate our approach on vision tasks, demonstrating improvements in test accuracy and training efficiency." !
1 Introduction
When training a neural network on a gradually changing dataset, the model tends to lose its plasticity, which refers to the model's 's ability adapt to information (Dohare al., 2021; Lyle al., 2023b; Nikishin et al., 2022). This phenomenon is particularly relevant in scenarios with non-stationary data distributions, such as reinforcement learning (Igl al., 2020; Nikishin et al., 2022) and continual learning (Chen et al., 2023; Kumar et al., 2023; Wu et al., 2021). While requiring to overwrite outdated knowledge as the environment changes, models overfitted previously encountered environments often struggle to cumulate new information, which in turn leads to reduced generalization performance (Lyle al., 2023b). Under this viewpoint, various efforts have been made to mitigate the loss of plasticity, such as resetting layers (Nikishin et al.. 2022), regularizing weights (Kumar et al., 2023), and modifying architectures (Lee al., 2023: Lyle al., 2023a; Nikishin et al., 2023).
Perhaps surprisingly, a similar phenomenon occurs in supervised learning settings, even where new data points sampled from stationary data distribution are added the dataset during training. is counterintuitive, as one would expect advantages in both generalization performance and computational efficiency when we warm-start from a model pre-trained on data points of the same distribution. For particular example, when a model is pre-trained using a portion of dataset and then we resume the training with the whole dataset, the generalization performance often worse than model trained from scratch (i.e., cold-start), despite achieving similar training accuracy (Ash and Adams, 2020; Berariu al., 2021; Igl et al., 2020). Liu al. (2020) report similar observation: training neural networks with random labels leads to spurious local minimum which challenging to escape from, even when retraining with a correctly labeled dataset. Interestingly, Igl et al. (2020) found that pre-training with random labels followed by the corrected dataset yields
Authors contributed equally to this paper.
'The NVIDIA GPU implementation can be found github. com/baakrok/DASHH 2 For the Intel Gaudi implementation, visit: wwthub.cco//aaeeee Ctthab.-oo//AAA---TTELLCCCCLLLL Labbeal----LLLaauadaaaaa haabb//audi-eee dash.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Test Accuracy (%6
Cold
20000
DASH
20-
Number
10000
10-
Number of Experiments
Number of Experiments
Figure 1: Performance comparison of various methods Tiny-ImageNe using ResNet-18. The same hyperparameters are used across all methods. The dataset is divided into 50 chunks, with a constant number of data points added to the training dataset in each experiment (x-axis), reaching the full dataset at the 50th experiment. Models are trained until achieving 99.9% train accuracy before proceeding to the next experiment; the plot on the right reports the number of update steps executed in each experiment. Results are averaged over three random seeds. "Cold" refers to cold-starting and "Warm" refers warm-starting. The Shrink & Perturb (S&P) method involves shrinking the model weights by a constant factor and adding noise (Ash and Adams, 2020). Notably, DASH, our proposed method, achieves better generalization performance compared both training from scratch and S&P, while requiring fewer steps to converge.
better generalization performance than pre-training with a small portion of the (correctly labeled) dataset and then training with the full, unaltered dataset. It is striking that warm-starting leads to such severe loss of performance, even worse than that of a cold-started model a model re-trained from parameters pre-traee pre-trained with random labels, despite the stationarity of the data distribution.
These counterintuitive results prompt us to investigate the underlying reasons for them. While some studies have attempted to explain the loss of plasticity deep neural networks (DNNs) under non-stationarity (Lewandowski et al., 2023; Lyle et al., 2023b; Sokar et al., 2023), their empirical explanations rely on various factors, such as model architecture, datasets, and other variables, making it difficult to generalize the findings (Lewandowski et al., 2023; Lyle al., 2023a). Moreover, there limited research that explores why warm-starting is problematic in stationary settings, highlighting the lack of fundamental understanding of the loss of plasticity phenomenon in both stationary and non non-stationary data distributions.
1.1 Our Contributions
In this work, we aim to explain why warm-starting leads to worse generalization compared to cold- starting, focusing on the stationary case. We propose an abstract framework that combines the popular feature learning framework initiated by Allen-Zhu and Li (2020) with recent approach by Jiang et al. (2024) that studies feature learning in a combinatorial and abstract manner. Our analysis suggests that warm-starting leads to overfitting by memorizing noise present in the newly introduced data rather than learning new features.
Inspired by this finding, we propose Direction-Awand SHrinking (DASH), which aims to encourage the model to forget memorized noise without affecting previously learned features. This enables the model to learn features that cannot be acquired through warm-starting alone, enhancing the model's generalization ability. We validate DASH using an expanding dataset setting, similar to the approach in Ash and Adams (2020), employing various models, datasets, and optimizers, As example, Figure/ shows promising results in terms both test accuracy and training time.
1.2 Related Works
Loss of Plasticity. Research has aimed to understand and mitigate loss of plasticity in non-stationary data distributions. Lewandowski al. (2023) explain that loss of plasticity co-occurs with a reduction in the Hessian rank of the training objective, while Sokar et al. (2023) attribute it to an increasing number of inactive neurons during training. Lyle et al. (2023b) find that changes the loss landscape curvature caused by non-stationarity lead to loss of plasticity. Methods addressing this issue in non-stationary settings include recycling dormant neurons (Sokar et al., 2023), regularizing weights towards initial values (Kumar et al., 2023), and combining techniques (Lee et al., 2023) like layer normalization (Ba al., 2016), Sharpness-Awann Minimization (SAM) (Foret et al., 2020), resetting layers (Nikishin et al., 2022), and Concatenated ReLU activation (Shang et al., 2016).

---

However, these explanations and methods diverge from the behavior observed in stationary data distributions. Techniques aimed at mitigating loss plasticity under non-stationarity are ineffective under stationary distributions, shown in Appendix C.I. in line with the observations in Lee et al. (2023). While some works study the warm-starting problem in stationary settings, they rely on empirical observations without theoretical analysis (Achille et al., 2018; Ash and Adams, 2020; Berariu et al., 2021). The most relevant work by Ash and Adams (2020) introduces the Shrink & Perturb (S&P) method, which mitigates the loss of plasticity in stationary settings to some extent by shrinking all weight vectors by a constant factor and adding noise. However, they do not explain why this phenomenon occurs or why S&P is effective. We develop theoretical framework explaining why warm- sarm--sarting suffers even under stationary distribution. Based on findings, we propose a method that shrinks the weight vector in direction-awee aware manner maintain properly learned features.
Feature Learning in Neural Networks. Recent studies have investigated how training methods and network architectures influence generalization performance, focusing on data distributions with label-dependent features and label- -ibeeinndeenndett noise (Allen-Zhu and Li, 2020; Cao et al., 2022; Deng al., 2023; Jelassi and Li, 2022: Oh and Yun. 2024; Zou et al., 2023). In particular, Shen al. (2022) examine a data distribution consisting of varying frequencies of features and large strengths of noise, emphasizing the significance of feature frequencies in learning dynamics. Jiang et al. (2024) propose a novel feature learning framework based on their observations in real-world scenarios, which also involves features with different frequencies but considers the learning process discrete sampling process. Our framework extends these ideas by incorporating features with varying frequencies, noise components, and the discrete learning process while introducing more intricate learning process capturing the key aspects of feature learning dynamics in expanding datasets.
2
A Framework of Feature Learning
2.1 Motivation and Intuition
We present the motivation and intuition behind our framework before delving into the formal description. Our framework captures key characteristics of image data, where the input includes both label-relevant information (referred to as features, e.g., cat faces in cat images) and label-irrelevant information (referred to as noise, e.g., grass in cat images). key intuition is that minimizing training loss involves two strategies: learning features and memorizing noise. This framework builds on insights from Shen et al. (2022) and integrates them into a discrete learning framework. We provide more detailed intuition on our framework, including the training process, in Appendix A.
Shen et al. (2022) consider a neural network trained on data with features of different frequencies and noise components stronger than the features. The negative gradient of the loss for each single data point aligns more with the noise than the features due to the larger scale of noise, making the model more likely to memorize noise rather than learn features. However, an identical feature appears in many data points, while noise appears only once and does not overlap across data points. Thus, if a feature appears at sufficiently high frequency in the dataset, the model can learn the feature. Thus, the learning of features or noise depends on the frequency of features and the strength of the noise.
Inspired by Shen et al. (2022), we propose a novel discrete feature learning framework. This section introduces framework describing single experiment, while Section 3 analyzes expanding dataset scenarios. As our focus is on gradually expanding datasets, carrying out the (S)GD analysis over many experiments as in Shen et al. (2022) is highly challenging. Instead, we adopt a discrete learning process similar to Jiang et al. (2024) but propose a more intricate process reflecting key ideas from Shen et al. (2022). In doing so, we generalize the concept of plasticity loss and analyze it without assuming any particular hypothesis class for more comprehensive understanding. whereas existing works are limited to specific architectures.
2.2 Training Process
We consider a classification problem with C' classes, and data are represented as (x, v) E X x [C], where X denotes the input space. A data point is associated with a combination of class-dependent features V(x) C Sy where S. {ve,1 Ve.2,... , Fe.K) is the set of all features for each class E [C]. Also, every data point contains point-specific noise which is inaeeedndeee.dett.
The model f X + [C] sequentially learns features based on their frequency. The training process is described by the set of learned features L C S & Upelo S. and the set of data points with non-zero
3

---

gradients N C T, where T = {( Mi y:) [ie(m] denotes training set. The set N, representing the data points with non-zero gradients, will be defined below. The frequency of a feature v in data points belonging to A is denoted by
ge: T.N)
Σ 1(c V(x)), (m.j)EN
where 1(-) is the indicator function, which equals I if the condition inside the parentheses is true and O otherwise. At each step of training, if C and A are given, the model chooses the most frequent feature among the features not yet learned, i.e., arbitrarily choose v E arg e (u; T.N).
The model decides whether to learn a selected feature e by comparing its signal strength, represented by [T] - g(u; TN,, with the signal strength of noise, given by 7, which reflects the key ideas of Shen et al. (2022). If the frequency of the selected feature v is no less than the threshold 7/ T[ ie.. g(e; N.) 2 7/[7]. the model learns v and adds it to its set of learned features L. The feature learning process continues until the model reaches a point where the selected feature v has < (u))
7/[T], indicating that the signal strength of every remaining feature is weaker than that of noise. At this point, the feature learning process ends.
We consider a data point to be well-classified if the model f has learned at least 7 features from V(x)), i.e., (()) 2 т, where т < K. In this case, we consider æ to have a zero gradient, meaning it cannot further contribute to the learning process. Throughout the feature learning process, the set N of data points with non-zero gradients is dynamically updated as new features are learned. At each step, when the model successfully learns a new feature, we update A by removing the data points that satisfy |C n (x)) 2 т, as they become well-classifedd due to the newly learned feature.
If the feature learning process ends and the model has learned as many features as it can, the remaining data points that have non-zero gradients will be memorized by fitting the random noise present in them and will be considered to have zero gradients. This step concludes the training process. Pictorial illustration can be found in Appendix and detailed algorithm of the learning process can be found in Algorithm 2 in Appendix E.
2.3 Discussion on Training Process
In our framework, the model selects features based on their frequency in the set of unclassified data points N.. The intuition behind this approach is that features appearing more frequently the set of data points will have larger gradients, leading to larger updates, and we treat g(e; T.N) as proxy of the gradient for particular feature v. As result, the model prioritizes sequentially learning these high-frrequnne features. However, if the frequency g T W.N of a particular feature is not sufficiently large, such that the total occurrence of is less than the strength of the noise, i.e., [T] 9(( (N) < 7 the model will struggle to learn that feature. Consequently, the model will prioritize learning the noise over the informative features. When this situation arises, the learning procedure becomes sub-optimal because the model fails to capture the true underlying features of the data and instead memorizes the noise to achieve high training accuracy.
The threshold determines when a data point is considered well-classified and acts as proxy for the dataset's 's complexity. A higher т requires the model learn more features for correct predictions, while a lower т allows accurate predictions with fewer learned features. Experiments in Appendix B Figure 11 and 12 support this interpretation.
Remark 2.1. We believe our analysis can be extended to scenarios where feature strength varies across data by treating the set of features multiset, where multiple instances of the same element are allowed. The analyses in these cases are nearly identical to ours; therefore, we assume all features have identical strengths for notational simplicity.
3 Warm-Starting versus Cold-Starting, and New Ideal Method
3.1 Experiments with Expanding Dataset
In this section, we up the scenario where the dataset grows after each experiment in our learning framework, allowing us to compare warm-start, cold-start, and a new ideal method, which will be defined later in Sections 3.3 and 3.4.
To better understand the loss of plasticity under stationary data distribution, we consider an extreme form of stationarity where the frequency of each feature combination remains constant in each

---

chunk of additional data. We investigate if the loss of plasticity can manifest even under this strong stationarity. The detailed description of the dataset across the entire experiment as follows:
Assumption 3.1. In each j-th experiment, we provided with training dataset Tj ((o15: ses)) with n samples. For each class E [C] and each possible feature combina- tion A C Sc. we assume that Tj contains exactly n.A 2 1 data points with associated feature set A, where the values A are independent j. Note that _=ElC) TIAA = n. the j-th experiment, we use the cumulative dataset Tig == Uieta Tr, the set of all training data up to the j-th experiment.
Remark 3.2. In each experiment, the feature combinations remain the same across the dataset, but the individual data points differ. This is because each data point is associated with its specific noise, which varies across samples. Although the underlying features are the same, the noise component of each data point is unique. This approach ensures that the model is exposed to a diverse set of samples.
We define a technical term h(v; L) A a. Eeeic| TAA (o E A LAn <>) to denote the portion of data points containing v not- not-well-classiiee by feature set L. This leads to assumption:
v1 Assumption 3.3. For any learned feature set L C S, if v1, U/2 E S. for some class e E [C] and
h(11; L) = (12; L), then 11 U2. Also, for any class E [C], there exists some distinct features , 0y E S. such that ((1;; T, Ty)...., 1( Ty, Tj) 2 7/n and g(v+; Ty Tj) < </nn.
This assumption leads to Lemma D.2, stating that the order in which features are learned within a class is deterministic. This is just for simplicity of presentation and can be relaxed. The last assumption is justified by the moderate number of data points in each chunk 7j. ensuring the existence of both T 1 learnable features and non-learnable feature within class. Throughout the following discussion, we will proceed under the above assumptions unless otherwise specified.
Notation. We denote a model at step s of the j-th experiment as f(iss) We denote the set of learned features and the set of memorized data for the model f(i.e) as L(.e) and M(U+s), respectively, We also define the set of data points with non-zero gradients at step s of the j-th experiment as A/U..,) We define respective versions of these sets and the model, with different initialization methods,
denoted by the subscripts (e.g., fwarm, f(ā.a) and f((ee)) We emphasize that each method initializes f(a.0) cli0), MM5.0), and (G.00) differently at the start of the j-th experiment.
3.2 Prediction Process and Training Time
We provide comparison of three initialization methods based on test accuracy and training time. To evaluate these metrics within our framework, we define the prediction process and training time.
Prediction Process. The model predicts unseen data points by comparing the learned features with features present in given data point æ. If the overlap between the learned feature set L and the features in æ. denoted as (xx), is at least т, i... (V(x) n C 2 т, the model correctly classifies the data point. Otherwise, the model resorts to random guessing.
Training Time. Accurately measuring training time within our discrete learning framework is challenging. To address this, we introduce an alternative for training time of j-th experiment: the number of training data points with non-zero gradients the start of j-th experiment, [ This represents the amount of "learning" required for the model to classify all data points correctly. We empirically validated this proxy in practical scenarios, as shown in Figures and in Appendix B. Additionally, Nakkiran et al. (2021) observe that in real-worle neural network training, when other components are fixed, the training time increases with the number of data points to learn.
3.3 Comparison Between Warm-Starting and Cold-Starting in Our Framework
Now we analyze the warm-start and cold-start initialization methods within our framework, focusing on test accuracy and training time. We note that, by definition, c(5, and M are both empty sets, while C - c and (i-))) = M where 8j denotes the last step of j-th experiment. Besides, we use shorthand notation for step s; of the experiment j that we drop s s 8, (e.g., L() = g(.s))). For the detailed algorithms based on our learning framework, see Algorithms 3 and in Appendix E.
5

---

In the test data, feature combination A C Sc of data point with class e E [C] appears with probability n.a/n /n along with data data-specific noise. By Section 3.2, test accuracy for learned set L and training time are defined as:
ACC(C) a 11-
Σ nA (1AN((AAnNAA) <T)
cE(C),ACS.
T(J) warm
Σ JENU
( warm
Tul
Σ
( cold
Based on these definitions, the following theorem holds:
Theorem 3.4. There exists nonempty G S ES such that we always obtain Luamm all J 2 2, the following inequalities hold:
- G. For
ACC
(Charr)
S ACC (Coola).
TDJ) warm
< (
Furthermore, ACC(C < ACC(
holds when J > In where - max h(e; 9) > 0. vES
Proof Idea. After the first experiment, the data points in Ti cannot further contribute to the learning process of the warm-started model. Consequently, even when a new data chunk is provided in subsequent experiments, the feature frequencies are too small, resulting in a weak signal strength of features that cannot overcome the noise signal strength. As a result, the model memorizes individual noise components of the new data points. This procedure is repeated with every experiment, causing the learned feature set to remain the same as at the end of the first experiment. In contrast, when receiving Ti at once startisg the signal strength of features large enough to overcome the noise signal strength, allowing the model to learn many more features.
L
Theorem 3.4 highlights trade-off between cold-starting and warm-starting. Regarding test accuracy, the theorem concludes that cold-starting can achieve strictly higher accuracy than warm-starting. However, warm-starting requires strictly shorter training time compared to cold-starting.
Detailed proof is provided in Appendix D. Theorem 3.4 suggests that the loss of plasticity in the incremental setting under the stationary assumption can be attributed to the noise memorization process. A similar observation is made in real-world neural network training. It widely believed that during the early stages of training, neural networks primarily focus on learning features from the dataset, and after learning these features, the model starts to memorize data points that it fails to classify correctly using the learned features. To investigate this phenomenon, we conducted an experiment where CIFAR-10 10 was divided into two chunks, each containing 50% of the training dataset. The model was pre-trained on one chunk and then further trained on the full dataset for 300 epochs. We used three-layer MI.P and ResNet-18 with SGD optimizer across 10 random seeds.
Figure shows the change in the model's performance based on the duration of pre-training. When pre-training is stopped a certain epoch and the model is then trained on the full dataset, test accuracy is maintained. However, if pre-training continues beyond a specific threshold (approximately 50% pre-training accuracy this case), warm warm-stariin starting significantly impairs the model's performance increasingly memorizes training data points. We attribute this phenomenon to the neural network's s memorization process after learning features. This is consistent with reports of a critical learning period where neural networks learn useful features in the early phase of learning (Achille et al., 2018; Frankle et al., 2020; Kleinman et al., 2024), and with findings that neural networks tend to learn features followed by memorizing noises (Arpit al., 2017; Jiang et al., 2020). Using the same experimental settings as in Figure 2, we tested with a large-scale dataset, ImageNet- 1k, and observed similar trends (see Figure 8 in Appendix B).
Remark 3.5. Igl et al. (2020) find that training a model on random labels followed by corrected labels results in better generalization compared pre-training on subset of correctly labeled data and then further training on the full dataset with the same distribution. Achille et al. (2018 8) also observe that pre-training with slightly blurred images followed by original images yields worse test accuracy than gre-training with random label or random noise images. These findings align with our observations: re-training with corrected labels after random label learning "revives" gradients for most memorized data points, enabling new feature learning. Conversely, with static distributions, gradients for memorized data points remain suppressed, leading to learning from only few data points with active gradients, causing memorization.

---

ResNet-18
70
-10
5835
80
Pre-Train Accuracy (96
-00
Accuracy
$8.0
67
66
67.5
60
$7.0
30
63
50
Pra-Train Epoch
Pre: Train Epoch
Random (Test)
Warm (Test)
Wirm (Pre-Trair)
The plot shows the test accuracy (left y-axis) when the model pretrained for varying epochs (X-axis) and then fine-tuned on the full data, along with the pretrain accuracy (right y-axis) plotted in brown. We trained three-layer MLP (left) and ResNet-18 18 (right). Each transparent line and point corresponds to specific random seed, and the median values are highlighted with opaque markers and solid lines. The 'Random' corresponds to training from random initialization (cold-start).
Figure 2:
3.4 An Ideal Method: Retaining Features and Forgetting Noise
In Section 3.3, we observed a trade-off between warm-starting and cold-starting. Cold-starting often achieves better test accuracy compared to stam--tarting, while warm-starting requires less time to converge. The results suggest that neither retaining all learned information nor discarding all Iearned information is ideal. To address this trade-off and get the best of both worlds, consider ideal algorithm where we retain all learned features while forgetting all memorized data points. For any experiment J Z 2, if we consider the ideal initialization, learned features c(d=)) are retained, and memorized data points MM(J1) are reset to an empty set. Pseudo-code for this method is given in Algorithm 5, which can be found in Appendix E. We define TT(J) - Σ}E[J ideal
| ideal as the training
((.0) time with the ideal method, where ideal represents the set of data points having non-zero gradient at the initial step of the j-th experiment. Then, we have the following theorem:
Theorem 3.6. For any experiment J 2 2, the following holds:
ACC
(Codld ACC (Cidea))
T(J) warm
T
ideal
The detailed proof is provided Appendix D. The ideal algorithm addresses the trade-off between cold-starting and warm-starting. We conducted an experiment to investigate the performance gap between these initialization methods.
Synthetic Experiment. To verify our theoretical findings in more realistic scenarios, we conducted an experiment that more closely resembles real-world settings. Instead of fixing the frequency of each feature set, we sampled each feature's 's existence from a Bernoulli distribution to construct V(x). This ensures that the experiment more representative of real-world scenarios. Specifically, for each data point (x, y), we uniformly sampled y E 1). From the feature set S, corresponding to the sampled class y. we sampled features where each feature's 's existence follows Bernoulli distribution, Ber(pk), for all Uy, Sy. This approach allows us to model the variability in feature occurrence that commonly observed real-world datasets while still maintaining the core principles of our learning framework. We set the number of features, K 50, with ps sampled from a uniform distribution, (00,022) Each chunk contained 1000 data points with total 50 experiments, with y 50, 3. We sampled 10000 test data from the same distribution.
1 (by,k (==))
As shown Figure 3, the results align with the above theorems. Random initialization, i.e. cold- starting, and ideal initialization achieve almost identical generalization performance, outperforming warm initialization. However, with warm initialization, the model converges faster, as evidenced by the number of non-zero gradient data points, which serves proxy for training time. Ideal initialization requires less time compared to cold-starting, which also consistent with Theorem 3.6. Due to the sampling process in our experiment, we observe gradual increase in the number of learned features and test accuracy warm-starting, mirroring real-world observations. These findings remained robust across diverse hyperparameter settings (see Figures 9-11 in the Appendix B).
7

---

Number of Leammed 1
92
Number oe Zz0 Grademn Dota
50000
40000
Teat Accuracy
Random
30009
Warm
60-
60-
Idiaal
50
20000
75-
40
0-
Number of Experiments
Number of Experiments
Number of Experiments
Figure 3: Comparison of random, warm, and ideal methods across random seeds (mean + dev). The test accuracy (left) and the number of learned features across all classes (middle) are nearly identical for random and ideal initializations, causing their plots to overlap. Warm initialization, however, exhibits lower test accuracy compared to both methods. Regarding training time (right), there a significant gap between random and warm initialization, which the ideal method addresses.
4 DASH: Direction-Aware SHrinking
The ideal method recycles memorized training samples by forgetting noise while retaining learned features. From now on, shift our focus to practical scenario: training neural networks with real-world data. This brings up the question of whether such an ideal approach can be applied in real-world settings. To address this, we propose our algorithm, Direction-Aware SHrinking (DASH), which intuitively captures this idea in practical training scenarios. The outlined behavior is illustrated in Figure 4. When new data is introduced, DASH shrinks each weight based on its alignment with the negative gradient of the loss calculated from the training data, placing more emphasis on recent data.
If the degree of alignment is small (i.e., the cosine similarity is close to or below 0), consider that the weight has not learned proper feature and shrinks significantly to make "forget" learned information. This allows weights to forget memorized noises and easily change their direction. On the other hand, if the weight and negative gradient are well-aligned (i.c., the cosine similarity is close to 1), we consider learned features and we shrink the weight to lesser degree to maintain the learned information. This method aligns with the intuition of the ideal method, as it allows us to shrink weights that have learned proper information while retaining weights that have learned commonly observed features.
Algorithm 1 Direction-Awart SHrinking (DASH)
Require:
Model fe with list of parameters θ after the (j -1)- th experiment
Training data points Try
Averaging coefficient < a S 1
Ny (G: w
Threshold > 0
Initialize:
Feature
G + 0, in
0
2: for i 1: do
3:
ft Loss(fe, Tt)
Vo LIO:T 1
Ue +- Gradient
f
@ θ do
6:
G + (1 a) x c (i--))
+ x Up
02
7:
end for
8: end for
Noise
for in θ
Figure 4: Illustration of DASH. We compute the loss L with training data Ti and obtain the negative gradient. Then, we shrink the weights proportionally to cosine similarity between the current weight and L. resulting in 0.
CosineSimilarity (-ob²),))
10:
11: 8 max(A, so}
12: end for
13: return model fe. initialized for the j-th experiment
The shrinking is done per neuron, where the incoming weights are grouped into a weight vector denoted as 8. For convolutional filters, the height and width of the kernel are flattened to form single weight vector for each pair of input and output filters. DASH has two hyperparameters: and ar. Hyperparameter X is the minimum shrinkage threshold, as each weight vector is shrunk by max (A, cos_sim), while a denotes the coefficient of exponential moving average of per-chunk loss gradients. Lower value gives more weight to previous gradients, resulting in less shrinkage. This

---

is because gradients of the previously seen data usually have high cosine similarity with learned weights; low a is advantageous for simpler datasets where preserving learned features helps. Note that DASH is an initialization method applied only once when new data is introduced. The detailed algorithm is presented in Algorithm
We discuss our intuition regarding the connection between DASH and the spirit of the ideal method. Generally, features from previous data are likely reappear frequently in new data, as they are relevant to the data class. In contrast, noise from previous data rarely reappears as it is not class- specific and varies with each data point. As result, the negative gradient of the loss naturally aligns more closely with learning features rather than memorizing noise from older data. This suggests that when neurons are aligned with its negative gradient of the loss, we can assume they have learned important features and should not be shrunk.
To validate our intuition, we plotted the accuracy on previously learned data points a few epochs after applying DASH in Figure 13, Appendix B. Our experiments show that DASH recovers training accuracy on previous datasets more quickly than other methods, likely because it preserves learned features while discarding memorized noise, As experiments progress, the growing number of learned features allows DASH retain more information, leading improved training accuracy across successive experiments. We further validate our intuition behind DASH in Figure 14 in Appendix B.
5 Experiments
5.1 Experimental Details
Our setup is similar to the one described Ash and Adams (2020). We divided the training dataset into 50 chunks, and at the beginning of each experiment, chunk is added to the existing training data. Models were considered converged and each experiment was terminated when training accuracy reached 99.9%, aligning with our learning framework. We conducted experiments with vanilla training ie. without data augmentations, weight decay, learning rate schedule, etc. Appendix C.4 presents additional results other settings, including the state-of-the-art (SoTA) settings that include the techniques mentioned above. We evaluated DASH on Tiny- Imagee,, CIFAR-1O, CIFAR- 100, and SVHN using ResNet-18 18, VGG-16, and three-layer MLP architectures with batch normalization layer. Models were trained using Stochastic Gradient Descent (SGD) and Sharpness- Aware Minimization (SAM) (Foret et al., 2020), both with momentum.
DASH was compared against baselines (cold-starting, warm-starting, and S&P (Ash and Adams, 2020j) and methods addressing plasticity loss under non-stationarity (1.2 INIT (Kumar et al., 2023) and Reset (Nakkiran et al., 2021)). Layer normalization (Ba et al., 2016) and SAM (Foret et al., 2020), known to mitigate plasticity loss in reinforcement learning (Lee et al., 2023), were applied to both warm and cold-starting. Consistent hyperparameters were used across all methods, with details provided in Appendix C.3. S&P, Reset, and DASH were applied whenever new data was introduced. We report two metrics for both test accuracy and number of steps required for convergence: the value from the final experiment and the average across all experiments.
5.2 Experimental Results
We first experimented with CIFAR-10 on ResNet- 18 to determine if methods from previous works for mitigating plasticity based on non-stationarity can be a solution to our incremental setting with stationarity. Appendix C. shows that L2 INIT, Reset, layer normalization, and reviving dead neurons, are not effective our setting. Thus, we conducted the remaining experiments without these methods. Additionally, Table 1 shows that warm-starting with SAM does not outperform cold-starting with SAM, indicating that SAM alone is not an effective method in our case. Table shows that DASH surpasses cold-starting (Random Init) and S&P in most cases. Training times were often shorter compared to training from scratch, and when longer, the performance gap in test accuracy was more pronounced. Omitted results are in Tables 3-6 located in Appendix C.2. Additionally, we confirm that DASH is computationally efficient, with details on the computation and memory overhead comparisons provided in Appendix C.5.
We argue that S&P can cause the model to forget learned information, including important features, due to shrinking every weight uniformly and perturbing weights. This leads to increased training time and relatively lower test accuracy, especially in SoTA settings (see Appendix C.4.1). In contrast, DASH addresses these issues by preserving learned features with direction-awart weight shrinkage.

---

Table 1: Results of training with various datasets using ResNet-18. Bold values indicate the best performance. For the number of steps, bold formatting is used for all methods except warm-starting Results are averaged across five random seeds, except for Tiny-Imagell which uses three random seeds. Standard deviations are provided in parentheses.
Test Acc at Last Experiment
Number of Steps
AVG of Test Acc
AVG of Number of Steps
ResNet-18
Last Experiment
across All Experiments
accoss All Exporiments
T-ImageNNer
SGD
SAM
SGD
SAM
SGD
SAM
SGD
SAM
Random Init
25,00 (0.13)
(0.09)
30237 (368)
40142 (368)
17.37 (0.00)
21.95 (0.11)
17503 (53)
22513 (74)
Warm Init
9.57 (0.24)
13.94 (0.37)
3388 (368)
5474 (0)
6.70 (0.04)
9.88 (0.21)
1785 (5)
2773 (7)
S&P
34.34 (0.48)
37.39 (0.18)
13815 (368)
26066 (1606)
25.43 (0.02)
28.47 (0.08)
7940 (15)
13172 (182)
DASH
46.11 (0.34)
49.57 (0.36)
8341 (368)
12251 (368)
33.06 (0.15)
35.93 (0.17)
4439 (48)
7900 (136)
CTFAR-10
Random Init
67.32 (0.51)
75.68 (0.39)
5161 (156)
17125 (292)
$7.66 (0.11)
66.27 (0.13)
2916 (37)
8121 (26)
Warm Init
63,53 (0.56)
70.99 (0.59)
1173 (0)
3910 (247)
54.87 (0.18)
63.27 (0.55)
665 (11)
2153 (23)
S&P
81.25 (0.14)
85.53 (0.22)
5395 (625)
32649 (978)
71.74 (0.16)
76.19 (0.04)
2766 (53)
15552 (1558)
DASH
84.08 (0.52)
86.75 (0.53)
6490
11886 (2771)
75.21 (0.33)
77.59 (0.69)
3454
8689 (527)
CHAAR-10
Random Init
(0.14)
40.27 (0.31)
10557 (247)
14310 (191)
(0.7)
(0.00)
5803 (79)
7588 (54)
Warm Init
(0.59)
32.02 (0.31)
1173 (0)
2346 (0)
19.18 (0.52)
24.01 (0.33)
854 (23)
1294 (12)
S&P
50.08 (0.23)
$2.95 (0.36)
4926 (191)
12277 (1226)
37.32 (0.14)
40.36 (0.18)
2929 (27)
S954 (187)
DASH
57.99 (0.28)
60.88
3519 (0)
11730
43.99 (0.14)
46.15 (0.58)
2041
6675 (797)
SVHN
Random Init
86.27 (0.46)
(0.24)
(156)
10869 (156)
78.01 (0.10)
83.31 (0.14)
3099 (1.5)
5546 (44)
Warm Init
84.01 (0:41)
88.85 (0.29)
938 (191)
1329 (191)
75.37 (0.50)
81.16 (0.54)
642 (18)
993 (15)
S&P
92.67 (0.17)
94.27 (0.07)
3597 (156)
1573 (191)
87.35 (0.14)
89.35 (0.05)
1858 (12)
5548 (94)
DASH
93.67 (0.13)
95.19 (0.09)
5161 (672)
14467 (989)
89.59 (0.07)
91.67 (0.03)
2619
8613 (728)
Theorem 3.6 shows that ideal initialization can achieve the same test accuracy as cold-starting. Yet in practice, DASH surpasses cold-starting in test accuracy. This could be due to the difference between the discrete learning process in our framework and the continuous learning process in real-world neural network training. Even if features have already been learned, DASH can learn them in greater strength compared to learning from scratch by preserving previously learned features during training.
Further insights into the applicability of DASH can be found in Appendix C.4. We evaluate DASH in various settings beyond our original setup. In the SoTA setting, different observations are made: DASH achieves test accuracy close to (but does not outperform) cold-starting, without requiring additional hyperparameter tuning, which aligns more closely with our theoretical analysis. We demonstrate DASH's scalability large-scale datasets such ImageNet-Ik. We also examine two additional practical scenarios: data-discarding setting and a situation where new data are continuously added. In such cases, applying DASH with an interval (rather than upon every arrival of new data) proves effective. Finally, we explore DASH's behavior in non non-stationary environments, specifically in Class Incremental Learning (CIL) with data accumulation settings.
6 Discussion and Conclusion
In this work, we defined an abstract framework for feature learning and discovered that warm-starting benefits from reduced training time compared random initialization but can hurt the generalization performance of neural networks due to the memorization of noise. Motivated by these observations, we proposed Direction- SHrinking (DASH), which shrinks weights that learned data cata-specifiic noise while retaining weights that learned commonly appearing features. We validated DASH in real-world model training. achieving promising results for both test accuracy and training time.
Loss of plasticity is problematic in situations where new data is continuously added daily, which is the case in many real-world application scenarios. Our research aimed to interpret and resolve this issue, preventing substantial waste of energy, time, and the environment. By elucidating the loss of plasticity phenomenon in stationary data distributions, we have taken a crucial step towards addressing challenges that may emerge in real-world AI, where the continuous influx of additional data is inevitable.
We hope our fundamental analysis of the loss of plasticity phenomenon sheds light on understanding this issue as well as providing remedy. To generalize findings any neural network architecture, we treated the learning process as a discrete abstract procedure and did not assume any hypothesis class. Future research could focus on understanding the loss of plasticity phenomenon via optimization or theoretically analyzing it in non-stationary data distributions, such as in reinforcement learning.
10

---

Acknowledgement
This work was partly supported by National Research Foundation of Korea (NRF) grant (No. RS-2024-004212033 funded by the Korean government (MSIT), and an Institute for Information & communications Technology Planning & Evaluation (IITP) grant (No. RS-2022-11220 Develop- ment and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics) funded by the Korean government (MSIT). This research was supported in part by the NAVER-Intel Co-Lab. The work was conducted by KAIST and reviewed by both NAVER and Intel.
References
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations, 2018.
Zeyuan Allen- Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 2020.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks, In International conference on machine learning, pages 233-242.
PMLR, 2017.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in neural information processing systems, 33:3884-3894, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06-450 2016.
Tudor Berariu, Wojeiech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks. arXiv preprint arXiv:2106.00044 106.00042, 2021.
Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convo- lutional neural networks. Advances in neural information processing systems, 35:25237-25250, 2022.
Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Adelani, Pontus Lars Erik Saito Stenetorp, Sebastian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active forgetting. Advances in Neural Information Processing Systems, 36:31543-31557, 2023.
Yihe Deng, Yu Yang, Baharan Mirzasoleiman, and Quanquan Gu. Robust learning with progressive data expansion against spurious correlation. Advances in Neural Information Processing Systems, 36, 2023.
Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint (iXX::22088.06255. 2021.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020..
Jonathan Frankle, David J Schwab, and Ari S Morcos. The early phase of neural network training. arXiv preprint arXiv20002.10365 2020.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. arXiv preprint arXiv:2006.05826, 2020.
Samy Jelassi and Yuanzhi Li. Towards understanding how momentum improves generalization in deep learning. In International Conference on Machine Learning, pages 9965-10040. PMLR, 2022.
Yiding Jiang, Christina Baek, and J Zico Kolter. On the joint interaction of models, data, and features. In The Twelfth International Conference on Learning Representations, 2024. URL

---

Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data overparameterized models. arXiv preprint N.Kv::2002.032006 2020.
Michael Kleinman, Alessandro Achille, and Stefano Soatto. Critical learning periods emerge even in deep linear networks. In The Twelfth International Conference on Learning Representations, 2024. URL sttpss//openreviii
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308. 17958, 2023.
Hojoon Lee, Hanseul Cho. Hyunseung Kim, Dachoon Gwak, Joonkee Kim, Jacgul Choo, Se- Young Yun, and Chulhee Yun. Plastic: Improving imput and label plasticity for sample efficient reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023.
Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos Machado. Curvature explains loss of plasticity. arXiv preprint (tt.Xii:2322.00224/. 2023.
Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and sgd can reach them. Advances Neural Information Processing Systems, 33:8543- 8552, 2020.
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Razvan Pascanu, James Martens, Hado van Hasselt, and Will Dabney. Towards perpetually trainable neural networks, 2023a.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, pages 23190-23211. PMLR, 2023b.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):12400) 2021.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference machine learning, pages 16828-16847. PMLR, 2022.
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems, 36, 2023.
Junsoo Oh and Chulhee Yun. Provable benefit of cutout and cutmix for feature learning. In
The eighty- Annual Conference on Neural Information Processing Systems, 2024. URL
https: / //pprrevee//eeeiie... htwptaawwwwwwweeeeeeeeeeeeeeeeee wett:/oorumidd-oonddttSt.
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference machine learning, pages 2217-2225. PMLR, 2016.
Ruoqi Shen, Sébastien Bubeck, and Suriya Gunasekar. Data augmentation as feature manipulation. In International conference on machine learning, pages 19773-19808, PMLR, 2022.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning, pages 32145-32166 PMLR, 2023.
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. Pretrained language model continual learning: comparative study. In International conference on learning representations, 2021.
Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. The benefits of mixup for feature learning. arXiv preprint arXiv:2303.084. 2023.
12

---

Contents
1
Introduction
1.1
Our Contributions
1.2
Related Works
2
A Framework of Feature Learning
2.1
Motivation and Intuition
2.2
Training Process
2.3
Discussion on Training Process
3
Warm-Starting versus Cold-Starting, and a New Ideal Method
3.1
Experiments with Expanding Dataset
3.2 Prediction Process and Training Time
3.3
Comparison Between Warm-Starting and Cold-Starting in Our Framework
3.4 An Ideal Method: Retaining Features and Forgetting Noise
DASH: Direction-Aware SHrinking
5
Experiments
5.1 Experimental Details
5.2
Experimental Results
6 Discussion and Conclusion
10
Detailed Intuition and Example of Our Framework
14
Connection Between Our Framework and Practical Scenarios
16
B.1 Justification Our Framework
16
B.2 Further Results on Warm-starting vs. Cold-starting
17
B.3 Experiments Supporting the Intuition behind DASH
19
Omitted Experimental Results
20
C.1 Methods for Non-Stationary Data Distribution Struggle in Stationary Settings
20
C.2 Results of Experiments Across Different Datasets, Models, and Optimizers
21
C,3 Hyperparameters
C.4 Discussions on the Broader Applicability of DASH
24
C.5 Computation and Memory Overhead Comparison
28
C.6 Comparison between NVIDIA A6000 and Intel Gaudi-v2
29
D Proof of Theorems
30
Omitted Algorithms
33
13

---

A Detailed Intuition and Example of Our Framework
In this section, we provide a detailed explanation of how our theoretical framework reflects the intuitive process of learning from image data. Our framework is designed to capture the characteristics of image data, where the input includes both relevant information for the image labels (which we refer to as "features" such as cow's ' ears, eyes, tail, and mouth in Figure 5a) and irrelevant information (which we refer to as "noise" such as the sky or grass circled in red Figure Sa). Our framework builds on the insights from Shen et al. (2022) and incorporates them into a discrete learning model.
Each point associated combination of features
(a) Examples of data points in the feature learning framework. Data points æ1 through Ds denotes each data point composed of a combination of features (left side of the dashed line). For instance, the first data point the left (æ1) correspond to the features of the first image the right, while the last data point the left (æa) correspond to the features of the second image the right. All data points also contain data-specific noise, such as sky or grass in cow images, circled red in cow image on the right.
each step, model the most frequent feature
Data points with sufficiently many learned features are well-classified
Frequency of features evaluated solely on not-well-classffied data points
Learned feature
Well-classified data
features are frequent enough, the model memorizes remaining data
Memarized data
(b) An illustration of our feature learning process framework. The model sequentially learns the most frequent features from the not-well-classified data points, and if the features insufficient to be leamed properly, the process ends by memorizing the noise. With 7 and Y = 2, the model learns features sequence (violet, blue, green). The remaining data point which corresponds to xg in Figure 5a is memorized with noise.
Figure 5: An illustration of our proposed feature learning framework with single class. Each data point is composed of features, shown in Figure 5a, and learned through the framework depicted in Figure 5b.
14

---

Before explaining the intuition of our learning framework, let us connect notation from Section 2 with Figure Figure shows six data points from a single cow class, where training set 7 {(z,, y) he Features are denoted by their color's s first letter: violet as blue as b, green as g, mint as m, yellow as y, and pink as p. The complete feature set S is therefore {v, b, g: m, p). The learned feature set L. is empty at initialization, and the set of data points with non-zero gradients . C ' is equal to T at initialization. We set the data learning threshold T = 2 and noise strength ? =
Our training process is based on the idea that features which occur more frequently in the training data are easier to learn, as the gradients tend to align more strongly with these frequent features. Consequently, our framework is designed to learn the most frequent features in sequential manner. For example, cow's s ears, shown in violet in Figure would be selected first since it is the most frequent feature:
arg max Ilc T,,1)) arg www.eeev.b.gg........ (T,TT)
where ((*) and A represent the learned feature set and the set of data points with non-zero gradients, respectively, at step 8. This holds because:
g(v: T(v:T. = - Σ
1(v E V(x)
5 6
(a.9)ENO)
g(c; T.A'))
for all C m.ypp. Since have set y to 2, the feature v is learned because its frequency is sufficiently high compared to the noise, i.e., g(v; (.)))) 2 7/|T| zh Also, since set to 2, no data point is considered well-classifedd at this stage, i.e., [[v) (2;))) < T for all i E [6]. As a result, ((²) = (v). A/()) T.
In the next step, the model learns feature b in the same way. Therefore, I,, and 24 are now considered well-classified since they share two overlapping features with the learned feature set ((2) b). This indicates that the model accumulates enough information accurately classify these cow images as cows. Once the model correctly classifies data based on these features, the influence of those data points on the learning process diminishes, as the loss gradients become smaller as the predictions become more confident. To account for this, our framework evaluates the frequency of features only on data points that are not yet well-classifedd. Therefore, the set of data points with non-zero gradients is now updated to Ar(2) ((x,, (2.366)) Subsequently, we compute the gradient proxy g(c; T.)(2)) using only the data points in A/22) In the third step, considering the features included in {æ>, 2s, 3, we get that the feature is the most frequent among those in S \ (2) and the feature meets the threshold for learning. Hence, the feature is newly learned, leading us to (3) {v, b, g] and A/(3) { y)}.
As training progresses, the algorithm may reach a point where the remaining features the not-well- classified data points are too infrequent to be learned effectively. At this point, the noise in each data point has a faster learning speed, which leads the model memorize noise instead of learning the features to achieve 100% training accuracy. In the example described in Figure 5, at the fourth step, the remaining feature m in Dg does not meet the threshold: (mm; T,N ,N(3)) 1/6 < 7/IT]. so the model unable learn feature m. Consequently, the model shifts memorizing the noise in the remaining data point Øg and completes training by memorizing the noise.
15

---

B Connection Between Our Framework and Practical Scenarios
This section demonstrates how our theoretical framework relates real-world scenarios. Section B.1 presents real-world experimental evidence supporting our theoretical framework. Section B.2 explores the robustness of our synthetic experiments across various hyperparameters. Finally, Section B.3 provides empirical validation of the key intuitions behind DASH.
B.1 Justification of Our Framework
Figure shows that the initial gradient norm of training data, [, can be proxy for the training time until convergence. As the initial gradient norm increases, the number of steps required for convergence also increases. While this figure uses the gradient norm instead of the number of non-zero gradient data points due to the continuous nature of real-world neural network training, we believe it resembles the behavior of non-zero gradient data points. Additionally, Figure 7 demonstrates that the number of steps required for convergence increases as the number of data points increases in various datasets.
Warm
Random
3000
12000
50
Number of Steps
10000
40
Number Experiments
2000
30
6000
20
1000
500
2000
40
60
100 120
140
0
250 500 750 1000 1250 1500 1750 Initial Gradient Norm of Training Data
Initial Gradient Norm Training Data
Figure 6: Trained on ResNer-18 with five random seeds, where CIFAR-10 is divided into 50 chunks and incrementally increased by adding new chunks at each experiment. Each point represents an individual experiment. The gradient norm is used as a proxy for the number of non-zero gradient data points, which in turn serves proxy for the training time. A larger gradient norm indicates the model needs to learn more features or memorize more data points to correctly classify all training data points.
F-MNIST
CIFAR-10
CIFAR-100
T-ImageNet
Number of Steps
25000
20000
15000
10000
5000
10
20
50
0 10
20
50
10
20
so
10 20
30 40 So
Number of Experiments
Number of Experiments
Number of Experiments
Number of Experiments
Figure 7: Figure trained on ResNet-18 with three random seeds. The dataset is divided into 50 chunks, and new chunks incrementally added for each experiment. The number of steps required for convergence increases with the amount of data when training a cold-started neural network, which is standard training.
We also investigate the effect the pretrain epoch on warm-starting in ImageNet 1k classification with 18, similar to the setup in Figure 2 in Section 3.3. We conducted experiments using different pretrain epochs: 30, 50, 70, and 150. Figure shows a declining trend accuracy for warm-started models as pretrain epoch increases. Interestingly, we noted a similar phenomenon in Figure in Section 3.3, where warm-starting does not negatively impact test accuracy when training resumes from earlier pretrained epochs, such as 30 in this case. This observation aligns with our
16

---

theoretical framework, which suggests that neural networks tend to learn meaningful features first and then begin to memorize noise later. We believe this phenomenon is persistent across different datasets, model architectures, and optimizers.
102
Test Accuracy (%))
60
Train Accuracy
50-
Cold
10-
30 Pretrain
Pretrain
-120
-50
50 150
-100 -50
50
100
152
Epochs
Epochs
Figure 8: Effect of pretraining epochs on warm-starting with the large-scale ImageNet- Ik dataset using ResNet-18. 18. The plots to the left of the dashed line represent the pretraining results, and the dotted lines indicate the starting points for each pretraining run. Test accuracy declines as the number of pretraining epochs increases, particularly beyond 50 epochs, which aligns with the observations in Figure in Section 3.3.
B.2 Further Results Warm-starting vs. Cold-starting
We conducted synthetic experiments across a wide range of hyperparameters. Figure '9 uses the same setup Section 3.4 but varies numbers of classes, C. Figure 10 investigates varying noise signal strengths, while Figure 11 explores different values of т. These results align with our findings from Theorems 3.4 and 3.6.
Numrec Bepriments
1.
Numba excomnetts
Nunoe Exprmmens
C=5
Namoer Eepriments
Numba txpelnerts C-
Nunoe Expermenns
Nameeron Expriments
Number Expeitments
Nunoer of Experments
Figure 9: Results using the same hyperparameters as described Section 3.4, except for the number of classes (C). Experiments persists across different values of C.
conducted with 10 random seeds. The trend observed in Figure
As stated in Section 2, we posited that could serve as proxy for dataset complexity. Figure 11 shows that increases, the threshold for considering data point well-classified also increases, making more difficult to correctly predict unseen data points. This difficulty is particularly pronounced for warm-starting leading to widening gap between the random initialization and warm initialization methods. Additionally, this phenomenon observed real-world neural network training, depicted in Figure 12. For datasets with higher complexity (from left to right), the gap between the two initialization methods widens, exhibiting the same trend as an increasing T.
17

---

10
Random
Warn
looal
Dicedments
Number Eipatments 30
Taper trents
Exsemmntt
Excrmena
txpa more
50
Expertments
Numbar Experment
Expar
Figure 10: Results using the hyperparameters described in Section 3, except for the strength of the noise, Experiments were conducted with 10 random seeds. The trend observed Figure persists across different values of 7.
T=1
fundsm
Wor
Ide
Namber tapamenn T-2
Expermena
Tos Acourcee
Number Eapa mom T=3
Expermeens
Tes accunon
bxpet
Expa
Experrenn
Figure 11: Results using the same hyperparameters as described in Section 3, except for the threshold for data point is considered well-classified (r). Experiments The trend observed in Figure 3 persists across different values of т.
conducted with 10 random seeds.
F-MNIST
CIFAR-10
CIFAR-100
T-ImageNet
25
90.0
Rest Accurasy 05)
60
30
87.5
50-
20
10-
80.0
Warm
10
30
10
20
30
40
50
10
20
30
40
50
10
20
10
0
10
20
30
40 50
Number of Experiments
Number Experiments
Number Experiments
Number Experiments
Figure 12: The same hyperparameters used described in Section 5.1 with three random seeds. The gap in test accuracy between the two initialization methods increases as dataset complexity increases.
18

---

B.3 Experiments Supporting the Intuition behind DASH
To validate whether previously learne//memorized data points do not have large gradients when further trained with combined dataset (existing + newly introduced data) using warm-starting, we plotted the train accuracy on the previous dataset for the first few epochs. Using ResNet- 18 on CIFAR- 10 (Figure 13), we found that warm-starting preserves performance on previously learned data even when training continues with combined datasets, supporting the main idea of Theorem 3.4.
To verify whether DASH truly captures our intuitions from the ideal algorithms, we conducted an experiment using CIFAR-I0 10 trained ResNet-18, with the same experimental settings. Figure 13 demonstrates that when applying DASH, the train accuracy on previous datasets increases more rapidly after a few epochs compared to other methods. We argue that this behavior stems from our algorithm's ability to forget memorized noise while preserving learned features. As the number of experiments increases, the number of learned features also grows. For fair comparison, we used = 0.05 for DASH, and when performing S&P and shrink. we shrank each weight by multiplying 0.05. In the case of S&P, after shrinking, we added noise sampled from N (O, 0.013).
Bandom
Number Exper impetteets
Number of Experiments
Number of Experiments
Figure 13: The results are averaged over random seeds. The x-axis represents the number of experiments, while the y-axis represents the training accuracy previous datasets. Warm-starting can retain previously learned data points when further trained with an incremented dataset. Additionally, DASH, plotted in green, can retain more information compared to other methods.
further validate our intuition that cosine similarity between the negative loss gradient and model weights can indicate whether the model has indeed learned features. We trained a 3-layer CNN on CIFAR-10, varying the size the training dataset. We observed that the cosine similarity between the negative gradient from the test data and the learned filters increases the training dataset size grows. The model trained with more data appears to lear more features, as evidenced by the rising trend in test accuracy, shown by the dashed line in Figure 14. This suggests that cosine similarity can capture whether the weights have learned features, which aligns with our expectations.
80
0.8
770
Cosine Similarity
60
Test Accuracy 196)
0.6
50
0.4
40
30
0.2
20
1000
25000
40000
50000
Train Data Size
Figure 14: We trained a 3-layer CNN CIFAR-10 and plotted cosine similarity values greater than 0.1. The left y- y-axis (black) represents the cosine similarity between the negative loss gradient from the training data and the model filters, while the right y-axis (green) shows the test accuracy.
19

---

C
Omitted Experimental Results
This section presents additional experimental results and their corresponding hyperparameters. Ap- pendix C.1 examines non-stationary solutions, demonstrating their limitations in stationary settings. The following section compares DASH against other baselines in various settings, excluding those methods from Appendix C.1 due to their poor performance in stationary conditions. Appendix C.3 provides a comprehensive list of hyperparameters used across experiments from Section as well as Appendix C.I and C.2. Additionally, we validate DASH's applicability including state-of-the-art (SoTA) settings in Appendix C.4, and analyze computational and memory overhead Appendix C.5.
We ran our experiments on two distinct hardware setups. The first server had an Intel Gaudi-v2 HPU with 96GB of VRAM with four Intel Xeon Platinum 8380 40-core CPUs. The second server had an NVIDIA A6000 GPU with 48GB of VRAM, paired with two AMD EPYC 7763 64-core CPUs. We compared the computing performance of these two hardware setups in Appendix C.6.
Unless specified otherwise, we follow the experimental settings outlined in Section 5.1. We conducted an experiment with incremental training dataset comprised of 50 chunks. At the start of each experiment, a new chunk is provided and added to the existing training dataset. Before proceeding to the next experiment, the model is trained until achieving 99.9% train accuracy.
C.1 Methods for Non-Stationary Data Distribution Struggle in Stationary Settings
In this subsection, we describe solutions that aim to mitigate plasticity loss under non-stationarity, which cannot remedy the loss of plasticity in an incremental setting with stationary data distribution. Table 2 shows L2 INIT (Kumar et al., 2023) and Reset (Nikishin et al., 2022) cannot be a solution in setting.
Table 2: Results of training CIFAR-10 dataset trained on various models with solutions proposed to mitigating loss of plasticity in non-stationary data distributions. Bold values indicate the best performance. For the number of steps, we did not provide bold formatting. Results are averaged over three random seeds, with standard deviations provided in parentheses.
Test Ace at
Number Steps
AVG of Test Ace
AVG of Number of Steps
CIFAR-10
last experiment
last experiment
across all experiments
across all experiments
ResNer-18
SGD
SAM
SGD
SGD
SAM
SGD
SAM
Random Init
66.75 (0.55)
75.55 (0.18)
(184)
17734 (184)
57.82 (0.04)
66.19 (0.01)
2889 (24)
8100
Warm Init
64.10 12)
20.56 (0.30)
1173 (0)
4040 (184)
55.11 (0. 10.10)
62.94 (0.47)
726 (29)
2160 (11)
INIT
64.24 (0.80)
70.32 (0.09)
1173 (0)
4040 (184)
55.47 (0:43)
62.55 (0.19)
(14)
2139 (15)
Reset
(0.95
72.03 (0.33)
(1
17986 (1596)
55.55 (0.35(
63.40
976 (51)
7225 (10)
VGG-16
Random Init
(0.35)
(1475)
(1243)
(0.22)
12743 (280)
(343)
Warm Init
78.93 (0.44)
82.04 (0.04)
1825 (184)
4692 (319)
70.62 (0.24)
74.00 (0.33)
1954 (42)
4277 (315)
INIT
82.79 (0.04)
82,11 (0.19)
193936 (58167)
6126 (665)
72.11 (0.14)
73.77 (0.37)
12489 (443)
4390 (94)
Reset
(0.26
81.88 (0.35)
1564 (0)
3910 (552)
70.45 (0.36)
73.31
1814 (30)
3230
MLP
Random
(0.13)
(184)
(184)
(0.42)
(0.24)
(166)
(127)
Warm Init
56.44 (0.33)
57.67 (0.45)
2346 (0)
2216 (184)
50,60 (0:41)
$1.98 (0. 14)
2309 (408)
1701 (34)
INIT
56.38 (0.39)
58.24 (0.06)
1955 (0)
2085 (184)
50.56 (0.51)
$2.15 (0.33)
2221 (423)
1604 (73)
(0.09)
6125 (487)
(184)
50.70
5955 (740)
2463
Furthermore, applying layer normalization cannot close the gap between cold-starting and warm-
starting; rather, the gap increases, as shown in Figure 15. Also, Nikishin et al. (2022) and Sokar et al. (2023) state that loss of plasticity in non-stationary data distributions arises from inactive neurons in the model. However, this is not the case in our setting, demonstrated in Figure 16.
20

---

72- Warm
02-
32-
20
40
Number of Experiments
Figure 15: The figure shows the results of training ResNet-18 18 CIFAR-10 with three random seeds. Layer normalization (dashed lines) is applied in place of batch normalization in ResNet-18, while solid lines represent the of standard batch normalization. The red lines denote warm-starting, and the blue lines denote cold-starting. The figure demonstrates that the layer normalization technique cannot serve solution for plasticity loss. Moreover, the gap between warm-starting and cold- starting performance increases when layer normalization is employed.
Block
Block
Block 3
Block
Dead Neuren Rabio
0.000
0.025
-0,050
10
20
30
40
50
50
10
20
30
40
50
10
20
30
40
50
Number Experiments
Number of Experiments
Number of Experiments
Number Experiments
Figure 16: The figure presents the results of training ResNet-18 on CIFAR-10 with three random seeds. The presence of dead neurons is assessed after each block ResNet-18 with training dataset, and the analysis reveals that there to revive dead neurons in non-stationary data distributions cannot effectively address the plasticity loss observed in the incremental learning setting with stationary data, which is the primary focus of our study.
dead neurons. This finding suggests that techniques designed
C.2 Results of Experiments Across Different Datasets, Models, and Optimizers
This subsection presents complete experimental results (Tables 3-6), following the settings detailed in Section 5. We evaluated multiple approaches, including "Warm ReM" (warm-starting with momentum reset), which also proved ineffective. We conducted comprehensive comparisons across multiple datasets (Tiny-ImageNet, CIFAR-10, CIFAR-100, and SVHN), architectures (ResNet- 18, VGG-16, three-layer MLP)), and optimizers (SGD and SGD-based SAM). The results demonstrate that DASH consistently outperforms baseline methods warm-starting cold-starting, and S&P while often requiring less training time. Results are averaged across five random seeds, except for Tiny-ImageNe which uses three random seeds due to its high computational cost.
21

---

Table 3: Average results from three training runs using various models on the Tiny-Imagell dataset.
Test Acc
Number Stops
AVG Test Ace
AVG of Number Steps
T-ImageNet
last experiment
last experiment
across all experiments
across all experiments
ResNer-18
SGD
SAM
SGD
SAM
SGD
SAM
SGD
SAM
Random Init
25.69 (0.13)
31.30 (0.09)
30237 (368)
40142 (368)
17.37 (0.06)
21,95 (0.11)
17503 (53)
22513 (74)
Warm Tnit
9.57 (0.24)
13:94 (0.37)
3388 (368)
5474 (0)
6:70 (0.04)
9.88 (0.21)
1785 (5)
2773
Warm ReM
9.20 (0. 16)
[3.71 (0.29)
3388 (368)
5474 (0)
6.67 (0.08)
9.93 (0.30)
1787 (17)
2795 (14)
(0.48)
37.39 (0.18)
13815 (368)
26066 (1606)
25.43 (0.02)
28.47 (0.08)
7150
13172 (182)
DASH
46.11 (0.34)
49.57 (0.36
8341 (368)
12251 (368)
33,06 (0.15)
38,93 (0.17)
4439 (48)
7900 (136)
VGG-16
Random Init
40,26 10 30)
42,41 (0.13)
92927 (8944))
29976 (664)
28, (0.03)
30,40 (0.04)
48878 (799)
17094 (1922
Warm Init
17.11 (0.44)
20.77 (0.32)
1955 (0)
2997 (184)
12.91 (0.18)
15.14 (0.35)
4359 (162)
2513 (13)
Warm ReM
17.51 (0.38)
20.23 (0.06)
2085 (184)
2867 (184)
12.97 (0.24)
14.87 (0.14)
4130 (99)
2472 (8)
S&P
36.56
38.63 (0.73)
59432 (5538)
18898 (368)
23.91 (0.09)
25,98 (0.22)
28747 (366)
10494 (45)
DASH
44.29 (0.55)
44,40 (0.19)
69989 (621.5)
22938 (1329)
28.47 (0.49)
29.11 (0.73)
31864 (362)
14258 (149)
Random Init
(0.06)
9.19 (0.25)
28934 (0)
42749 (975)
6.94 (0.01)
7.22 (0.03)
13596 (35)
17871 (71)
Warm Init
(0.11).1))
7.74 (0.25)
4692 (0)
4952 (368)
6.18 (0.03)
6.41 (0.11)
2437 (17)
2797 (38)
Warm ReM
7.54 (0.19)
7,86 (0.08)
(368)
5474 (0)
6.34 (0.04)
(0.05)
2411 (35)
2821 (36)
S&P
9.61 (0.23)
10.28 (0.25)
33365 (2879)
55782 (975)
7.27 (0.01)
7.57 (0.04)
16227 (1458)
21126 (94)
DASH
10.17 (0.19)
10.77 (0.12)
30237 (975)
47702 (638)
7.67 (0.02)
8.12 (0.03)
17743 (899)
19455 (212)
Table 4: Average results from five training runs using various models the CIFAR-10 dataset.
Test Acc at
Number Steps
AVG Test Ace
AVG Number Steps
CIFAR-10
last experiment
last experiment
across all experiments
across all experiments
RexNer-18
SGD
SAM
SGD
SAM
SGD
SAM
SGD
SAM
Random Init
67.32 (0.51)
75.68 (0.39)
5161 (156)
17125 (292)
57.66 57.66
66.27 (0.13)
2916
(26)
Warm Init
63.53 (0.56)
70.99 (0.59)
1173 (0)
3910 (247)
54.87 (0.18)
63.27 (0.55)
665 (11)
2153 (23)
Warm ReM
63.96 (0.64)
70.82 (0.36)
1173 (0)
3988 (292)
55.03 (0.47)
63.17 (0.60)
703 (49)
2158 (4)
S&P
81.25 (0.14)
85.53 22)
5395 (625)
32649 (978)
71.24 (0. 16)
76.19 (0.04)
2766 (53)
15582 (1558)
DASH
84,08 (0.52)
86.75 (0.53)
6490 (399)
11886 (2771)
75.21 (0.33)
77.39 (0.69)
3454 (55)
8689 (527)
VGG-16
Random Init
84.11
84.67
23225 (2565)
21270
75.64 (0.16)
75.77 (1.81)
12723 (233)
9358 (4306)
Warm Init
79.01 (0.45)
82,09 (0.16)
2111 (530)
4770 (455)
70.90 (0.41)
74.03 (0.26)
1950 (34)
4180 (271)
Warm ReM
78.82 (0.32)
81.66 (0.44)
2737 (2358)
4532 (312)
71.23 (0.31)
73:43 (0.59)
2056 (111)
4051 (63)
S&P.
84,96 (0.46)
88,02 (0:21)
21426 (672)
34251 (10994)
76.62 (0.16)
79.13 (0.19)
11812 (180)
14452 (751)
DASII
87,57 (0.26)
90.68 (0.26)
17008 (2150)
45668 (15184)
79.63 (0.32)
83.07 (0.20)
10171 (266)
20814 (6416)
MLP
Random Enit
57.42 (0.31)
58.53 (0.55)
13528 (191)
19315 (191)
$1.09 (0.37)
$1.94 (0.23)
7598 (167)
9770 (113)
Warm Init
56.28 (0.42)
37.60 (0.37)
2346 (0)
2189 (191)
50.39 (0.41)
51.82 (0.23)
2195 (352)
1706 (27)
Warn RcM
56.07 (0.37)
57.71 (0.42)
2189 (191)
2111 (191)
(0.44)
51.81 (0.26)
2519 (351)
1650 (54)
S&P
57.02 (0.40)
58.19 (0.52)
6647 (428)
7038 (349)
50.87 (0.26)
$1.93 (0.18)
7939 (969)
4188 (112)
DASH
57,41
58.60 (0.36)
6021 (681)
6021 (398)
51.20 (0.25)
52.26 (0.43)
7821 (1308)
3772 (66)
Table 5: Average results from five training runs using various models on the CIFAR- 100 dataset.
Test Acc
Number of Stops
AVG of Test Ace
AVG of Number of Stops
CIFAR-100
last experiment
last experiment
across all experiments
across all experiments
R8
SGD
SAM
SGD
SAM
SGD
SAM
SGD
SAM
Random Init
(0.14)
40.27 (0.31)
10557 (247)
(191)
25.72 (0.11)
(9.90(0.066
5803 (79)
7588 (54)
Warm Init
25.12
32.02 (0.31)
1173 (0)
2346 (0)
19.18 (0.52)
24.01 (0.33)
854 (23)
1294 (12)
Warn RcM
24.83 (0.61)
31.63 (0.58)
1173 (0)
2346
18.98
(0.41)
822 (27)
(14)
S&P
50.08 (0.23)
52.95 (0.36)
4926 (191)
12277 (1226)
37.32 (0.
40.36 (0.18)
2929 (27)
5954 (187)
DASH
57.99 (0.28)
60.85 (0.29)
3519 (0)
11730 (1211)
43.99 (0.14)
46.15 (0.58)
2041 (51)
6675 (797)
VGG-16
Random Init
54.03 (0.45)
$7.29 (2.29)
62560 (5251)
26900 (1512)
39.78 (0)
(1.70)
29436 (477)
18107 (1295)
Warn Init
(1.22
39.91 (0.58)
3362 (191)
4379 (292)
28.98
30.07 (0.51)
4196 (216)
3482 (227)
Warm ReM
58.21 (0.81)
39.58 (0.72)
3362 (191)
3988 (156)
28.98 (0.99)
30.58 (0.49)
4196 (216)
3251 (58)
59.61 (0.43)
63.67 (0.44)
29637 (834)
11573 (635)
45.36 (0.17)
47.92 (0.18)
(149)
7644 (197)
DASHI
59.79 (0.28)
61.91 (1.10)
53109 (1 (11451)
26275 (5716)
44.01 (0.34)
45.38 (0.68)
26577 (4650)
16163 (4461)
MLP
Random Init
28.25 (0.40) 29.46 (0.34)
17516 (292)
25571 (725)
22.39 (0.11)
23.53
13245 (2270)
12467 (301)
Warm Init
26:20 (0.34)
27.45 (0.
3449 (156)
3128 (247)
21.56 (0. 12)
22.47 (0.07)
5461 (1089)
2793 (158)
Warm
(0.26)
27.54 (0.46)
3362. (191)
21.41 (0.05)
22.57 (0.11)
8422 (3059)
2866 (144)
S&P
30.12
30.04 (0.20)
10948 (428)
23851 (1504)
23.44 (0.13)
23.79 (0.09)
37317 (6536)
14873 (810)
DASHI
30.13 (0.35)
31.22 (0.44)
16578 (635)
22052 (805)
23.42 (0.08)
24.43 (0.08)
52328 (3918)
13408 (579)
22

---

Table Average results from five training runs using various models on the SVHN dataset.
Test
Number Steps
AVG of Test Acc
AVG Number Steps
SVHN
experiment
experiment
all experiments
all experiments
RexNer-18
SGD
SAM
SGD
SAM
SGD
SAM
SGD
SAM
Kandom Init
86.27 (0.46)
89.84 (0.24)
3552(156)
10869 (156)
78.01 (0.10)
83.31 (0.14)
3099(13) (13)
5546 (44)
Warm Tnit
(0.41)
88.85 (0.29)
(191)
1329 (191)
(0.50)
(0.54)
642(118)
(15)
Warm ReM
83.85
88.75 (0.27)
782
1485 (156)
75.41 (0.85)
81.03 (0.62)
640
1006
S&P
92.67 (0.17)
94.27 (0.07)
3897 (156)
11573 (191)
87.35 (0.14)
89.35 (0.05)
1858 (12)
5548 (94)
DASH
93.67 (0.13)
95.19 (0.09)
5161 (672)
14467 (989)
89.59 (0.07)
91.67 (0.03)
2619 (68)
8613 (728)
VGG-15
Random Init
93.65 (0.20)
93.88 (0.17)
16187 (1201)
12355 (312)
90.43 (0.05)
90.53 (0.07)
8617 (222)
7379 (275)
Warm Init
92.67 (0.18)
93.08 (0.19)
1485 (625)
938 (191)
89.61 (0.05)
89.80 (0.10)
1122 (34)
959 (37)
Warm ReM
92.85 (0.26)
93.24
1329 (191)
1016 (191)
89.64 (0.27)
89.83 (0.14)
1128 (29)
935 (43)
S&P
94.58 (0.20)
94.83 (0.16)
9853 (758)
8289 (455)
91.82 (0.10)
91.94 (0.09)
5979 (104)
4979 (122)
DASH
94.72 (0.19)
94.84 (0.20)
12668 (1925)
8836 (530)
91.84 (0.10)
92.05 (0.14)
6844 (225)
5769 (331)
MLP
Random Init
82,92 (0.24)
83.68 (0.26)
31768 (1942)
36206 (585)
77.19 (0.13)
78.18 (0.06)
19861 (456)
18278 (277)
Warm Init
81.17 (0.17)
82.29 (0.21)
4789 (324)
2893 (191)
76.51 (0.21)
77.55 (0.07)
7317 (806)
2510 (48)
Warm ReM
81.21
82.25 (0.04)
4398 (507)
3128 (319)
77.46 (0.05)
6147 (553)
2626
S&P
82.07 (0:27)
82.81 (0.33)
28621 (3376)
16734 (518)
77.00 (0.15)
77.94 (0.13)
16530 (1019)
9802 (222))
DASH
82,30 (0.38)
83.02 (0.26)
25571 (1411)
T5405 (944)
76.77 (0.13)
77,89 (0.07)
21092 (1535)
8956 (255)
C.3 Hyperparameters
In this section, we provide the details of the hyperparameters used in our experiments from Section 5, as well as Appendix C.I and C.2. Additionally, we present heatmaps illustrating the results for a wide range of two typerparaeteer,, a and X, in DASH. The heatmaps in Figure 17 suggest that DASH exhibits robustness to hyperparameter variations, indicating that its performance less affected by the choice of hyperparameter values.
We fixed the momentum 0.9 and the batch size to 128. The learning rate is set to 0.001 for training for SAM chosen
ResNet-18 18, and for other models, a learning rate of 0.01 is used. The value based the performance of cold-starting. The default value of a 0.3 is used, and did change this value frequently. The perturbation parameter a used in the Shrink & Perturb (S&P) procedure is and Adams (2020). Initially, we tested o = 0.1 1 as the perturbation parameter, since Ash and Adams (2020) reported slightly better accuracy compared experienced significantly poorer generalization performance with o 0.1 compared to a = 0.01, as shown in Figure 18. The hyperparameters used in our experiments are described in Table 7.
0.01, this value is considered optimal for perturbation,
described in Ash
0.01 in some cases, However,
Experimentts
Stops across Experiment
83.23
91
74.97
336B
3276
540
16:40
06.10
10.
88 (8:40)
74.99
74.05
2384 12261
3250
2853
00.30
16:751
17:30)
$4.28
83.89
75,01
744
74:
2450
3231
2054
541
14.79
-0.4
TUA
19:55
06.21
14,00
Figure 17: The performance of DASH with various hyperparameter values on the CIFAR-10 dataset using a ResNet-18 architecture. Three runs averaged with standard deviation. Darker colors indicate higher values. The first two heatmaps show that higher values preferable, while the last heatmap demonstrates that lower values (brighter colors) are preferable.
23

---

13-52
81:227
16.00
45.00
12:23
71, 65
41
1241
BB
2777
7800 (314
1M
2014
59-16
5
78:00
2004
100
Figure 18: The performance of S&P CIFAR-10 using ResNet-18 with varying values. While Ash and Adams (2020) reported better test accuracy when a = 0.1 compared to 0.01, we exhibited significantly lower performance compared to 0.01.
Table The hyperparameters used in our experiments. Values before the are SGD, and values after are for SAM. In the case of S&P, the À value corresponds to the shrinkage parameter, while the a parameter controls the magnitude of the noise added to the weights. For L2 INIT, we did not perform experiments except for CIFAR-10.
DASH
S&P
1.2 INIT
ResNet-18
Momentum
Batch Size
A
Tiny-Imagenet
0.9
0.001
128
0.05
0.05
0.3
0.05
0.01
CIFAR-10
0.9
0.001
128
0.1
0.05/0.3
0.3
0.3
0.01
le-4
CIFAR-I00
0.001
128
0.05
0.1
0.01
SVHN
0.001
128
0.3
0.01
VGG16
Tiny- Imagentte
0.9
0.01
128
0.05
0.05
0.3
0.05
0.01
CIFAR-10
128
0.05/0.1
le.4
CIFAR-100
0.9
0.01
128
0.03
0.05
0.9/0.3
0.3
0.01
SVHN
0.9
0.01
128
0.01
0.1
0,9/0.3
0.3
0.01
MLP
Tiny-Imagenet
0.9
0.01
128
0.1
0.1
1.0
0.3/0.1
0.01
CIFAR-10
0.9
0.01
128
0.1
0.7/0.5
1.0
0.7/0.5
0.01
1c-4
CIFAR-100
0.9
0.01
128
0.1
0.1
1.0
0.3/0.1
0.01
SVIIN
0.9
0.01
128
0.1
0.3
1.0
0.3
0.01
C.4 Diseussions the Broader Applicability DASH
In this subsection, we explore how DASH performs in a variety of settings, including state-of-the-art (SoTA) configurations, larger datasets, and scenarios where previous data cannot be stored. We also examine how DASH compares to other methods when data is continuously introduced throughout training, as well as its effectiveness in non-stationary data distribution environments, such as Class Incremental Learning (CIL) setups.
C.4.1 Performance in State-of-the-Art Settings
In the state-of-the-an (SoTA) setting, we employed weight decay and standard data augmentation techniques, such as horizontal flipping and random cropping. We also used a learning rate scheduler that reduces the learning rate step-wise by a factor of 0.2 at 60, 120, and 200 epochs. By applying the learning rate scheduler, there no need to compare training time since training is completed at roughly the learning rate was set to 0.1. All other settings remain the same as mentioned above. We tested this setup CIFAR-I0 and CIFAR- 1-0 using the ResNet-18 architecture.
epoch across all experiments. The weight decay was set to 0.0005, and the initial
The results in Table 8 show that DASH performs similarly to or slightly worse than starting from random initialization. It appears that this is partly because all hyperparameters are tuned to maximize

---

the performance of cold-starting, to achieve the (close-to-)SoTi) test accuracy numbers. Due to the lack of computational resources, we were unable tune hyperparameters specifically for DASH.
Furthermore, we believe this aligns more closely with our theoretical anylsis in Theorem 3.6, as the hyperparameters are tuned to allow the model learn many features as possible, making it difficult for DASH to outperform cold-starting.
Moreover, we observe that S&P cannot be used in these SoTA settings. We believe this is due to the nature of S&P, which shrinks all weights, while the SoTA setting is likely designed to avoid learning unuseful features, unlike the previous setting. Consequently, it is plausible that retaining learned features is more important than forgetting them, making S&P unsuitable for SoTA settings. Although DASH performs slightly worse than cold-starting it is conceivable that it is better at retaining features compared to S&P and other warm-starting methods, resulting in better overall performance.
The gap between warm-starting and cold-starting has been significantly reduced, likely due to data augmentation techniques and the increase learning rate when new data introduced. Data augmentation techniques increase the amount of feature information, allowing warm-starting to learn features that vanilla training (without augmentation) cannot (Shen et al., 2022). Furthermore, as the learning rate is set to a higher value at the beginning of each new experiment, the model can forget previously memorized data points and escape spurious minima that were difficult to escape from, which is consistent with the findings of Berariu et al. (2021). Despite these improvements, gap still exists between warm staatina and oold-starting.
Table 8: Results of training CIFAR-IO. CIFAR-100 dataset trained ResNet-18 18 with SoTA set- tings. Bold values indicate the best performance, while underlined values denote the second-best performance. For the number of steps, we did not provide bold formatting since we used learning rate scheduling. Results are averaged over three random seeds, with standard deviations provided in parentheses.
Test
Number of Steps
AVG of Test Ace across all experiments
AVG of Number of Steps
ReaNer-18
last experiment
last experiment
across all experiments
CIFAR-10
SGD
SAM
SGD
SAM
SGD
SAM
SGD
SAM
Random Init
94.73 (0.74)
95.47 (0.17)
50439 (319)
47832 (184)
88.77 (0.04)
89.24
24826 (62)
23751
Warm Init
94.35 (0.31)
94.80 (0.20)
51221 (552)
47832 (184)
87.94 (0.26)
88.62 (0.57)
23759 (57)
21821 (174)
Warm ReM
94.56 (0:25)
95.00 (0.29)
51612 (319)
47962 (184)
88.20 (0.33)
88.56 (0.60)
23775 (16)
21786 (79)
S&P
94.15 (0)10)
94.73 (0.07)
51351 (184)
48353 (184)
88 38 (0.03)
89.27 (0.26)
25369 (49)
22805 (16)
(0.25)
95.06 (0.36)
51872 (487)
(184)
88.65 (0.24)
(0.40)
(75)
22233 (85)
CIFAR-100
Random Init
75.98 (0.01)
76.09 (0.12)
63081 (184)
56825 (184)
61.49 (0.09)
61.81 (0.08)
27536 (194)
25521
Warm Init
(0.09)
74.21 (0.26)
69598 (1462)
58128
58.40 (0.24)
58.44
28012 (114)
24562 (243)
Warm ReM
74.05 (0.13)
74.36 (0.13)
68425 (1689)
57216 (664)
58.32 (0.24)
58.33 (0.15)
27965 (190)
24534 (139)
S&P
72.96 (0.34)
73.71 (0.37)
64775 (664)
61387 (552)
57.33 (0. 10)
57.68 (0.06)
28809 (148)
26476 (212)
DASH
(0.07)
(0.09)
(1815)
59953 (1208)
(0.20)
(0.13)
(306)
25630 (100)
C.4.2 Scalability on Large Datasets
We validate the scalability of DASH for larger datasets such as ImageNet-Ik lk. However, conducting experiments for such datasets is challenging, as it would require repeating the training process 50 times until convergence an extremely time-consuming process. As an alternative, we trained ImageNet-1 1K on ResNet18 using setup similar Figure 2 in Section 3.3. Our setup involved pretraining on 50% of the data before fine-tuning on the complete dataset. We used shrinkage parameter À= = 0.3 for both DASH and S&P. Results shown in Figure 19 demonstrate that DASH achieves superior performance compared to all baseline methods- warm initialization, and S&P-both in terms of test accuracy and convergence speed. Notably, DASH achieves faster convergence and marginally better test accuracy than S&P, demonstrating its effectiveness even on challenging large-scale datasets.
cold initialization,
C.4.3 Effectiveness in Data- Data-discarding Setting
To explore how our algorithm can be applied to a broader range of scenarios further, we consider situations where storing all data is not feasible (e.g., due to memory constraints), rather than ac- cumulating and retaining data. Specifically, divided the CIFAR-10 10 dataset into 50 chunks and
25

---

50-
Train Accuracy (%)
60-
30
424
20
Test
Cold
10-
Warm
S&P
-150
50
Epochs
Epochs
Figure 19: Evaluation of DASH on the ImageNet-Ik Ik dataset using ResNet18. We pretrained for 150 epochs using 50% of the dataset (shown to the left of the dashed line) and then continued training for another 150 epochs with the full dataset for 'Warm', 'S&P' and 'DASH' methods. For cold initialization, we trained for 150 epochs on the full dataset starting from random initialization. DASH outperforms all baseline methods in terms of test accuracy and convergence speed.
experimented with ResNet- 18, running each experiment on single chunk before moving on to the next. All other settings remained consistent with the details provided in Section 5.1.
We configured our experiments to apply DASH at specific intervals (e.g., every 10, 15, or 20 experiments) instead of after each one. Since there no previous data available in this scenario, set a = 0. After the 40th experiment, when the model had sufficiently learned, we stopped applying the shrinking process. We also conducted similar experiments with the S&P method for comparison. It's important to note that this variant is feasible because both DASH and S&P focus on adjusting the model's 's initialization.
We tested DASH and S&P with intervals of 10, 15, and 20 epochs. For both methods, we explored shrinkage parameter (A) of 0.05, 0. and 0.3. We plotted the results using the best hyperparameters for each method in Figure 20. Notably, DASH's test accuracy consistently exceeded that of S&P across all hyperparameter configurations, as shown in Figure 21. These findings demonstrate that DASH outperforms both the warm-startinn baseline and the S&P method in terms of test accuracy. Based this evidence,
conclude that DASH is well-suited for data- dittaddicarding scenarios.
70
Test Accuracy (%)
Warm
Number of Steps
60
DASH
50
100
40
10
20
Number Experiments
Number of Experiments
Figure 20: Performance comparison between warm-starting, S&P, and DASH in scenario without access to previous data, using optimal hyperparameters for each method. For S&P, the shrinkage parameter 0.3 with shrinkage interval 20. For DASH, the shrinkage parameter À 0.3 with shrinkage interval 15. DASH significantly outperforms warm-starting, while S&P performs even worse than the warm-starting baseline in terms of test accuracy. Since there available, we observe sharp drops test accuracy during shrinkage events, but DASH quickly recovers while S&P struggles regain performance.
previous data
26

---

$6.25
$8.55
60 84
$1.22
$1.93
53:01
51
40
S&P Peroo
505
04.00
10-36
53:12
05.00
56 71
(2,40)
50.22
19-231
10:44
10.54
12:
10:241
40.
53
50
54
57.29 (5.09)
49.20
35
(3.23)
(0:35)
19:231
10:371
10:321
19)
10:
100
Act Experiment
Experiments
AVG Num Stapa A Exper
63
54.55
54
DASH Pertod
87.98
68.30
66- 96
Pelice
$4,82
$5,82
15.59
Perced
25
45
14
64-05
67.15
66.52
53.12
$5,68
$5.50 10:221
30
30
(6.70
(5:50)
(5:40
10:441
141
Figure 21: Heatmaps comparing S&P (top row) and DASH (bottom row) performance without access previous data. The x-axis shows results for different shrinkage parameters À (0.1, 0.3, 0.5), while the y-axis shows different shrinkage periods (10, 15, 20). Left heatmaps display final test accuracy, where DASH outperforms S&P across all hyperparameter configurations. Middle heatmaps show average test accuracy throughout all experiments, with DASH consistently maintaining superior performance across all configurations. Lastly, right heatmaps show average number of steps to converge across all experiments.
C.4.4 Performance with Continuously Added Data Setting
We designed experiment mimicking real-world scenarios where new data arrives continuously during training. Following Igl et al. (2020)'s approach, we sampled new data randomly each epoch for the first 500 epochs, combining it with the existing dataset, then continued training for another 500 epochs. This setting assumes scenario where data continuously arrives before the model has fully converged, as often does epochs using the entire dataset from the beginning. We used the CIFAR- 10 dataset with ResNet18 across five random seeds. We applied both DASH and S&P every 50 epochs through the first 500 epochs with shrinkage parameter À 0.3 for both methods, following similar setup Figure 20.
situations. Cold-started models were trained for 1000
Train Accuracy (%6)
Cold
Test Test Accuracy (%6
80
Warm
75
S&P
60
DASH
25
20
01
200
600
200
600
Number of Epochs
Number Epochs
Comparison of training accuracy (left) and test accuracy (right) for ResNet18 on CIFAR- 10 over 1000 epochs. During the first 500 epochs, new samples of equal size are added in i.i.d. manner, followed by 500 epochs of training on the complete dataset. Training accuracy is calculated using only the data available at each epoch during training.
Figure 22:
27

---

As shown in Figure 22, while there is a clear performance gap between warm-starting and cold-starting approaches. both DASH and S&P aim to address this difference. Despite using the same shrinkage value of 0.3, DASH successfully preserved learned features while S&P did not. This difference is reflected the training accuracy: DASH showed steady improvement as training progressed, while S&P struggled during the shrinking phases. As a result, DASH not only converged faster but also achieved higher test accuracy, while S&P lost the convergence advantage that warm-starting provides.
C.4.5 Experiments on Class Incremental Learning Setting
Experiments so far focused exclusively on stationary data distributions, To broaden our understanding, we explored DASH's performance non-stationary settings, particularly in Class Incremental Learning (CIL) an important area in continual learning where data distributions shift over time.
We began with experiments CIFAR-10 10, dividing the dataset into 10 chunks, each representing a distinct class. The model, based on ResNet- 18, was introduced to one chunk at the start of each experiment and trained without access previously encountered data. However, this setup proved challenging; even with warm-starting as baseline, the model simply overfitted to each new class, resulting in only 10% test accuracy across all experiments, equivalent to random guessing.
Given these limitations, we adjusted our experimental setup while preserving the non-stationary nature of the task. Using the same configurations with above, instead of completely discarding previous data, we accumulated data over time. Thus, each new chunk was combined with previously seen data. During evaluation, tested the model only classes it had encountered during training.
This revised approach yielded more promising results, as shown Figure 23, DASH surpasses other baselines in terms of test accuracy despite requiring longer convergence times. While these results demonstrate DASH's potential in certain non-stationary environments, we recognize that our modified setup simplifies true non-stationarity, such as in continual learning scenarios.
Test Accuracy (%6)
100
Number of Steps
8000
90
80
Cald
4000
Warm
70
2000
DASH
60
0
3
4
6
9
10
2
3
6
9
10
Number of Experiments
Number of Experiments
Figure 23: Performance on CIFAR-10 using ResNet-18 18, averaged over five random seeds. Left: Test accuracy evaluated on classes encountered so far. Right: Number of steps required for convergence.
C.5 Computation and Memory Overhead Comparison
In this subsection, we will compare the computational and memory overhead of each method. It important note that DASH is applied only once when new data is introduced. Since DASH calculates the gradient of the whole dataset just once, its memory complexity is proportional to the batch size and model size. When comparing experiments with the same number of epochs, we can think of DASH as adding approximately one extra epoch to the total training time. Similarly, the computational overhead of DASH is roughly equivalent to running one additional epoch.
We provide the experimental results of the computation and memory overhead for each method in Table The experiments were conducted under the same conditions described in Section 5.1 on CIFAR- 10, using ResNet-18 with the SGD optimizer. We provide aggregated results for FLOPS, CPU/CUDA memory usage, and training time on an NVIDIA A6000. These measurements were obtained using the torch.profile library, with values summed across all operations. We only profiled two out of the 50 experiments until convergence (99.9% training accuracy) because profiling adds significant overhead to the training process, and running it continuously slows down the entire training pipeline. A comparison of the total training times each method for all 50 experiments without profiling is provided in Section C.6. Table 9 indicates that S&P requires two more training epochs than DASH. Since DASH only adds approximately one extra epoch of overhead, the results show that DASH's total computational and memory overhead is about one epoch less than S&P's.
28

---

Table 9: Computational and memory requirements for each method, comparing two experiments on CIFAR- 10 using ResNet-18 18. We report the total number of epochs, total training time, and total computational cost (in TeraFL.OPS). Memory usage is measured in gigabytes, showing aggregated CPU and CUDA memory consumption.
Epochs
Training Time (s)
TFLOPS
CPU Memory (GB)
CUDA Memory (GB)
Cold Init
39
29.90
21.87
7.86
813
Warm Init
30
21.96
16.82
6.03
592
S&P
34
25.72
19.06
6.95
690
DASH
32
24.79
18.51
6.65
666
C.6
Comparison between NVIDIA A6000 and Intel Gaudi-v2
We conducted experiments on two different hardware platforms: Intel Gaudi-v2 HPUs and NVIDIA A6000 GPUs. In this section, we compare their implementation code and training times. Using CIFAR-100 and ResNet-18, 18, we measured the training times on both platforms while maintaining the same experimental settings as described in Section 5.1 with SGD optimizer. It is important to note that no hardware-specific optimizations were performed on either platform, and all hyperparameters were kept constant.
To ensure fair comparison, we ported our NVIDIA implementation to Intel Gaudi-v2, keeping the other settings consistent. The porting process was straightforward, requiring only about three extra lines of code, shown in the example below. The full Gaudi-v2 implementation can be found in our GitHub repository at https: //tpt/gittttb/t com/NAVER -NNEER-ITEL----E Co Co-Lab/ hes///audddeee
Import moduies
import babaana_fraewooree torch.cors as htcore additional processing code
for inputs, targets in data_loader:
outputs model (inputs) loss criterion (outputs, targets)
loss losswwackward()
Mark the step for the optimization process Aicorr.mark_stee
11
optimizer.step
Mark the step again to complete the optimization cycle
sicoore.arr_stte.
15
16
Detimizer..ero_grae
additional processing code
Listing Example of Gaudi-v2 code, with additional lines highlighted red.
In Table 10, we present the results of the training time comparison until convergence. The table shows that the Intel Gaudi-v2 is slightly faster than the NVIDIA A6000 across all initialization methods. For instance, the Intel Gaudi-v2 achieves a speedup of 1.16x over the NVIDIA A6000 with cold initialization.
Table 10: Training time comparison between two different hardware platforms under same settings. Intel Gaudi-v2 is slightly faster than NVIDIA A6000.
NVIDIA A6000
Intel Gaudi-v2
Cold Init
94m 28s
81m 26s
Warm Init
22m 5s
21m 16s
S&P
52m 23g
51m 11s
DASH
40m 8s
37m 13s
29

---

D Proof of Theorems
This section provides the proof for Theorems 3.4 and 3.6, stated in Section 3, respectively. Before presenting the main proof, we state some technical lemmas.
Lemma D.1. For any learned feature set A, B C S that satisfies A ç B and Ann 2 forrr any c E [C], then we have ACC(A) < (B.(
Proof of Lemma D.I. Since A & B, it is trivial that for any E [C] and A C Se, we have
A| 7) 2 1 (IAN nI <)
From the given condition, we can choose e* E [C] such that there exists distinct features
U1...., - U1 E A. and E (B.8.)) (Ans.)) Our choice of A (Vi....., Ur} C S satisfies
1[A* n < <> > (JA* nB] <)).
(2)
From (1). (2), and the definition of ACC(-), we have
ACC(A)= = 1 - 1
Σ
NA 1 (AA [ <))
C CE(C)ACC.
C-1
E
a 1 (AnB <T)
n
CE(C),ACS,
ACC(B).
For case of presentation, let us say "a model learns NA" a model cannot learn any features. For example, if model learns u1,--. M, S during steps of training process and feature learning process ends in (s + 1)-th step, let us say that we learn u1.-.. Ug NA, NA, .....
Using the notion above, we prove that our learning process uniquely determines the behavior within the same class regardless of the randomness of the training process, where the randomness may come from tie-bread fie-breaking that can happen in the choice of the most frequent non-learned feature.
Lemma D.2. Suppose we train two models with different randomness on Tı.5 for some j E N starting from a learned set L and without any memorized data. use u, and u' to denote features learned in 8-th step of training process by two models, respectively. The i-th learned feature within class e E [C] is denoted as We, for the first model and for the second model. Then, Ue = f all
C [C] and i E N.
Proof of Lemma D.2. Suppose there exists some class E [C] and E N such that Ueo o u/e, and choose one with the smallest Without loss of generality, we may assume v'es * NA. Then, we have
max WES, 1M 1)
1e(;; {ue,).... Mo.i-13
max
1e05 fu veg-1))
vES WES.A
h(ul, (u.. (')-)1))
jn
The first equality holds since (Me.i..... Mc {u', (u..,..... u from our choice of i and the
second equality holds since the second model learns un..- The last inequality holds since v'as * NA. Hence, Me, * NA and
1( (Ue.1..... (uc.1s..., (.,,11))) M i}
N(u;; (Mo.1..... (c.i-1))
From our Assumption 3.3 and since (Mc.1..... (e.--1) (u6,1:.., 1o we have uesi = Vo This is contradictory and we have our desired conclusion.
30

---

Lemma D.3. Suppose we train two models on Tı:j; and Tr for some jı > ja starting from learned set and without any memorized data. We use us and u to denote features learned in s-th step of the training process by two models trained on Tri and Tiaja respectively. The i-th learned feature within class E [C] denoted as ue,3 for the first model and u'o for the second model. Then, Vo,i = =ule or u for all e E [C] and i E
Proof of Lemma D.3. Suppose there exists some class E [C] and E N such that Ma,i # u and v/ # NA. Choose one with the smallest i. Since u/c,, * NA and from our choice and i, we have
max vES, 1e11
h(v; (Me,l...... Mc,i-1))
h(u; (. ( vES. (4..
hu I u/ 11(,-11)
7
2 j2n
Jin
Hence, Me,i * NA and
M( (ue,1y.... (e,
max
Me( (Me,1: r.. Mese-1 1))-
1)
From our Assumption 3.3 and since (Ue.1..... Ne.i-1) fele u in we have dest unas. This is contradictory and we have our desired conclusion.
With above Lemma D.1, D.2 and D.3, we have the following theorems.
Theorem 3.4. There exists nonempty G Ç such that we always obtain Clu -(() all J 2 2, the following inequalities hold:
G. For
ACC (charm) S ACC (Clold)
TDJ)
< To(d)
warm
Furthermore, ACC( < ACC(C cold) holds when J
8n where & max h(v; 9) > 0.
vES'
Proof of Theorem 3.4. By Lemma D.2 for the case j we immediately have our first conclusion by defining G as learned feature set from the first experiment. Furthermore, we have G S. and |9 S. 2 r for any class E [C] from our feature learning framework and Assumption 3.3.
We want to show that for any 2 2, C(J) = G. Since we never forget the learned feature in warm (J) training, it is clear that G C C Lwarm- We may assume that the existence of J such 2. Then, the first step of J" experiment, model learns some feature From training process, satisfies
that ((]) S c(T") and choose the smallest
n h-- 9) |T g(u; Ti. TN''", (K(J".0) .0)
2 7
and since J denotes the first experiment that can learn beyond 9, (J)" -1)
G and
warm 1.0)) Th-=~11: g(u; -(J"
= n h(u; 9) 2%
means that u must have been already learned the (J* 1)-th experiment and contradictory.
Thus, we have (J) have
(1)) L = (()) C (J) for all J 2 2 and combining with Lemma D.1, we
AC = ACC( warmm) ACC(C())) S AACO((())
To show that strict inequality for J > 2. it suffices to show that L Ç. C(J) we already showed that C S and |g on Sal 2 r- 1 for any class using cold-starting, by Lemma D.3, a model first learns features in G, say, in the first steps. In the + 1)-th step, cold-starting model learns a new feature since
coold for J >>>>>> 2 since [C]. J-th experiment
max |Twal g(:; Ti. A cold rESIg
max Jn h(v; (v: vESig
from the condition in the theorem statement. Hence, have our conclusion for the test accuracy.
31

---

For the train time, since the following holds, we conclude Tol)
<
PlJ)
cold when J 2 2:
nd (J++1)
Σ
T
S Jn <
cold
2
JE(
jE
Theorem 3.6. For any experiment J 2 2, the following holds:
ACC (Fool)) ACC (Fidea)
T(J) warm
TT()) ideal
<T(
Proof of Theorem 3.6. Recall that the ideal algorithm works by forgetting memorized data points while retaining previously learned features. In other words, at the initial step of the (j + 1) experiment, we have iddell
c(i+1,0) c(ح)
ideal Additionally, ( T1s+1 115+11110 ideal
h(v; ideall holds
for all S since MD+1.0 0.
We will show that Loold (J)
ideal holds for all J 2 by using induction.
When J 1, by applying Lemma D.2, it holds since Loold f Suppose C(J-1)
c(d-1)) for Lideal We have the following the first step of the J-th
some / 2 2 and we will prove that Loold experiment for all v E S:
c(J)
g(u; Th.J, Adell) h(u; (
For the cold-starting method in the J-th experiment, by Lemma D.2, let s be the step at which the model first finishes learning features in cod Then, at the (s + 1)-th step for all v S:
g(v; Tus,, cold h(u; ((dra))
Since we assumed hu;; (odla)) = lu;;; ((1;))))))) ) the cold-starting method starts to behave identically to the ideal method from the (s 1)-th time step onwards, by Lemma D.2, resulting in c(J) = L
|NJ1 ideal
[T:.] for since ( nSc 2 for some class [C] due Assumption 3.3.
Thus, the training time of the ideal method, Tr()) "ddeal* is follows:
Σ
T(J)
(())
Σ
nJJ(J 1)
ideal
cold
2
SEM
JE(J]
32

---

E Omitted Algorithms
In this section, we provide detailed training algorithms for our proposed learning framework. Al- gorithm outlines the standard training method within our learning framework. Subsequently, compare the Cold-starting, Strm-starting, and Ideal methods using the given abstract algorithm in the following algorithms.
Algorithm 2 Training Process
Require:
C: Set of learned features
M: Set of memorized data points
T: Training dataset
7: Threshold for learning features
r: Threshold for the number of learned features a data point needs be considered well- well-classified
1: function TRAININGPROCESSCC.MM, 7.7%
2: Initialize:
N + {(xr v) ET: [V(x) n <T (x, & M
0 while N * do
3:
4:
++1
5:
g(;: 1म ((ay)e (v E V(x)) for v E S
6:
Us c arg max g (u: break ties arbitrarily
WESIe
7:
if g(,;; 2 7/ [T] then
8:
[ LU {uz)
9:
N - [(x,0) E (())
10:
else
11:
M - ((.))
<T}
N
13:
end if
14:
end while
15:
return L, M
16:
end function
Algorithm 3 Cold-Starting until J-th Experiment
Require:
Ti::: Training dataset
7: Threshold for learning features
T: Threshold for the number of learned features a data point needs be considered well-classified
1: Initialize:
((0) Ø
M( +- 0
2: for in J do
3: (U)) M() + TrainingProcsss((C--)), M(-11), Tr.js T.
4: C()) <- 0
5: MU) 0
6: end for
7: return (G)
33

---

Algorithm 4 Warn-Starting until J-th Experiment
Require:
Ti::: Training dataset
7; Threshold for learning features
т: Threshold for the number of learned features a data point needs to be considered well-classified
1: Initialize:
[(9) 0
2: for j M <-0 0
3:
(()) (M TrainingProcess(((V-)) MU-1), 7,7)
4: end for
5: return (()
Algorithm 5 Ideal-Starting until J-th Experiment
Require:
Ti::: Training dataset
7: Threshold for learning features
T: Threshold for the number of learned features a data point needs be considered well-classified
1: Initialize:
((9) <- 0
MM f- 0
2: for in 1: : do
3: Cl)) M66 +- Trainngrrocss((CU-))), M(--1), Tr.j. 7.
4: MUI < 0
5: end for
6: return L()
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the main claims and contributions of the paper, including the development of learning framework to study loss of plasticity in stationary data distributions, identifying noise memorization as the primary cause, and proposing the DASH method to mitigate the issue. The claims match the theoretical and empirical results presented.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper includes "Discussion and Conclusion" section that reflects on the scope and limitations of the work. For example, it notes that the theoretical analysis treats the learning process as discrete and does not assume any specific hypothesis class in order to generalize the findings.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
35

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
The main theoretical results (Theorems3 .4 and 3.6) clearly state assump- tions and provide proof sketches in the main text with references to complete proofs in the Appendix D. The proofs rely reasonable assumptions and appear to be correct stated in
Justification:
Section 3.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The experimental setup and training details are described in sufficient detail in Section 5.1 to reproduce the main results, including dataset splits, model architectures, optimizers, and hyperparameters. Appendix C.3 provides additional specifics on the hyper- parameter settings used.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
36

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provided open access to the code. Please refer to footnotes and 2.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
htaiii//uiides/Coeeubmmii. comiiiissioliico for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. wwtppplcccgggpciiiccceccooocccce gttg//uues///e/// httpbi//gguddee/ddeuubbiiiii
/tt////oooiissiiooooc swiiiissssopiiiyyyoo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- sparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5.1 describes the key experimental settings including the dataset setup, model architectures, optimizers, and hyperparameters. Appendix C.3 provides additional details the specific hyperparameter values used for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
The experimental results include standard deviations over multiple random seeds to quantify the statistical significance and stability of the findings. The number of random seeds used is reported throughout the paper.
Justification:
37

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experimental compute resources are reported in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This paper does not violate the NeurIPS Code of Ethics in terms of data privacy, potential for misuse, or other ethical issues. The work is largely theoretical and the experiments common open datasets.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
38

---

Answer: [Yes]
Justification: Section discusses the positive impact of the work in enabling more efficient and sustainable Al that adapt to increasing real-world data. It notes this can help prevent wasted time, energy and environmental impact. Potential negative impacts are not explicitly discussed the work is largely theoretical.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper focuses on theoretical analysis and experiments common open datasets. does not introduce any models or datasets with high risk of misuse that would require safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: This paper does not rely on or adapt any existing code or model assets. The datasets used are common open datasets and cited appropriately.
39

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce any new datasets, code or models. It focuses on theoretical analysis and experiments using existing datasets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This research does not involve any human subjects crowdsourcing. It is based on theoretical analysis and experiments on open datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
40

---

Answer: [NA]
This research does not involve any human subjects, so IRB approval is not applicable. It focuses theoretical work and experiments open datasets.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
41


Reviews:
[{"review_id": "bYeUdkI6dH", "reviewer": "Reviewer_LBum", "summary": "This paper investigates the reasons why warm-starting a neural network by pre-training it on a subset of the full training set leads to suboptimal performance when compared to training it on the full dataset from scratch. In particular, it proposes an abstract combinatorial model of feature learning that according to the authors captures the essence of this phenomenon. In this model, warm-starting leads to overfitting on feature noise that hinders learning new features when fine-tuning on the full distribution when compared to cold-starting. The authors then show that there exist an idealized learning algorithm in this abstract setting that can perform as well as cold-starting by forgeting the memorized noise prior to fine-tuning. Based on this theoretical intuition, they then propose Direction-Aware SHrinking (DASH) a technique that shrinks the learning step per-neuron based on its alignment with previous warm-starting epochs. DASH seems to outperform other techniques in a synthetic warm-starting benchmark.", "questions": "I would appreciate a deeper description of how DASH works:\n- Is the running average of gradients computed at every step or every epoch? \n- Is the shrinking performed after every step, every epoch, every experiment?\n- Can DASH be combined with any off-the-shelf deep learning optimizer? If so, how?\n- What is the memory complexity of DASH?\n\nIf these questions are addressed and the explanations are clear and made part of the final manuscript I would be open to increase my score.", "limitations": "The authors briefly mention mention some of the limitations of their work (such as the unexplained outperformance of DASH over cold-starting) or specificity of their results to the stationary setting.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Intuitive and insightful theoretical model**: Personally, I find the discrete abstract model of learning of this paper to be very insightful. Theorems 3.4 and Theorems 4.1 are good examples of theoretical propositions that can help cement our empirical observations in deep learning. This model may not provide a rigorous path towards mathematically \"proving\" the observed warm-starting phenomenon, but it is useful to reason about it.\n2. **Convincing explanations of warm-starting**: The provided explanations for the observed behavior of deep models under warm starting and the evidence to support them are solid and convincing.\n3. **Good performance of DASH**: DASH seems to perform strongly against prior work on the studied warm-starting benchmarks.", "weaknesses": "1. **Unclear details of DASH**: The description of DASH in section 4.5 is not clear and the algorithm listing does not fully explain the algorithm (see questions). This makes it hard to assess the complexity of implementing DASH and its practical relevance.\n2. **Limited evaluation of DASH in online learning**: The authors argue that one of the main motivations of studying the decrease of performance under warm-starting is the fact that many networks are trained in an online setting with new data being collected sequentially. However, the only evaluation of DASH is a synthetic warm-starting benchmark that only tests a narrow and unrealistic setting. I am not against this particular benchmark, but I believe the paper would have benefited from further investigations of DASH in a more realistic online setting (including comparing it against strong baselines in that space).\n3. **Reasons for improved performance of DASH over cold-starting are not explained**: As the authors highlight, DASH seems to outperform cold-starting in their benchmarks. This phenomenon, however, cannot be explained by their theoretical model and thus it is a weakness of the model itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our appreciation to the reviewer for your valuable and constructive comments.  In the following, we address the points raised by the reviewer.\n\n## **W1, Questions. Detailed description of DASH**\n\nWe acknowledge that our description of DASH may have caused some confusion regarding the algorithm. DASH is a shrinking method that can be applied whenever new data is provided. As a result, the running average gradient computation (Q1) and shrinking (Q2) are performed at the beginning of the “experiment” (i.e., when a new data chunk is added). We realize that this description was omitted from Section 4.2 and was only mentioned in the Experimental Details (Section 5.1, line 359-360).  \n\nQ3: Since DASH is an initialization scheme performed independently of the optimizer, it can be combined with any optimizer, such as SGD, Adam, or SAM. Indeed, we have provided results using two different optimizers: SGD and SAM.\n\nQ4: DASH calculates the weighted average gradient in training data sequentially, resulting in a memory complexity  proportional to the batch size and model size, as only the gradient needs to be stored. In a scenario where gradients are computed one at a time, DASH's memory complexity would be equivalent to the model size. Please refer to our global response for more details, where we have included experimental results about both the memory requirements and computational overhead of DASH. \n\nWe plan to include these detailed explanations in our next revised version. \n\n## **W2. Evaluation in online learning**\n\nThank you for suggesting an evaluation of DASH in an online learning setting. We believe this is a great suggestion, as it addresses situations where storing all data is not feasible (e.g., due to memory constraints) and helps us explore how our algorithm can be applied to a wider range of scenarios. However, we'd like to clarify that our current setting is indeed realistic. It reflects real-world scenarios where new data continuously accumulates, such as in financial markets or social media platforms. This type of data accumulation is also studied in the work of Ash & Adams (2020).\n\nWhile we are not entirely sure of the specific online learning setting you have in mind, we have conducted experiments with a setup where, instead of accumulating and storing data, the model learns from new data as it becomes available. For example, we divided CIFAR-10 into 50 chunks and performed experiments with ResNet-18, where each experiment was conducted on a single chunk of data before moving to the next. All other settings remained consistent with experiment details provided in Section 5.1.\n\nTo address the issue of catastrophic forgetting, which is common in online learning, we configured our experiments to apply DASH at specific intervals (e.g., every 10, 15, or 20 experiments), rather than after each experiment. Since there is no previous data available, we set $\\alpha = 1$. After the 40th experiment, when the model had sufficiently learned, we stopped applying the shrinking process. We conducted similar experiments with the S&P method for comparison. It's worth noting that this variant is feasible because both DASH and S&P are algorithms that focus on adjusting the initialization of the model.\n\nWe tested DASH and S&P with intervals of 10, 15, and 20 epochs. For both methods, we explored shrinkage parameter ($\\lambda$) of 0.05, 0.1, and 0.3. Notably, DASH's test accuracy consistently surpassed that of S&P across all hyperparameter configurations. In Figure 8 of our PDF, we plotted the results using the best hyperparameters for each method. These results demonstrate that DASH outperforms both the warm-starting baseline and the S&P method in terms of test accuracy. Based on these findings, we can conclude that DASH is suitable for online learning scenarios. If you could provide more details about the specific online learning setting you mentioned, as well as any additional baselines, we'd be happy to conduct further experiments to explore this area.\n\n## **W3. Reasons for improved performance of DASH over cold-starting**\n\nThe reason our theoretical framework does not explain DASH's superior performance in cold-start scenarios may be due to the binary nature (learn/not learn, memorize/not memorize) of our framework. However, we can provide an intuitive explanation based on our approach of retaining features and forgetting noise. When applying DASH in practical scenarios, even features previously considered \"learned\" can be further \"improved\" (i.e., model learns the feature with a greater strength) after noise is forgotten through shrinking, due to the continuous nature of real learning scenarios. We believe further improvement of feature has contributed to the performance improvement.\n\nThanks for your time and consideration.\n\nBest regards,\n\nAuthors\n\n---\nReference\n- Jordan Ash and Ryan P Adams. On warm-starting neural network training. In NeurIPS 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JSnAB5IcU6", "reviewer": "Reviewer_t732", "summary": "Warm-starting neural networks may lead to poor generalization performance or loss of plasticity, likely due to overfitting. This paper presents a framework that hypothesizes that noise memorization is the primary cause of loss of generalization in warm-starting settings. The authors then present an algorithm motivated by their framework that improves the generalization and plasticity of the neural networks. The algorithm works on a per-neuron level and measures the cosine similarity between the negative gradient of the input weight to each neuron and its weights. This cosine similarity is used as a utility measure to allow for more shrinking for the input weights with low cosine similarity. The authors showed the effectiveness of their method on a wide range of problems and compared it against other baselines demonstrating superior performance.", "questions": "- In Figure 4, the authors presented an arrow with a feature direction. What does that mean exactly in a rigorous way?\n- What is the definition of a feature? The word is used in the paper many times without a proper definition, leading to further confusion. In neural networks, the learner always improves its set of features, so they are always changing.", "limitations": "No. I think the main limitation of the method is that the cosine-similarity metric seems to be heuristic-based and not motivated by the introduced framework. It is unclear if such a metric is ideal or if it reflects what is suggested by the theoretical framework.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important problem: maintaining plasticity and generalizing neural networks. The proposed algorithm is novel and effective, and the experimental results seem comprehensive.", "weaknesses": "- The main problem of this work is that the link between the framework and the algorithm is not clear. For example, it’s unclear why different parts of the algorithm are designed this way, especially the cosine similarity part. Why is the momentum direction assumed to be the direction of features? \n- The evaluation may be unfair and statistically insignificant. In multiple experiments (e.g., figure 1),  the results are generated based on three seeds. This is a very low number to have statistically significant results. Additionally, the reported methods use the same hyperparameter across multiple methods, which may be unfair to the baselines since they are not well-tuned compared to DASH. The results should be based on the best hyperparameter of each method. This might explain why the authors found approaches such as L2 Init, resetting, layer normalization, SAM, or reviving dead neurons to be less effective. In contrast, the only effective one was S&P, which uses a hyperparameter (shrinking) similar to the DASH method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our appreciation to the reviewer for your valuable and constructive comments.  In the following, we address the points raised by the reviewer.\n\n## **W1, Limitation. Clarification on our framework and link between the framework and the algorithm**\n\nWe apologize for the unclear explanation regarding the framework and connection to our algorithm, DASH. We have clarified this in the general response. In it, we explain the role of cosine similarity and how shrinking with cosine similarity effectively captures feature retaining and noise forgetting. We hope this addresses any concerns.\n\n## **W2. Additional results with different seeds**\n\nWe understand your concerns about the statistical significance of our experimental results. While we aim to include as many random seeds as possible, our incremental learning framework is computationally intensive, which initially limited us to three random seeds. To address this issue, we're actively working on increasing the number of random seeds to enhance statistical significance.\nIn response to your concerns, we can now provide results from two additional seeds (bringing the total to five) for experiments trained with SGD. As shown in Table 1 attached PDF in the global response, these new results demonstrate trends similar to our previously reported values. We hope this improvement addresses part of the concerns you raised.\n\nUnfortunately, we cannot currently provide results for Tiny-ImageNet on VGG16 with SGD, as these experiments take approximately a week to train. However, we'd like to draw your attention to the results table in Appendix A.2, which shows our approach with Tiny-ImageNet on VGG outperforming other baselines. We hope this helps alleviate some of your concerns. In our revised version, we plan to include results with five random seeds for all experiments to ensure comprehensive statistical significance across our findings.\n\n## **W2. Additional results with different hyperparameters**\n\nIn our experiments, we used the same hyperparameters across multiple datasets and methods. It's important to note that these hyperparameters were not specifically tuned for DASH; rather, this approach was chosen for fair comparison, following the methodology of Ash & Adams 2020.\nIdeally, as you pointed out, we would have preferred to use the best hyperparameters for each method. However, due to limited computational resources, we couldn't exhaustively search for optimal values of base hyperparameters (such as learning rate and batch size) or method-specific parameters, since some runs take as long as a week. Given these constraints, we concluded that the best way to compare baselines was to maintain consistent hyperparameters across all methods.\n\nIn our original vanilla setting described in Section 5.1, the only hyperparameters available for tuning were learning rate and batch size. To address your concerns, we conducted additional experiments with CIFAR-10 on ResNet18 where we varied the learning rate while keeping the batch size fixed at 128.\nYou can see these results illustrated in Figure 5, 6 and Table 2. Our findings confirm what we previously mentioned: there is little to no effect when using L2 Init, resetting, layer normalization, SAM, or reviving dead neurons. L2 Init and resetting show worse generalization performance compared to warm-started neural networks. \n\n## **Questions**\n\nWe have provided high-level intuition and explanations with examples for our framework in our global response, and we encourage you to refer to it. We hope this will clarify any misunderstandings.\n\n**Q1. What does the feature direction mean exactly in a rigorous way?**\n\nAs detailed in our global response, there are two ways to minimize training loss: learning features and memorizing noise, with only learning features being beneficial for unseen test data. Therefore, the negative gradient can be decomposed into components of feature and noise. A feature direction represents *the direction of updates that is effective for both training and test data*.\n\n**Q2. What is the definition of feature?**\n\nWe emphasize that the notion of features in our work is totally different from what is commonly referred to as 'features' in the context of neural network outputs, specifically the last hidden layer outputs. As discussed in our global response, we consider features to be the information contained in the input that is relevant to the label and therefore useful for generalizing to unseen data. \n\nThanks for your time and consideration.\n\nBest regards,\n\nAuthors\n\n---\nReference\n\n- Jordan Ash and Ryan P Adams. On warm-starting neural network training. In NeurIPS 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vmoOJzzgQ3", "reviewer": "Reviewer_hPM9", "summary": "The paper examines warm-start and the loss of plasticity, identifying that noise memorization from warm-start impairs generalization in stationary data distributions. It proposes a method called DASH, which selectively forgets previously memorized noise while preserving learned features.", "questions": "- Does the apparent non-linearity in the correlation between initial gradient norms and training steps in Figure 10 change how Figure 1(b) should be interpreted?\n- In Figure 2, the test accuracy drops with longer pre-training when the model is pre-trained on 50% of the training data and then fine-tuned on the full dataset. Is this phenomenon universal or does it only occur with the CIFAR dataset? Could the authors provide similar results for high-dimensional datasets, such as ImageNet-1K (224x224)? If the network learns the same set of semantic categories during the pre-training and fine-tuning stages, the fine-tuning stage only differs in terms of the number of samples. Is this phenomenon related to the size of the training data? How much do pre-training and fine-tuning datasets differ in terms of distribution shifts? Is the pre-training data class balanced? How does DASH perform in a similar setting?\n- Does DASH need access to the entire dataset (all chunks), including current and previous datasets/chunks, for computing gradients and shrinking weights during every training step?\n- In *Line 116*, the set `S_y` is undefined, and may be corrected to `S_c`.\n- In *Line 134*, `Tau < K` may be corrected to `Tau <= K`, where the number of learned features can also equal the set of learnable class-dependent features, K.", "limitations": "The paper provides little to no detail on the limitations of this work. While it mentions the assumptions of a discrete learning process and the lack of consideration for a hypothesis class, it does not address whether DASH can be extended to larger problem settings. If such limitations exist, they should be explicitly stated by the authors.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Feature Learning Framework**: The paper introduces a feature learning framework to investigate the warm-start issue. It thoroughly explores the properties of warm-start and cold-start training, the role of noise memorization in the loss of plasticity, and the selective retention (forgetting) of features (noise). These core findings are well-supported through theoretical grounding and experimentation.\n- **Performance:** In vanilla settings, DASH significantly outperforms other methods on several small datasets in terms of accuracy and training steps. And, in the SoTA setting, DASH rivals other methods.", "weaknesses": "- **Compute Overhead**: Gradient-based approaches typically incur significant compute overhead. While DASH is claimed to achieve faster convergence, it remains unclear if its shrinking process slows down the overall training. Quantitative comparisons in terms of FLOPs and training time would be insightful. Furthermore, DASH has not been compared with baselines regarding compute and memory overhead. The training convergence based on the gradient norm does not consider the compute and memory overhead associated with DASH’s gradient calculations.\n- **Assumption: Gradient norm as a proxy for the number of training steps**: The assumption that the number of training steps can be reliably approximated by the initial gradient norm is insufficiently supported by the data presented in Figure 10 of the paper. The correlation between training steps and the initial gradient norm does not appear to be linear, particularly for the Warm start. \n- **In-depth Analysis**: The experiments do not demonstrate a direct link between learned/memorized data and the cosine similarity of gradients and weights. Conducting ablation studies might provide further insights into this relationship. Additionally, the proposed method appears to be a variant of the shrink and perturb approach.\n- **Evaluation and Scalability**: Evaluation is performed exclusively on toy datasets, leaving DASH’s scalability to larger and more challenging datasets, such as ImageNet-1K, unknown.\n- **Generality**: In SOTA settings, as shown in Table 7, DASH does not consistently outperform the compared methods, particularly warm initialization. This raises questions about the wider applicability and robustness of DASH. Its efficacy seems to diminish when using augmentations and weight decay.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for providing valuable and constructive feedback on our work. Below, we address the questions and concerns raised by the reviewer.\n\n## **W1 & W3**\nCheck out our global response where we address these issues. We hope our explanation and ablation study resolve all concerns you may have.\n\n## **W2, Q1. Proxy for the number of training steps**\nWe believe there may be some confusion, let us clarify this point. For all our real-world experiments including Figure 1(b), we count the actual number of training steps until near-convergence (99.9% training accuracy), not the initial gradient norm. In contrast, we use the number of non-zero gradient data as a proxy of the training time ONLY in our abstract learning framework, for the sole purpose of theoretical analysis.\n\nThe reason for adopting a proxy of the training time is to capture the concept of “training time” within our abstract learning framework in which features/noise are learned/memorized in an “on/off” manner. Due to the binary (zero vs non-zero gradient) nature of our framework, we think the number of non-zero gradient data can serve as a reasonable proxy of the norm of gradient in practice. In addition, since the initial gradient norm is positively correlated with the number of training steps (in practice), we believe that the number of non-zero data point in our abstract framework can role as proxy for the number of training steps in practice. While the relationship between gradient norm and training time is not strictly linear, we think the number of non-zero gradient data to be the best possible measure of training time in our framework.\n\n## **W4, Q2. Evaluation on ImageNet-1K**\nWe appreciate your comments on scalability for larger datasets like ImageNet-1K. However, including experimental results for such datasets are challenging, as it would require repeating the training process 50 times until convergence—an extremely time-consuming process.\nAs an alternative, we trained ImageNet-1K on ResNet18 using a setup similar to Figure 2 in Section 3.2. We used 50% of the data for pretraining and the full dataset for fine-tuning. Results in Figure 3 (attached PDF), show DASH outperforming both cold and warm starts in test accuracy and convergence speed, demonstrating its effectiveness on challenging datasets. Due to time constraints, we couldn't include results from training ImageNet-1K divided into 50 chunks in the paper. However, we plan to add results in the future using more than two chunks.\n\nWe also investigate the effect of pretrain epoch on warm-starting in ImageNet-1K classification with ResNet18. We conducted experiments using different pretrain epochs: 30, 50, 70, and 150. Figure 4 (attached PDF) shows a decline trend in accuracy for warm-started models as pretrain epoch increases.\nInterestingly, we noted a similar phenomenon in Figure 2 in Section 3.2, where warm-starting does not negatively impact test accuracy when training resumes from earlier pretrained epochs, such as 30 in this case. This observation aligns with our theoretical framework, which suggests that neural networks tend to learn meaningful features first and then begin to memorize noise later. We believe this phenomenon is persistent across different datasets, model architectures, and optimizers.\n\n## **W5. Inconsistency in outperformance of DASH compared to warm-start**\nWe acknowledge that there is some case that DASH underperform warm-start in the test accuracy at last experiment. However, It is important to note that the performance differences among all methods are not large, with most falling within standard deviation ranges. Also, we emphasize that performance at every experiment is crucial in practice. Therefore, we suggest examining not only the test accuracy at the final experiment but also the average test accuracy across all experiments. We ask the reviewer to revisit Table 7 in Appendix A.3., where we show that DASH consistently outperforms warm-starting in terms of average test accuracy, with improvements ranging from as little as 0.45% to as much as nearly 3%.\n\n## **Minor Questions**\n- Q2. You're right that this is a matter of training data size. If a network isn't provided with sufficient data to learn enough features during the initial experiment, it may struggle to learn feature even when new data is introduced at following experiments. This concept is central to our Theorem 3.4. In scenarios where data is sampled from a stationary and class-balanced distribution (like real-world image data), there is less distribution shift due to class imbalance, as the samples are drawn iid.\n- Q3. DASH is applied at the start of each experiment, not at every training step. When DASH's hyperparameter $\\alpha$ is set to 1, only new data is needed for shrinking. Figures 7-8 in Appendix A.4 show DASH behaves similarly across different $\\alpha$ values, including when $\\alpha=1$. This indicates that DASH can be effective even when only the new data chunk is accessible.\nFurthermore, DASH can be utilized in online learning settings, where only newly introduced data is available during training. We evaluate DASH in such a setting using CIFAR-10 divided into 50 chunks and trained on ResNet18. Each experiment was conducted on a single chunk of data before moving to the next, with all other settings consistent with those detailed in Section 5.1. The results, shown in Figure 8 (attached PDF), show that periodically applying DASH outperforms baselines including S&P in terms of test accuracy. This confirms DASH's effectiveness with limited data access.\n- Q4. Our use of $S_y$ is intended to represent the set of features associated with the true label $y$.\n- Q5. We consider the case where $\\tau$ is strictly smaller than the total number of features within a class. If $\\tau = K$, then only data points containing all possible features would be correctly classified during testing.\n\nThanks for your time and consideration.\n\nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LME6Qsw3za", "reviewer": "Reviewer_rFKW", "summary": "This paper examines a setting in which a model is learned with an expanding dataset, such that new data is introduced after the model hits a training accuracy threshold. In particular, the new data comes from the same distribution as the existing data (or sometimes has the exact same statistics), making this a stationary learning problem (unlike the non-stationary problems commonly explored in plasticity loss settings). After demonstrating that plasticity loss occurs in this setting, the paper proposes a new method, DASH, to tackle the problem. DASH shrinks weights based on the cosine similarity between the weight vector and the negative gradient of the training loss. There are a series of experiments which generally show DASH outperforming other plasticity loss prevention approaches, with more in the appendix.", "questions": "- Considering a different learning setting, where data is continuously added rather than in predictable batches, would DASH still be able to perform (given at the moment, it is applied whenever new data is provided)? Similarly, how is its performance when we are not adding data from a stationary distribution, given this is possibly more realistic?\n- How do the authors think DASH could work in the more generic learning problem (i.e., where we are not learning features based on their frequency)?\n- What is actually meant by the frequency of features - can you offer an example please? I find it hard to determine exactly what is meant by the frequency of these discrete features?\n\n- In table 8, the authors write CIF1R rather than CIFAR.\n- What is meant in line 325: 'This allows weights to easily change its direction.'?", "limitations": "- In the appendix, there are examples where DASH does not outperform other methods, or is within the error margin. However, there is clearly a strong signal that DASH on average provides benefit to the training in all but the SOTA setting. The authors are upfront about this.\n- I believe the authors are generally upfront about the limitations of their work, though I feel perhaps highlighting that they are working in a slightly artificial setting, and demonstrating how DASH could be implemented to a typical machine learning pipeline, would be useful.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Based on my knowledge and research, I think this is a novel approach to preventing plasticity loss. It is quite original in its idea, which the authors have tried to ground in theory.\n- This is a relatively underexplored area (plasticity loss under stationary settings), and thus is useful for understanding plasticity loss further.\n- DASH seems to offer genuine performance improvements over most methods, though it is worth noting that this is not always the case (i.e., in the appendix more results are included which are either significantly more marginal, or sometimes favouring other methods). In particular, the focus that test accuracy is not only improved, but we carry over some of the benefits of warm starting (i.e., quicker convergence) is notable.\n- The figures/graphs in the paper were generally good and useful for understanding what was being discussed.", "weaknesses": "Before noting weaknesses of the paper, which I generally feel come down to clarity issues. I am open to increasing my score in light of discussion based on the author's response to my highlighted weaknesses and questions.\n\n- I found a lot of the theory quite confusing, and couldn't follow section 3 ('Comparison Between Warm-Start and Cold-Start'). Similarly, I found some explanations about the method, and the training process quite confusing. Perhaps demonstrating with examples could be useful, but it is currently quite hard to parse exactly what the problem setting is.\n- This work has not been studied in a particularly standard learning process. Instead, the focus is on a setting where features are learned based on their frequency of appearance in data, rather than their signal strength. This is quite an artificial setting, and it is not exactly clear to me that DASH scales to more conventional learning frameworks.\n- Appendix A.3 feels slightly burrowed away, but considers the SOTA setting for CIFAR-10 and CIFAR-100, in which DASH underperforms random initialisation. I think this is quite a crucial result, and arguably outweights the results from Table 1 in which a more naive approach is taken. While it is good that the authors include these results in the appendix, it would be good to try to include this in a more upfront way (perhaps including discussion about how DASH is helpful in a naive setting and helps reduce the barrier to entry with less studied datasets?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for providing valuable and constructive feedback on our work. We also thank you for bringing to our attention typos in Table 8. Below, we address the questions and concerns raised by the reviewer.\n## **W1, Q3. Clarification on our Framework and DASH**\nWe acknowledge that there may be some confusion regarding our theoretical framework. To clarify, we provide a high-level intuition and an example to the global response to better explain our framework. We hope this will help in understanding our approach. In our next version, we will include these explanations.\n\nRegarding Q3, in our framework, the frequency of a feature in dataset is estimated by the number of data that contain the feature, divided by the total number of data. For example, in Figure 1 of the attached pdf, the feature represented by the violet has a frequency of 5/6. Also, we emphasize that the concept of learning by the frequency of features in our framework is intended to capture the characteristics of learning in real-world in abstract manner.\n\nWe hope this clarifies any confusion.\n## **W2, Q2, L2. DASH is practical method**\nWe believe that there may be some confusion. DASH is a practical algorithm that has been evaluated on real-world datasets such as CIFAR-10, CIFAR-100, and Tiny ImageNet. We suspect that we didn’t deliver this well enough and made the reviewer think that DASH operates only within the feature learning framework introduced in Section 2. However, that framework is intended to mathematically model the training procedure and hence analyze the test accuracy and training time trade-off between cold-start and warm-start. We propose a method which is ideal in our framework and show that the method mitigates the trade-off. Motivated by this ideal method, we propose DASH as a practical method which intuitively captures the idea of the idealized method.\n## **Q5. What is meant 'This allows weights to easily change its direction.'?**\nIt means that the mechanism enables weights to change their direction with even small updates. Here’s why this is important:\n\nIn unseen data, features are more likely to reappear, while noise from old data is less likely to be present. As a result, learning these features effectively enhances generalization. In other words, it is beneficial for neurons to align with class-relevant feature directions. By shrinking weights that are less aligned with these features, we make it easier for the model to adjust and align with the feature directions and enhance generalization performance.\n## **W3 & L1 Underperformance DASH in SOTA setting**\nWe agree that there was a lack of discussion in the main text regarding the underperformance of DASH compared to cold-starting in the SOTA setting, which we had only addressed in the appendix. We plan to include this in the main text in the next revision. As described in Appendix A.3, we believe that this underperformance is partially due to using hyperparameter optimized for cold-starting. Even though we did not tune the hyperparameters for DASH, it remains comparable to cold-start, which in fact better aligns with our theory (Theorem 4.1.). We believe that specifically tuning the hyperparameters for DASH might further enhance its performance and potentially surpass the cold-start method in the SOTA setting. The fact that our method performs better with naive training suggests it might be particularly effective for understudied datasets where SOTA settings are not well-established, as you pointed out.\n## **Q1. The setting in which data is continuously added**\nWhile it is not entirely clear to us what the reviewer meant by “continuously added data”, we conducted an additional experiment that might address your concern. In this experiment, we added new data at each epoch for the first 500 epochs, then continued training for an additional 500 epochs (similar to Igl et al. (2021)). Cold-started models are trained for 1000 epochs using the entire dataset from the beginning. We used the CIFAR-10 on ResNet-18 with five random seeds. \n\nAs shown in the result plot in Figure 7 in attached PDF, there is a clear performance gap between warm-starting and cold-starting methods. We applied DASH and S&P every 50 epochs periodically until the 500th epoch, as you mentioned it cannot be done every time new data is introduced. Interestingly, DASH preserved learned features while S&P did not, even when we used the same $\\lambda$ value of 0.3. As a result, DASH showed an increasing trend in train accuracy as the number of epochs increased, while S&P failed to learn anything during shrinking. Consequently, DASH converged faster than S&P while achieving better test accuracy.\n## **Q1.  The setting in which data is added from non-stationary distribution**\nDASH is designed for stationary data distributions, assuming features of the new data remain consistent with previous data. Because of this, the negative gradient has a high cosine similarity with the neurons that already learned features, allowing learned features to be retained. However, in non-stationary cases, the features may change continuously as new data is introduced. This means that it might be challenging to retain already learned features in such scenarios. The effectiveness of DASH may be limited in these non-stationary environments. \n\nAdapting DASH for non-stationary scenarios (e.g., reinforcement learning) might require incorporating techniques similar to S&P, where perturbations are introduced to reactivate dead neurons. This approach could be crucial for maintaining network adaptability in changing data distributions. However, addressing these challenges in non-stationary environments is beyond the scope of our current work. We believe that this is a very important future direction.\n\nThanks for your time and consideration.\n\nBest regards,\n\nAuthors\n\n---\nReference\n- Igl et al. Transient Non-Stationarity and Generalization in Deep Reinforcement Learning. In ICLR 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We express our gratitude for your time and valuable comments. We would like to address the concerns and confusion raised by multiple reviewers.

## **Main intuition on our theoretical framework**
We would like to provide a clearer explanation of how our theoretical framework reflects the intuitive process of learning from image data. Figure 1 in the attached file illustrates the main ideas, which we believe will help readers better understand our approach.

Our framework is designed to capture characteristics of image data, where the input contains information relevant to both the image labels (which we refer to as “feature”, e.g., a cow’s ears, eyes, tail, and mouth in Figure 1) and irrelevant to the labels (which we refer to as “noise” e.g., sky, grass). Our framework is based on the insights from Shen et al. (2022) and incorporates these into a discrete learning framework.

Our training process is based on the belief that features that appear more frequently in train data are easier to learn, as gradients often align more strongly with these features. Thus, our framework is designed to sequentially learn the most frequent features (e.g., a cow’s ears) first.

As the model sequentially learns these features (e.g. a cow’s ears, eyes, mouth, & tail), it accumulates sufficient information to correctly classify the image in Figure 1 as a cow. Once the model correctly classifies data using these features (which we call "well-classified"), the impact of these data points on the learning decreases, as loss gradients become smaller as the predictions become more confident. To capture this characteristic, our framework evaluates the frequency of features solely on data points that are not well-classified.

As training proceeds, the algorithm may arrive at a point where all remaining features in non-well-classified train data points are not frequent enough to be learned. In this case, the noise in each data point has a faster learning speed, which leads the model to memorize noise instead of features to achieve 100% training accuracy.

## **Connection between ideal method and DASH**
First of all, we want to emphasize that, while DASH is inspired by the analysis of an abstract framework, it is a practical algorithm designed for real-world.

We acknowledge that our draft falls short of elucidating how DASH captures the principle of the idealized method. Here, we provide a clearer explanation of how DASH implements these principles in practice. A key component of DASH is shrinking by cosine similarity. We will detail how this technique helps the model retain features and forget noise.

The negative loss gradient for each neuron represents the direction to minimize the loss. As we discussed, minimizing loss involves two strategies: learning features/memorizing noise. We can view the negative loss gradient as a combination of directions for learning features and memorizing noise.

When a new chunk of data is added, features from the old data are likely to reappear frequently in the new data because these features are relevant to the class of data. In contrast, noise from the old data, which is class-independent and data-specific, is less likely to appear in the new data. As a result, neurons that have learned relevant features from previous experiments will have a high cosine similarity with the negative gradient of the new data. Conversely, neurons that have memorized noise will show a low cosine similarity with the negative gradient. Thus, shrinking by cosine similarity with the negative gradient of new data captures the essence of the idealized method: retaining features and forgetting noise. 

We further experimented to validate that DASH aligns with our intuition. We trained a 3-layer CNN on CIFAR10, varying the size of the training dataset. The model trained using more data seems to learn more features, evidenced by the rising trend in test accuracy shown by the dashed line in Figure 2 in our attached pdf when we plot the cosine similarity greater than 0.1. Also, we observe a trend where the cosine similarity between the negative gradient from test data and the learned weights increases as the training data size grows, which aligns with our intuition.

## **Computation and memory overhead**
We will address the reviewers' concerns about the computational and memory overhead of DASH. First, it's important to note that DASH is applied only when new data is introduced, not at every step or epoch. Since DASH calculates the gradient of the whole dataset just once, its memory complexity is proportional to the batch size and model size.
To clarify concerns about training time, when comparing experiments with the same number of epochs, you can think of DASH as adding approximately one extra epoch to the total training time. Similarly, the computational overhead of DASH is roughly equivalent to running one additional epoch.
We provide aggregate results for FLOPS, CPU/CUDA memory usage, and training time for the first two experiments on CIFAR-10 using ResNet-18. These measurements were obtained using the `torch.profile` library, and the values were summed across all operations.

**Total FLOPS, CPU/CUDA Memory**
- **Warm** FLOPS: 19 trillion, CPU: 7.4 GB, CUDA: 724 GB, Training Time: 37s
- **Cold** FLOPS: 25 trillion, CPU: 9.6 GB, CUDA: 1010 GB, Training Time: 48s
- **S&P** FLOPS: 21 trillion, CPU: 8.6 GB, CUDA: 828 GB, Training Time: 42s
- **DASH** FLOPS: 21 trillion, CPU: 8.3 GB, CUDA: 827 GB, Training Time: 41s

It is worth noting that S&P requires one additional epoch compared to DASH in the second experiment. Despite this difference, our analysis shows that the total computational/memory overheads of S&P and DASH are remarkably similar.

We plan to include these discussions in the next revised version of our paper.

We hope our response helps to resolve any concerns and confusion.

Best regards,  
Authors

---
Reference
- Shen et al. Data augmentation as feature manipulation. In ICML 2022

Meta-review: This paper introduces Direction-Aware SHrinking (DASH), a novel method to mitigate plasticity loss in warm-started neural networks under stationary data distributions (i.e., IID distributions). The approach is well-grounded and supported by experimental results, although these results are only on relatively small, low-resolution, image datasets.

**Strengths:**
- **Novel Contribution:** DASH presents a unique approach to addressing the warm start problem, grounded in a solid theoretical framework.
- **Experimental Validation:** The method shows promising results on benchmark datasets.

**Weaknesses:**
- **Clarity and Theory:** The connection between the theoretical model and the practical implementation of DASH lacks clarity, particularly regarding the role of cosine similarity.
- **Evaluation Scope:** The evaluation is limited to toy datasets, raising concerns about DASH's scalability and performance on larger datasets like ImageNet-1K.
-**Efficiency:** While efficiency is one of the paper's central focuses, it is unclear how efficient the method is. While the number of steps (gradient updates) is reported, timing experiments are not detailed (although some timing information was provided during the rebuttal). Moreover, they only study SGD and SAM, rather than optimizers often used for training deep neural networks in the current era (e.g., Adam, AdamW).
- **Applicability:** The broader applicability of DASH, especially in non-stationary settings like class incremental learning, remains unclear. The authors did provide information during the rebuttal period, showing that their method worked poorly for class incremental learning.

**Recommendation: Accept.** The authors are encouraged to incorporate reviewer feedback and include the issues regarding non-stationary distributions in the limitations section.
Decision: Accept (poster)