DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity

Authors: baekrok  shin
junsoo  oh
hanseul  cho
chulhee  yun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 03:50:45

Abstract: Warm-starting neural network training by initializing networks with previously learned weights is appealing, as practical neural networks are often deployed under a continuous influx of new data. However, it often leads to *loss of plasticity*, where the network loses its ability to learn new information, resulting in worse generalization than training from scratch. This occurs even under stationary data distributions, and its underlying mechanism is poorly understood. We develop a framework emulating real-world neural network training and identify noise memorization as the primary cause of plasticity loss when warm-starting on stationary data. Motivated by this, we propose **Direction-Aware SHrinking (DASH)**, a method aiming to mitigate plasticity loss by selectively forgetting memorized noise while preserving learned features. We validate our approach on vision tasks, demonstrating improvements in test accuracy and training efficiency.

Main paper: 
 introduction: 
 When training a neural network on a gradually changing dataset, the model tends to lose its plasticity, which refers to the model's ability to adapt to new information (Dohare et al., 2021; Lyle et al., 2023b; Nikishin et al., 2022) . This phenomenon is particularly relevant in scenarios with non-stationary data distributions, such as reinforcement learning (Igl et al., 2020; Nikishin et al., 2022) and continual learning (Chen et al., 2023; Kumar et al., 2023; Wu et al., 2021) . While requiring to overwrite outdated knowledge as the environment changes, models overfitted to previously encountered environments often struggle to cumulate new information, which in turn leads to reduced generalization performance (Lyle et al., 2023b) . Under this viewpoint, various efforts have been made to mitigate the loss of plasticity, such as resetting layers (Nikishin et al., 2022) , regularizing weights (Kumar et al., 2023) , and modifying architectures (Lee et al., 2023; Lyle et al., 2023a; Nikishin et al., 2023) .Perhaps surprisingly, a similar phenomenon occurs in supervised learning settings, even where new data points sampled from a stationary data distribution are added to the dataset during training. It is counterintuitive, as one would expect advantages in both generalization performance and computational efficiency when we warm-start from a model pre-trained on data points of the same distribution. For a particular example, when a model is pre-trained using a portion of a dataset and then we resume the training with the whole dataset, the generalization performance is often worse than a model trained from scratch (i.e., cold-start), despite achieving similar training accuracy (Ash and Adams, 2020; Berariu et al., 2021; Igl et al., 2020) . Liu et al. (2020) report a similar observation: training neural networks with random labels leads to a spurious local minimum which is challenging to escape from, even when retraining with a correctly labeled dataset. Interestingly, Igl et al. (2020) found that pre-training with random labels followed by the corrected dataset yields The dataset is divided into 50 chunks, with a constant number of data points added to the training dataset in each experiment (x-axis), reaching the full dataset at the 50th experiment. Models are trained until achieving 99.9% train accuracy before proceeding to the next experiment; the plot on the right reports the number of update steps executed in each experiment. Results are averaged over three random seeds. "Cold" refers to cold-starting and "Warm" refers to warm-starting. The Shrink & Perturb (S&P) method involves shrinking the model weights by a constant factor and adding noise (Ash and Adams, 2020) . Notably, DASH, our proposed method, achieves better generalization performance compared to both training from scratch and S&P, while requiring fewer steps to converge.better generalization performance than pre-training with a small portion of the (correctly labeled) dataset and then training with the full, unaltered dataset. It is striking that warm-starting leads to such a severe loss of performance, even worse than that of a cold-started model or a model re-trained from parameters pre-trained with random labels, despite the stationarity of the data distribution.These counterintuitive results prompt us to investigate the underlying reasons for them. While some studies have attempted to explain the loss of plasticity in deep neural networks (DNNs) under non-stationarity (Lewandowski et al., 2023; Lyle et al., 2023b; Sokar et al., 2023) , their empirical explanations rely on various factors, such as model architecture, datasets, and other variables, making it difficult to generalize the findings (Lewandowski et al., 2023; Lyle et al., 2023a) . Moreover, there is limited research that explores why warm-starting is problematic in stationary settings, highlighting the lack of a fundamental understanding of the loss of plasticity phenomenon in both stationary and non-stationary data distributions.In this work, we aim to explain why warm-starting leads to worse generalization compared to coldstarting, focusing on the stationary case. We propose an abstract framework that combines the popular feature learning framework initiated by Allen-Zhu and Li (2020) with a recent approach by Jiang et al. (2024) that studies feature learning in a combinatorial and abstract manner. Our analysis suggests that warm-starting leads to overfitting by memorizing noise present in the newly introduced data rather than learning new features.Inspired by this finding, we propose Direction-Aware SHrinking (DASH), which aims to encourage the model to forget memorized noise without affecting previously learned features. This enables the model to learn features that cannot be acquired through warm-starting alone, enhancing the model's generalization ability. We validate DASH using an expanding dataset setting, similar to the approach in Ash and Adams (2020) , employing various models, datasets, and optimizers. As an example, Figure 1 shows promising results in terms of both test accuracy and training time.Loss of Plasticity. Research has aimed to understand and mitigate loss of plasticity in non-stationary data distributions. Lewandowski et al. (2023) explain that loss of plasticity co-occurs with a reduction in the Hessian rank of the training objective, while Sokar et al. (2023) attribute it to an increasing number of inactive neurons during training. Lyle et al. (2023b) find that changes in the loss landscape curvature caused by non-stationarity lead to loss of plasticity. Methods addressing this issue in non-stationary settings include recycling dormant neurons (Sokar et al., 2023) , regularizing weights towards initial values (Kumar et al., 2023) , and combining techniques (Lee et al., 2023) like layer normalization (Ba et al., 2016) , Sharpness-Aware Minimization (SAM) (Foret et al., 2020) , resetting layers (Nikishin et al., 2022) , and Concatenated ReLU activation (Shang et al., 2016) .However, these explanations and methods diverge from the behavior observed in stationary data distributions. Techniques aimed at mitigating loss of plasticity under non-stationarity are ineffective under stationary distributions, as shown in Appendix C.1, in line with the observations in Lee et al. (2023) . While some works study the warm-starting problem in stationary settings, they rely on empirical observations without theoretical analysis (Achille et al., 2018; Ash and Adams, 2020; Berariu et al., 2021) . The most relevant work by Ash and Adams (2020) introduces the Shrink & Perturb (S&P) method, which mitigates the loss of plasticity in stationary settings to some extent by shrinking all weight vectors by a constant factor and adding noise. However, they do not explain why this phenomenon occurs or why S&P is effective. We develop a theoretical framework explaining why warm-starting suffers even under stationary distribution. Based on findings, we propose a method that shrinks the weight vector in a direction-aware manner to maintain properly learned features.Feature Learning in Neural Networks. Recent studies have investigated how training methods and network architectures influence generalization performance, focusing on data distributions with label-dependent features and label-independent noise (Allen-Zhu and Li, 2020; Cao et al., 2022; Deng et al., 2023; Jelassi and Li, 2022; Oh and Yun, 2024; Zou et al., 2023) . In particular, Shen et al. (2022) examine a data distribution consisting of varying frequencies of features and large strengths of noise, emphasizing the significance of feature frequencies in learning dynamics. Jiang et al. (2024) propose a novel feature learning framework based on their observations in real-world scenarios, which also involves features with different frequencies but considers the learning process as a discrete sampling process. Our framework extends these ideas by incorporating features with varying frequencies, noise components, and the discrete learning process while introducing a more intricate learning process capturing the key aspects of feature learning dynamics in expanding datasets.2 A Framework of Feature LearningQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and introduction clearly state the main claims and contributions of the paper, including the development of a learning framework to study loss of plasticity in stationary data distributions, identifying noise memorization as the primary cause, and proposing the DASH method to mitigate the issue. The claims match the theoretical and empirical results presented. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
motivation and intuition: 
 We present the motivation and intuition behind our framework before delving into the formal description. Our framework captures key characteristics of image data, where the input includes both label-relevant information (referred to as features, e.g., cat faces in cat images) and label-irrelevant information (referred to as noise, e.g., grass in cat images). A key intuition is that minimizing training loss involves two strategies: learning features and memorizing noise. This framework builds on insights from Shen et al. (2022) and integrates them into a discrete learning framework. We provide more detailed intuition on our framework, including the training process, in Appendix A. Shen et al. (2022) consider a neural network trained on data with features of different frequencies and noise components stronger than the features. The negative gradient of the loss for each single data point aligns more with the noise than the features due to the larger scale of noise, making the model more likely to memorize noise rather than learn features. However, an identical feature appears in many data points, while noise appears only once and does not overlap across data points. Thus, if a feature appears at a sufficiently high frequency in the dataset, the model can learn the feature. Thus, the learning of features or noise depends on the frequency of features and the strength of the noise.Inspired by Shen et al. (2022) , we propose a novel discrete feature learning framework. This section introduces a framework describing a single experiment, while Section 3 analyzes expanding dataset scenarios. As our focus is on gradually expanding datasets, carrying out the (S)GD analysis over many experiments as in Shen et al. (2022) is highly challenging. Instead, we adopt a discrete learning process similar to Jiang et al. (2024) but propose a more intricate process reflecting key ideas from Shen et al. (2022) . In doing so, we generalize the concept of plasticity loss and analyze it without assuming any particular hypothesis class for a more comprehensive understanding, whereas existing works are limited to specific architectures.We consider a classification problem with C classes, and data are represented as(x, y) ∈ X × [C],where X denotes the input space. A data point is associated with a combination of class-dependent features V(x) ⊂ S y where S c = {v c,1 , v c,2 , . . . , v c,K } is the set of all features for each class c ∈ [C]. Also, every data point contains point-specific noise which is class-independent.The model f : X → [C] sequentially learns features based on their frequency. The training process is described by the set of learned features L ⊂ S ≜ c∈[C] S c and the set of data points with non-zero gradients N ⊂ T , where T = {(x i , y i )} i∈[m] denotes a training set. The set N , representing the data points with non-zero gradients, will be defined below. The frequency of a feature v in data points belonging to N is denoted byg(v; T , N ) = 1 |T | (x,y)∈N 1(v ∈ V(x)),where 1(•) is the indicator function, which equals 1 if the condition inside the parentheses is true and 0 otherwise. At each step of training, if L and N are given, the model chooses the most frequent feature among the features not yet learned, i.e., arbitrarily choose v ∈ arg max u∈S\L g (u; T , N ).The model decides whether to learn a selected feature v by comparing its signal strength, represented by |T | • g(v; T , N ), with the signal strength of noise, given by γ, which reflects the key ideas of Shen et al. (2022) . If the frequency of the selected feature v is no less than the threshold γ/|T |, i.e., g(v; T , N ) ≥ γ/|T |, the model learns v and adds it to its set of learned features L. The feature learning process continues until the model reaches a point where the selected feature v has g(v; T , N ) < γ/|T |, indicating that the signal strength of every remaining feature is weaker than that of noise. At this point, the feature learning process ends.We consider a data point x to be well-classified if the model f has learned at least τ features from V(x), i.e., |L ∩ V(x)| ≥ τ , where τ < K. In this case, we consider x to have a zero gradient, meaning it cannot further contribute to the learning process. Throughout the feature learning process, the set N of data points with non-zero gradients is dynamically updated as new features are learned. At each step, when the model successfully learns a new feature, we update N by removing the data points that satisfy |L ∩ V(x)| ≥ τ , as they become well-classified due to the newly learned feature.If the feature learning process ends and the model has learned as many features as it can, the remaining data points that have non-zero gradients will be memorized by fitting the random noise present in them and will be considered to have zero gradients. This step concludes the training process. Pictorial illustration can be found in Appendix A and a detailed algorithm of the learning process can be found in Algorithm 2 in Appendix E.In our framework, the model selects features based on their frequency in the set of unclassified data points N . The intuition behind this approach is that features appearing more frequently in the set of data points will have larger gradients, leading to larger updates, and we treat g(v; T , N ) as a proxy of the gradient for a particular feature v. As a result, the model prioritizes sequentially learning these high-frequency features. However, if the frequency g(v; T , N ) of a particular feature v is not sufficiently large, such that the total occurrence of v is less than the strength of the noise, i.e., |T | • g(v; T , N ) < γ, the model will struggle to learn that feature. Consequently, the model will prioritize learning the noise over the informative features. When this situation arises, the learning procedure becomes sub-optimal because the model fails to capture the true underlying features of the data and instead memorizes the noise to achieve high training accuracy.The threshold τ determines when a data point is considered well-classified and acts as a proxy for the dataset's complexity. A higher τ requires the model to learn more features for correct predictions, while a lower τ allows accurate predictions with fewer learned features. Experiments in Appendix B Figure 11 and 12 support this interpretation. Remark 2.1. We believe our analysis can be extended to scenarios where feature strength varies across data by treating the set of features as a multiset, where multiple instances of the same element are allowed. The analyses in these cases are nearly identical to ours; therefore, we assume all features have identical strengths for notational simplicity.3 Warm-Starting versus Cold-Starting, and a New Ideal MethodQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: This paper includes a "Discussion and Conclusion" section that reflects on the scope and limitations of the work. For example, it notes that the theoretical analysis treats the learning process as discrete and does not assume any specific hypothesis class in order to generalize the findings. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Section 6 discusses the positive impact of the work in enabling more efficient and sustainable AI that can adapt to increasing real-world data. It notes this can help prevent wasted time, energy and environmental impact. Potential negative impacts are not explicitly discussed as the work is largely theoretical. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments with expanding dataset: 
 In this section, we set up the scenario where the dataset grows after each experiment in our learning framework, allowing us to compare warm-start, cold-start, and a new ideal method, which will be defined later in Sections 3.3 and 3.4.To better understand the loss of plasticity under stationary data distribution, we consider an extreme form of stationarity where the frequency of each feature combination remains constant in each chunk of additional data. We investigate if the loss of plasticity can manifest even under this strong stationarity. The detailed description of the dataset across the entire experiment is as follows: Assumption 3.1. In each j-th experiment, we are provided with a training dataset T j := {(x i,j , y i,j )} i∈ [n] with n samples. For each class c ∈ [C] and each possible feature combination A ⊂ S c , we assume that T j contains exactly n A ≥ 1 data points with associated feature set A, where the values of n A are independent of j. Note that c∈[C],A⊂Sc n A = n. In the j-th experiment, we use the cumulative dataset T 1:j := l∈[j] T l , the set of all training data up to the j-th experiment.Remark 3.2. In each experiment, the feature combinations remain the same across the dataset, but the individual data points differ. This is because each data point is associated with its specific noise, which varies across samples. Although the underlying features are the same, the noise component of each data point is unique. This approach ensures that the model is exposed to a diverse set of samples.h(v; L) ≜ 1 n c∈[C],A⊂Sc n A • 1 (v ∈ A ∧ |A ∩ L| < τ )to denote the portion of data points containing v not-well-classified by feature set L. This leads to assumption:Assumption 3.3. For any learned feature set L ⊂ S, if v 1 , v 2 ∈ S c for some class c ∈ [C] and h(v 1 ; L) = h(v 2 ; L), then v 1 = v 2 . Also, for any class c ∈ [C], there exists some τ distinct features v 1 , . . . , v τ ∈ S c such that g(v 1 ; T j , T j ), . . . , g(v τ -1 ; T j , T j ) ≥ γ/n and g(v τ ; T j , T j ) < γ/n.This assumption leads to Lemma D.2, stating that the order in which features are learned within a class is deterministic. This is just for simplicity of presentation and can be relaxed. The last assumption is justified by the moderate number of data points in each chunk T j , ensuring the existence of both τ -1 learnable features and a non-learnable feature within a class. Throughout the following discussion, we will proceed under the above assumptions unless otherwise specified.Notation. We denote a model at step s of the j-th experiment as f (j,s) . We denote the set of learned features and the set of memorized data for the model f (j,s) as L (j,s) and M (j,s) , respectively. We also define the set of data points with non-zero gradients at step s of the j-th experiment as N (j,s) . We define respective versions of these sets and the model, with different initialization methods, denoted by the subscripts (e.g., f (j,s) warm , f (j,s) cold , and f (j,s) ideal ). We emphasize that each method initializes f (j,0) , L (j,0) , M (j,0) , and N (j,0) differently at the start of the j-th experiment.We provide a comparison of three initialization methods based on test accuracy and training time. To evaluate these metrics within our framework, we define the prediction process and training time.Prediction Process. The model predicts unseen data points by comparing the learned features with features present in a given data point x. If the overlap between the learned feature set L and the features in x, denoted as V(x), is at least τ , i.e., |V(x) ∩ L| ≥ τ , the model correctly classifies the data point. Otherwise, the model resorts to random guessing.Training Time. Accurately measuring training time within our discrete learning framework is challenging. To address this, we introduce an alternative for training time of j-th experiment: the number of training data points with non-zero gradients at the start of j-th experiment, |N (j,0) |. This represents the amount of "learning" required for the model to classify all data points correctly. We empirically validated this proxy in practical scenarios, as shown in Figures 6 and 7 in Appendix B.1. Additionally, Nakkiran et al. (2021) observe that in real-world neural network training, when other components are fixed, the training time increases with the number of data points to learn.Now we analyze the warm-start and cold-start initialization methods within our framework, focusing on test accuracy and training time. We note that, by definition, L , where s j denotes the last step of j-th experiment. Besides, we use a shorthand notation for step s j of the experiment j that we drop s if s = s j (e.g., L (j) := L (j,sj ) ). For the detailed algorithms based on our learning framework, see Algorithms 3 and 4 in Appendix E.In the test data, a feature combination A ⊂ S c of data point with class c ∈ [C] appears with probability n A /n along with data-specific noise. By Section 3.2, test accuracy for a learned set L and training time are defined as:ACC(L) ≜ 1 -C-1 C • 1 n c∈[C],A⊂Sc n A • 1 (|A ∩ L| < τ ) T (J) warm ≜ j∈[J] N (j,0) warm , T (J) cold ≜ j∈[J] N (j,0) coldBased on these definitions, the following theorem holds:Theorem 3.4. There exists nonempty G ⊊ S such that we always obtain L(1) warm = L(1) cold = G. For all J ≥ 2, the following inequalities hold:ACC L (J) warm ≤ ACC L (J) cold , T (J) warm < T (J) coldFurthermore, ACC(L(J) warm ) < ACC(L (J) cold ) holds when J > γ δn where δ ≜ max v∈S\G h(v; G) > 0.Proof Idea. After the first experiment, the data points in T 1 cannot further contribute to the learning process of the warm-started model. Consequently, even when a new data chunk is provided in subsequent experiments, the feature frequencies are too small, resulting in a weak signal strength of features that cannot overcome the noise signal strength. As a result, the model memorizes individual noise components of the new data points. This procedure is repeated with every experiment, causing the learned feature set to remain the same as at the end of the first experiment. In contrast, when receiving T 1:j at once (cold-starting), the signal strength of features is large enough to overcome the noise signal strength, allowing the model to learn many more features.Theorem 3.4 highlights a trade-off between cold-starting and warm-starting. Regarding test accuracy, the theorem concludes that cold-starting can achieve strictly higher accuracy than warm-starting. However, warm-starting requires a strictly shorter training time compared to cold-starting.Detailed proof is provided in Appendix D. Theorem 3.4 suggests that the loss of plasticity in the incremental setting under the stationary assumption can be attributed to the noise memorization process. A similar observation is made in real-world neural network training. It is widely believed that during the early stages of training, neural networks primarily focus on learning features from the dataset, and after learning these features, the model starts to memorize data points that it fails to classify correctly using the learned features. To investigate this phenomenon, we conducted an experiment where CIFAR-10 was divided into two chunks, each containing 50% of the training dataset. The model was pre-trained on one chunk and then further trained on the full dataset for 300 epochs. We used three-layer MLP and ResNet-18 with SGD optimizer across 10 random seeds.Figure 2 shows the change in the model's performance based on the duration of pre-training. When pre-training is stopped at a certain epoch and the model is then trained on the full dataset, test accuracy is maintained. However, if pre-training continues beyond a specific threshold (approximately 50% pre-training accuracy in this case), warm-starting significantly impairs the model's performance as it increasingly memorizes training data points. We attribute this phenomenon to the neural network's memorization process after learning features. This is consistent with reports of a critical learning period where neural networks learn useful features in the early phase of learning (Achille et al., 2018; Frankle et al., 2020; Kleinman et al., 2024) , and with findings that neural networks tend to learn features followed by memorizing noises (Arpit et al., 2017; Jiang et al., 2020) . Using the same experimental settings as in Figure 2 , we tested with a large-scale dataset, ImageNet-1k, and observed similar trends (see Figure 8 in Appendix B). Remark 3.5. Igl et al. (2020) find that training a model on random labels followed by corrected labels results in better generalization compared to pre-training on a subset of correctly labeled data and then further training on the full dataset with the same distribution. Achille et al. (2018) also observe that pre-training with slightly blurred images followed by original images yields worse test accuracy than pre-training with random label or random noise images. These findings align with our observations: re-training with corrected labels after random label learning "revives" gradients for most memorized data points, enabling new feature learning. Conversely, with static distributions, gradients for memorized data points remain suppressed, leading to learning from only a few data points with active gradients, causing memorization. In Section 3.3, we observed a trade-off between warm-starting and cold-starting. Cold-starting often achieves better test accuracy compared to warm-starting, while warm-starting requires less time to converge. The results suggest that neither retaining all learned information nor discarding all learned information is ideal. To address this trade-off and get the best of both worlds, we consider an ideal algorithm where we retain all learned features while forgetting all memorized data points. For any experiment J ≥ 2, if we consider the ideal initialization, learned features L (J-1) ideal are retained, and memorized data points M (J-1) ideal are reset to an empty set. Pseudo-code for this method is given in Algorithm 5, which can be found in Appendix E. We define T(J) ideal ≜ j∈[J] N (j,0)ideal as the training time with the ideal method, where N (j,0) ideal represents the set of data points having a non-zero gradient at the initial step of the j-th experiment. Then, we have the following theorem: Theorem 3.6. For any experiment J ≥ 2, the following holds:ACC L (J) cold = ACC L (J) ideal , T (J) warm < T (J) ideal < T (J) coldThe detailed proof is provided in Appendix D. The ideal algorithm addresses the trade-off between cold-starting and warm-starting. We conducted an experiment to investigate the performance gap between these initialization methods.Synthetic Experiment. To verify our theoretical findings in more realistic scenarios, we conducted an experiment that more closely resembles real-world settings. Instead of fixing the frequency of each feature set, we sampled each feature's existence from a Bernoulli distribution to construct V(x). This ensures that the experiment is more representative of real-world scenarios. Specifically, for each data point (x, y), we uniformly sampled y ∈ {0, 1}. From the feature set S y corresponding to the sampled class y, we sampled features where each feature's existence follows a Bernoulli distribution,1 (v y,k ∈ V(x)) ∼ Ber(p k ), for all v y,k ∈ S y . This approach allows us to model the variability in feature occurrence that is commonly observed in real-world datasets while still maintaining the core principles of our learning framework. We set the number of features, K = 50, with p k sampled from a uniform distribution, U(0, 0.2). Each chunk contained 1000 data points with total 50 experiments, with γ = 50, τ = 3. We sampled 10000 test data from the same distribution.As shown in Figure 3 , the results align with the above theorems. Random initialization, i.e. coldstarting, and ideal initialization achieve almost identical generalization performance, outperforming warm initialization. However, with warm initialization, the model converges faster, as evidenced by the number of non-zero gradient data points, which serves as a proxy for training time. Ideal initialization requires less time compared to cold-starting, which is also consistent with Theorem 3.6. Due to the sampling process in our experiment, we observe a gradual increase in the number of learned features and test accuracy in warm-starting, mirroring real-world observations. These findings remained robust across diverse hyperparameter settings (see Figures 9-11 in the Appendix B). there is a significant gap between random and warm initialization, which the ideal method addresses. 
dash: direction-aware shrinking: 
 The ideal method recycles memorized training samples by forgetting noise while retaining learned features. From now on, we shift our focus to a practical scenario: training neural networks with real-world data. This brings up the question of whether such an ideal approach can be applied in real-world settings. To address this, we propose our algorithm, Direction-Aware SHrinking (DASH), which intuitively captures this idea in practical training scenarios. The outlined behavior is illustrated in Figure 4 . When new data is introduced, DASH shrinks each weight based on its alignment with the negative gradient of the loss calculated from the training data, placing more emphasis on recent data.If the degree of alignment is small (i.e., the cosine similarity is close to or below 0), we consider that the weight has not learned a proper feature and shrinks it significantly to make it "forget" learned information. This allows weights to forget memorized noises and easily change their direction. On the other hand, if the weight and negative gradient are well-aligned (i.e., the cosine similarity is close to 1), we consider it learned features and we shrink the weight to a lesser degree to maintain the learned information. This method aligns with the intuition of the ideal method, as it allows us to shrink weights that have not learned proper information while retaining weights that have learned commonly observed features.Noiseθ 1 θ 2 θ1 θ2 -∇ θ 1 L(Θ; 𝒯 1:j ) -∇ θ 2 L(Θ; 𝒯 1:j )Figure 4 : Illustration of DASH. We compute the loss L with training data T 1:j and obtain the negative gradient. Then, we shrink the weights proportionally to the cosine similarity between the current weight θ and ∇ θ L, resulting in θ.Algorithm 1 Direction-Aware SHrinking (DASH)Require:• Model fΘ with list of parameters Θ after the (j-1)-th experiment • Training data points T1:j • Averaging coefficient 0 < α ≤ 1 • Threshold λ > 0 1: Initialize: G (0) θ ← 0, ∀θ in Θ 2: for i in 1 : j do 3: ℓ ← Loss(fΘ, Ti) 4: UΘ ← Gradient of loss ℓ 5: for θ in Θ do 6: G (i) θ ← (1 -α) × G (i-1) θ + α × U θ 7:end for 8: end for 9: for θ in Θ do 10:s θ ← CosineSimilarity -G (j)θ , θ 11:θ ← θ ⊙ max{λ, s θ } 12: end for 13: return model fΘ, initialized for the j-th experimentThe shrinking is done per neuron, where the incoming weights are grouped into a weight vector denoted as θ. For convolutional filters, the height and width of the kernel are flattened to form a single weight vector θ for each pair of input and output filters. DASH has two hyperparameters: λ and α. Hyperparameter λ is the minimum shrinkage threshold, as each weight vector is shrunk by max{λ, cos_sim}, while α denotes the coefficient of exponential moving average of per-chunk loss gradients. Lower α value gives more weight to previous gradients, resulting in less shrinkage. This is because gradients of the previously seen data usually have high cosine similarity with learned weights; low α is advantageous for simpler datasets where preserving learned features helps. Note that DASH is an initialization method applied only once when new data is introduced. The detailed algorithm is presented in Algorithm 1.We discuss our intuition regarding the connection between DASH and the spirit of the ideal method. Generally, features from previous data are likely to reappear frequently in new data, as they are relevant to the data class. In contrast, noise from previous data rarely reappears as it is not classspecific and varies with each data point. As a result, the negative gradient of the loss naturally aligns more closely with learning features rather than memorizing noise from older data. This suggests that when neurons are aligned with its negative gradient of the loss, we can assume they have learned important features and should not be shrunk.To validate our intuition, we plotted the accuracy on previously learned data points a few epochs after applying DASH in Figure 13 , Appendix B. Our experiments show that DASH recovers training accuracy on previous datasets more quickly than other methods, likely because it preserves learned features while discarding memorized noise. As experiments progress, the growing number of learned features allows DASH to retain more information, leading to improved training accuracy across successive experiments. We further validate our intuition behind DASH in Figure 14 in Appendix B. 
experimental details: 
 Our setup is similar to the one described in Ash and Adams (2020) . We divided the training dataset into 50 chunks, and at the beginning of each experiment, a chunk is added to the existing training data. Models were considered converged and each experiment was terminated when training accuracy reached 99.9%, aligning with our learning framework. We conducted experiments with vanilla training i.e. without data augmentations, weight decay, learning rate schedule, etc. Appendix C.4 presents additional results on other settings, including the state-of-the-art (SoTA) settings that include the techniques mentioned above. We evaluated DASH on Tiny-ImageNet, CIFAR-10, CIFAR-100, and SVHN using ResNet-18, VGG-16, and three-layer MLP architectures with batch normalization layer. Models were trained using Stochastic Gradient Descent (SGD) and Sharpness-Aware Minimization (SAM) (Foret et al., 2020) , both with momentum.DASH was compared against baselines (cold-starting, warm-starting, and S&P (Ash and Adams, 2020) ) and methods addressing plasticity loss under non-stationarity (L2 INIT (Kumar et al., 2023) and Reset (Nakkiran et al., 2021) ). Layer normalization (Ba et al., 2016) and SAM (Foret et al., 2020) , known to mitigate plasticity loss in reinforcement learning (Lee et al., 2023) , were applied to both warm and cold-starting. Consistent hyperparameters were used across all methods, with details provided in Appendix C.3. S&P, Reset, and DASH were applied whenever new data was introduced. We report two metrics for both test accuracy and number of steps required for convergence: the value from the final experiment and the average across all experiments.We first experimented with CIFAR-10 on ResNet-18 to determine if methods from previous works for mitigating plasticity based on non-stationarity can be a solution to our incremental setting with stationarity. Appendix C.1 shows that L2 INIT, Reset, layer normalization, and reviving dead neurons, are not effective in our setting. Thus, we conducted the remaining experiments without these methods. Additionally, Table 1 shows that warm-starting with SAM does not outperform cold-starting with SAM, indicating that SAM alone is not an effective method in our case. Table 1 shows that DASH surpasses cold-starting (Random Init) and S&P in most cases. Training times were often shorter compared to training from scratch, and when longer, the performance gap in test accuracy was more pronounced. Omitted results are in Tables 3-6 located in Appendix C.2. Additionally, we confirm that DASH is computationally efficient, with details on the computation and memory overhead comparisons provided in Appendix C.5.We argue that S&P can cause the model to forget learned information, including important features, due to shrinking every weight uniformly and perturbing weights. This leads to increased training time and relatively lower test accuracy, especially in SoTA settings (see Appendix C.4.1). In contrast, DASH addresses these issues by preserving learned features with direction-aware weight shrinkage. Init 25.69 (0.13) 31.30 (0.09) 30237 (368) 40142 (368) 17.37 (0.06) 21.95 (0.11) 17503 (53)  22513 (74  
discussion and conclusion: 
 In this work, we defined an abstract framework for feature learning and discovered that warm-starting benefits from reduced training time compared to random initialization but can hurt the generalization performance of neural networks due to the memorization of noise. Motivated by these observations, we proposed Direction-Aware SHrinking (DASH), which shrinks weights that learned data-specific noise while retaining weights that learned commonly appearing features. We validated DASH in real-world model training, achieving promising results for both test accuracy and training time.Loss of plasticity is problematic in situations where new data is continuously added daily, which is the case in many real-world application scenarios. Our research aimed to interpret and resolve this issue, preventing substantial waste of energy, time, and the environment. By elucidating the loss of plasticity phenomenon in stationary data distributions, we have taken a crucial step towards addressing challenges that may emerge in real-world AI, where the continuous influx of additional data is inevitable.We hope our fundamental analysis of the loss of plasticity phenomenon sheds light on understanding this issue as well as providing a remedy. To generalize our findings to any neural network architecture, we treated the learning process as a discrete abstract procedure and did not assume any hypothesis class. Future research could focus on understanding the loss of plasticity phenomenon via optimization or theoretically analyzing it in non-stationary data distributions, such as in reinforcement learning.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper focuses on theoretical analysis and experiments on common open datasets. It does not introduce any models or datasets with high risk of misuse that would require safeguards. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: This paper does not rely on or adapt any existing code or model assets. The datasets used are common open datasets and cited appropriately. 
 
 Appendix: In this section, we provide a detailed explanation of how our theoretical framework reflects the intuitive process of learning from image data. Our framework is designed to capture the characteristics of image data, where the input includes both relevant information for the image labels (which we refer to as "features" such as a cow's ears, eyes, tail, and mouth in Figure 5a ) and irrelevant information (which we refer to as "noise" such as the sky or grass circled in red in Figure 5a ). Our framework builds on the insights from Shen et al. (2022) and incorporates them into a discrete learning model.Each data point is associated with a combination of featuresx 1 x 2 x 3 x 4 x 5 x 6x 6 x 1 Before explaining the intuition of our learning framework, let us connect notation from Section 2 with Figure 5 . Figure 5 shows six data points from a single cow class, where training set T = {(x i , y)} i∈ [6] . Features are denoted by their color's first letter: violet as v, blue as b, green as g, mint as m, yellow as y, and pink as p. The complete feature set S is therefore {v, b, g, m, y, p}. The learned feature set L is empty at initialization, and the set of data points with non-zero gradients N ⊂ T is equal to T at initialization. We set the data learning threshold τ = 2 and noise strength γ = 2.Our training process is based on the idea that features which occur more frequently in the training data are easier to learn, as the gradients tend to align more strongly with these frequent features. Consequently, our framework is designed to learn the most frequent features in a sequential manner. For example, a cow's ears, shown in violet in Figure 5 would be selected first since it is the most frequent feature:v = arg max c∈S\L (0) g(c; T , N (0) ) = arg max c∈{v,b,g,m,y,p} g(c; T , T )where L (s) and N (s) represent the learned feature set and the set of data points with non-zero gradients, respectively, at step s. This holds because:g(v; T , N (0) ) = 1 |T | (x,y)∈N (0) 1(v ∈ V(x)) = 5 6 > g(c; T , N (0) )for all c ∈ {b, g, m, y, p}. Since we have set γ to 2, the feature v is learned because its frequency is sufficiently high compared to the noise, i.e., g(v; T , N (0) ) ≥ γ/|T | = 2 6 . Also, since τ set to 2, no data point is considered well-classified at this stage, i.e., |{v} ∩V(x i )| < τ for all i ∈ [6]. As a result, L (1) = {v}, N (1) = T .In the next step, the model learns feature b in the same way. Therefore, x 1 , x 3 , and x 4 are now considered well-classified since they share two overlapping features with the learned feature set L (2) = {v, b}. This indicates that the model accumulates enough information to accurately classify these cow images as cows. Once the model correctly classifies data based on these features, the influence of those data points on the learning process diminishes, as the loss gradients become smaller as the predictions become more confident. To account for this, our framework evaluates the frequency of features only on data points that are not yet well-classified. Therefore, the set of data points with non-zero gradients is now updated to N (2) = {(x i , y)} i∈{2,5,6} . Subsequently, we compute the gradient proxy g(c; T , N (2) ) using only the data points in N (2) . In the third step, considering the features included in {x 2 , x 5 , x 6 }, we get that the feature g is the most frequent among those in S \ L (2) , and the feature meets the threshold for learning. Hence, the feature g is newly learned, leading us to L (3) = {v, b, g} and N (3) = {(x 6 , y)}.As training progresses, the algorithm may reach a point where the remaining features in the not-wellclassified data points are too infrequent to be learned effectively. At this point, the noise in each data point has a faster learning speed, which leads the model to memorize noise instead of learning the features to achieve 100% training accuracy. In the example described in Figure 5 , at the fourth step, the remaining feature m in x 6 does not meet the threshold: g(m; T , N (3) ) = 1/6 < γ/|T |, so the model is unable to learn feature m. Consequently, the model shifts to memorizing the noise in the remaining data point x 6 and completes training by memorizing the noise.This section demonstrates how our theoretical framework relates to real-world scenarios. Section B.1 presents real-world experimental evidence supporting our theoretical framework. Section B.2 explores the robustness of our synthetic experiments across various hyperparameters. Finally, Section B.3 provides empirical validation of the key intuitions behind DASH. We also investigate the effect of the pretrain epoch on warm-starting in ImageNet-1k classification with ResNet18, similar to the setup in Figure 2 in Section 3.3. We conducted experiments using different pretrain epochs: 30, 50, 70, and 150. Figure shows a declining trend in accuracy for warm-started models as pretrain epoch increases. Interestingly, we noted a similar phenomenon in Figure 2 in Section 3.3, where warm-starting does not negatively impact test accuracy when training resumes from earlier pretrained epochs, such as 30 in this case. This observation aligns with our theoretical framework, which suggests that neural networks tend to learn meaningful features first and then begin to memorize noise later. We believe this phenomenon is persistent across different datasets, model architectures, and optimizers. As stated in Section 2, we posited that τ could serve as a proxy for dataset complexity. Figure 11 shows that as τ increases, the threshold for considering a data point well-classified also increases, making it more difficult to correctly predict unseen data points. This difficulty is particularly pronounced for warm-starting, leading to a widening gap between the random initialization and warm initialization methods. Additionally, this phenomenon is observed in real-world neural network training, as depicted in Figure 12 . For datasets with higher complexity (from left to right), the gap between the two initialization methods widens, exhibiting the same trend as an increasing τ . To validate whether previously learned/memorized data points do not have large gradients when further trained with a combined dataset (existing + newly introduced data) using warm-starting, we plotted the train accuracy on the previous dataset for the first few epochs. Using ResNet-18 on CIFAR-10 (Figure 13 ), we found that warm-starting preserves performance on previously learned data even when training continues with combined datasets, supporting the main idea of Theorem 3.4.To verify whether DASH truly captures our intuitions from the ideal algorithms, we conducted an experiment using CIFAR-10 trained on ResNet-18, with the same experimental settings. Figure 13 demonstrates that when applying DASH, the train accuracy on previous datasets increases more rapidly after a few epochs compared to other methods. We argue that this behavior stems from our algorithm's ability to forget memorized noise while preserving learned features. As the number of experiments increases, the number of learned features also grows. For a fair comparison, we used λ = 0.05 for DASH, and when performing S&P and shrink, we shrank each weight by multiplying 0.05. In the case of S&P, after shrinking, we added noise sampled from N (0, 0.01 2 ). We further validate our intuition that cosine similarity between the negative loss gradient and model weights can indicate whether the model has indeed learned features. We trained a 3-layer CNN on CIFAR-10, varying the size of the training dataset. We observed that the cosine similarity between the negative gradient from the test data and the learned filters increases as the training dataset size grows. The model trained with more data appears to learn more features, as evidenced by the rising trend in test accuracy, shown by the dashed line in Figure 14 . This suggests that cosine similarity can capture whether the weights have learned features, which aligns with our expectations. This section presents additional experimental results and their corresponding hyperparameters. Appendix C.1 examines non-stationary solutions, demonstrating their limitations in stationary settings. The following section compares DASH against other baselines in various settings, excluding those methods from Appendix C.1 due to their poor performance in stationary conditions. Appendix C.3 provides a comprehensive list of hyperparameters used across all experiments from Section 5, as well as Appendix C.1 and C.2. Additionally, we validate DASH's applicability including state-of-the-art (SoTA) settings in Appendix C.4, and analyze computational and memory overhead in Appendix C.5. We ran our experiments on two distinct hardware setups. The first server had an Intel Gaudi-v2 HPU with 96GB of VRAM with four Intel Xeon Platinum 8380 40-core CPUs. The second server had an NVIDIA A6000 GPU with 48GB of VRAM, paired with two AMD EPYC 7763 64-core CPUs. We compared the computing performance of these two hardware setups in Appendix C.6.Unless specified otherwise, we follow the experimental settings outlined in Section 5.1. We conducted an experiment with an incremental training dataset comprised of 50 chunks. At the start of each experiment, a new chunk is provided and added to the existing training dataset. Before proceeding to the next experiment, the model is trained until achieving 99.9% train accuracy.In this subsection, we describe solutions that aim to mitigate plasticity loss under non-stationarity, which cannot remedy the loss of plasticity in an incremental setting with a stationary data distribution. Table 2 shows L2 INIT (Kumar et al., 2023) and Reset (Nikishin et al., 2022 ) cannot be a solution in our setting. Furthermore, applying layer normalization cannot close the gap between cold-starting and warmstarting; rather, the gap increases, as shown in Figure 15 . Also, Nikishin et al. (2022) and Sokar et al. (2023) state that loss of plasticity in non-stationary data distributions arises from inactive neurons in the model. However, this is not the case in our setting, as demonstrated in Figure 16 . and the analysis reveals that there are no dead neurons. This finding suggests that techniques designed to revive dead neurons in non-stationary data distributions cannot effectively address the plasticity loss observed in the incremental learning setting with stationary data, which is the primary focus of our study.This subsection presents our complete experimental results (Tables 3 4 5 6 ), following the settings detailed in Section 5.1. We evaluated multiple approaches, including "Warm ReM" (warm-starting with momentum reset), which also proved ineffective. We conducted comprehensive comparisons across multiple datasets (Tiny-ImageNet, CIFAR-10, CIFAR-100, and SVHN), architectures (ResNet-18, VGG-16, three-layer MLP), and optimizers (SGD and SGD-based SAM). The results demonstrate that DASH consistently outperforms baseline methods -warm-starting, cold-starting, and S&Pwhile often requiring less training time. Results are averaged across five random seeds, except for Tiny-ImageNet which uses three random seeds due to its high computational cost. In this section, we provide the details of the hyperparameters used in our experiments from Section 5, as well as Appendix C.1 and C.2. Additionally, we present heatmaps illustrating the results for a wide range of two hyperparameters, α and λ, in DASH. The heatmaps in Figure 17 suggest that DASH exhibits robustness to hyperparameter variations, indicating that its performance is less affected by the choice of hyperparameter values.We fixed the momentum to 0.9 and the batch size to 128. The learning rate is set to 0.001 for training ResNet-18, and for other models, a learning rate of 0.01 is used. The value of ρ for SAM is chosen based on the performance of cold-starting. The default value of α = 0.3 is used, and we did not change this value frequently. The perturbation parameter σ used in the Shrink & Perturb (S&P) procedure is set to 0.01, as this value is considered optimal for perturbation, as described in Ash and Adams (2020). Initially, we tested σ = 0.1 as the perturbation parameter, since Ash and Adams (2020) reported slightly better test accuracy compared to σ = 0.01 in some cases. However, we experienced significantly poorer generalization performance with σ = 0.1 compared to σ = 0.01, as shown in Figure 18 . The hyperparameters used in our experiments are described in Table 7 . In this subsection, we explore how DASH performs in a variety of settings, including state-of-the-art (SoTA) configurations, larger datasets, and scenarios where previous data cannot be stored. We also examine how DASH compares to other methods when data is continuously introduced throughout training, as well as its effectiveness in non-stationary data distribution environments, such as Class Incremental Learning (CIL) setups.In the state-of-the-art (SoTA) setting, we employed weight decay and standard data augmentation techniques, such as horizontal flipping and random cropping. We also used a learning rate scheduler that reduces the learning rate step-wise by a factor of 0.2 at 60, 120, and 200 epochs. By applying the learning rate scheduler, there is no need to compare training time since training is completed at roughly the same epoch across all experiments. The weight decay was set to 0.0005, and the initial learning rate was set to 0.1. All other settings remain the same as mentioned above. We tested this setup on CIFAR-10 and CIFAR-100 using the ResNet-18 architecture.The results in Table 8 show that DASH performs similarly to or slightly worse than starting from random initialization. It appears that this is partly because all hyperparameters are tuned to maximize the performance of cold-starting, to achieve the (close-to-)SoTA test accuracy numbers. Due to the lack of computational resources, we were unable to tune hyperparameters specifically for DASH.Furthermore, we believe this aligns more closely with our theoretical anylsis in Theorem 3.6, as the hyperparameters are tuned to allow the model to learn as many features as possible, making it difficult for DASH to outperform cold-starting.Moreover, we observe that S&P cannot be used in these SoTA settings. We believe this is due to the nature of S&P, which shrinks all weights, while the SoTA setting is likely designed to avoid learning unuseful features, unlike the previous setting. Consequently, it is plausible that retaining learned features is more important than forgetting them, making S&P unsuitable for SoTA settings. Although DASH performs slightly worse than cold-starting, it is conceivable that it is better at retaining features compared to S&P and other warm-starting methods, resulting in better overall performance.The gap between warm-starting and cold-starting has been significantly reduced, likely due to data augmentation techniques and the increase in learning rate when new data is introduced. Data augmentation techniques increase the amount of feature information, allowing warm-starting to learn features that vanilla training (without augmentation) cannot (Shen et al., 2022) . Furthermore, as the learning rate is set to a higher value at the beginning of each new experiment, the model can forget previously memorized data points and escape spurious minima that were difficult to escape from, which is consistent with the findings of Berariu et al. (2021) . Despite these improvements, a gap still exists between warm-starting and cold-starting. We validate the scalability of DASH for larger datasets such as ImageNet-1k. However, conducting experiments for such datasets is challenging, as it would require repeating the training process 50 times until convergence-an extremely time-consuming process. As an alternative, we trained ImageNet-1K on ResNet18 using a setup similar to Figure 2 in Section 3.3. Our setup involved pretraining on 50% of the data before fine-tuning on the complete dataset. We used shrinkage parameter λ = 0.3 for both DASH and S&P. Results shown in Figure 19 demonstrate that DASH achieves superior performance compared to all baseline methods-including cold initialization, warm initialization, and S&P-both in terms of test accuracy and convergence speed. Notably, DASH achieves faster convergence and marginally better test accuracy than S&P, demonstrating its effectiveness even on challenging large-scale datasets.To explore how our algorithm can be applied to a broader range of scenarios further, we consider situations where storing all data is not feasible (e.g., due to memory constraints), rather than accumulating and retaining data. Specifically, we divided the CIFAR-10 dataset into 50 chunks and experimented with ResNet-18, running each experiment on a single chunk before moving on to the next. All other settings remained consistent with the details provided in Section 5.1.We configured our experiments to apply DASH at specific intervals (e.g., every 10, 15, or 20 experiments) instead of after each one. Since there is no previous data available in this scenario, we set α = 0. After the 40th experiment, when the model had sufficiently learned, we stopped applying the shrinking process. We also conducted similar experiments with the S&P method for comparison. It's important to note that this variant is feasible because both DASH and S&P focus on adjusting the model's initialization.We tested DASH and S&P with intervals of 10, 15, and 20 epochs. For both methods, we explored shrinkage parameter (λ) of 0.05, 0.1, and 0.3. We plotted the results using the best hyperparameters for each method in Figure 20 . Notably, DASH's test accuracy consistently exceeded that of S&P across all hyperparameter configurations, as shown in Figure 21 . These findings demonstrate that DASH outperforms both the warm-starting baseline and the S&P method in terms of test accuracy.Based on this evidence, we can conclude that DASH is well-suited for data-discarding scenarios. Figure 20 : Performance comparison between warm-starting, S&P, and DASH in a scenario without access to previous data, using optimal hyperparameters for each method. For S&P, the shrinkage parameter λ = 0.3 with shrinkage interval 20. For DASH, the shrinkage parameter λ = 0.3 with shrinkage interval 15. DASH significantly outperforms warm-starting, while S&P performs even worse than the warm-starting baseline in terms of test accuracy. Since there are no previous data available, we observe sharp drops in test accuracy during shrinkage events, but DASH quickly recovers while S&P struggles to regain performance.As shown in Figure 22 , while there is a clear performance gap between warm-starting and cold-starting approaches, both DASH and S&P aim to address this difference. Despite using the same shrinkage value of 0.3, DASH successfully preserved learned features while S&P did not. This difference is reflected in the training accuracy: DASH showed steady improvement as training progressed, while S&P struggled during the shrinking phases. As a result, DASH not only converged faster but also achieved higher test accuracy, while S&P lost the convergence advantage that warm-starting provides.Experiments so far focused exclusively on stationary data distributions. To broaden our understanding, we explored DASH's performance in non-stationary settings, particularly in Class Incremental Learning (CIL) -an important area in continual learning where data distributions shift over time.We began with experiments on CIFAR-10, dividing the dataset into 10 chunks, each representing a distinct class. The model, based on ResNet-18, was introduced to one chunk at the start of each experiment and trained without access to previously encountered data. However, this setup proved challenging; even with warm-starting as a baseline, the model simply overfitted to each new class, resulting in only 10% test accuracy across all experiments, equivalent to random guessing.Given these limitations, we adjusted our experimental setup while preserving the non-stationary nature of the task. Using the same configurations with above, instead of completely discarding previous data, we accumulated data over time. Thus, each new chunk was combined with previously seen data. During evaluation, we tested the model only on classes it had encountered during training.This revised approach yielded more promising results, as shown in Figure 23 . DASH surpasses other baselines in terms of test accuracy despite requiring longer convergence times. While these results demonstrate DASH's potential in certain non-stationary environments, we recognize that our modified setup simplifies true non-stationarity, such as in continual learning scenarios. In this subsection, we will compare the computational and memory overhead of each method. It is important to note that DASH is applied only once when new data is introduced. Since DASH calculates the gradient of the whole dataset just once, its memory complexity is proportional to the batch size and model size. When comparing experiments with the same number of epochs, we can think of DASH as adding approximately one extra epoch to the total training time. Similarly, the computational overhead of DASH is roughly equivalent to running one additional epoch.We provide the experimental results of the computation and memory overhead for each method in Table 9 . The experiments were conducted under the same conditions described in Section 5.1 on CIFAR-10, using ResNet-18 with the SGD optimizer. We provide aggregated results for FLOPS, CPU/CUDA memory usage, and training time on an NVIDIA A6000. These measurements were obtained using the torch.profile library, with values summed across all operations. We only profiled two out of the 50 experiments until convergence (99.9% training accuracy) because profiling adds significant overhead to the training process, and running it continuously slows down the entire training pipeline. A comparison of the total training times of each method for all 50 experiments without profiling is provided in Section C.6. We conducted experiments on two different hardware platforms: Intel Gaudi-v2 HPUs and NVIDIA A6000 GPUs. In this section, we compare their implementation code and training times. Using CIFAR-100 and ResNet-18, we measured the training times on both platforms while maintaining the same experimental settings as described in Section 5.1 with SGD optimizer. It is important to note that no hardware-specific optimizations were performed on either platform, and all hyperparameters were kept constant.To ensure a fair comparison, we ported our NVIDIA implementation to Intel Gaudi-v2, keeping the other settings consistent. The porting process was straightforward, requiring only about three extra lines of code, as shown in the example below. The full Gaudi-v2 implementation can be found in our GitHub repository at https://github.com/NAVER-INTEL-Co-Lab/gaudi-dash. In Table 10 , we present the results of the training time comparison until convergence. The table shows that the Intel Gaudi-v2 is slightly faster than the NVIDIA A6000 across all initialization methods. For instance, the Intel Gaudi-v2 achieves a speedup of 1.16× over the NVIDIA A6000 with cold initialization. Lemma D.3. Suppose we train two models on T 1:j1 and T 1:j2 for some j 1 > j 2 starting from a learned set L and without any memorized data. We use u s and u ′ s to denote features learned in s-th step of the training process by two models trained on T 1:j1 and T 1:j2 , respectively. The i-th learned feature within class c ∈ [C] is denoted as u c,i for the first model and u ′ c,i for the second model. Then, u c,i = u ′ c,i or u ′ c,i = NA for all c ∈ [C] and i ∈ N.Proof of Lemma D.3. Suppose there exists some class c ∈ [C] and i ∈ N such that u c,i ̸ = u ′ c,i and u ′ c,i ̸ = NA. Choose one with the smallest i. Since u ′ c,i ̸ = NA and from our choice of c and i, we havemax v∈Sc\{uc,1,...,uc,i-1} h(v; {u c,1 , . . . , u c,i-1 }) = max v∈Sc\{u ′ c,1 ,...,u ′ c,i-1 } h(v; {u ′ c,1 , . . . , u ′ c,i-1 }) = h(u ′ c,i ; {u ′ c,1 , . . . , u ′ c,i-1 }) ≥ γ j 2 n > γ j 1 n . Hence, u c,i ̸ = NA and h(u c,i ; {u c,1 , . . . , u c,i-1 }) = max v∈Sc\{uc,1,...,uc,i-1} h(v; {u c,1 , . . . , u c,i-1 }).From our Assumption 3.3 and since {u c,1 , . . . , u c,i-1 } = {u ′ c,1 , . . . , u ′ c,i-1 }, we have u c,i = u ′ c,i . This is contradictory and we have our desired conclusion. warm . We may assume that the existence of J * ≥ 2 such that L(1) warm ⊊ L (J * ) warm and choose the smallest J * ≥ 2. Then, in the first step of J * -th experiment, a model learns some feature u. From our training process, u satisfiesn • h(u; G) = |T 1:J * | • g(u; T 1:J * , N (J * ,0) warm ) ≥ γ,and since J * denotes the first experiment that can learn beyond G, L(J * -1) warm = G and |T 1:J * -1 | • g(u; T 1:J * -1 , N (J * -1,0) warm ) = n • h(u; G) ≥ γ.It means that u must have been already learned in the (J * -1)-th experiment and it is contradictory.Thus, we have L | • g(v; T 1:J , N (J,s) cold ) = max v∈S\G Jn • h(v; G) > γ,from the condition in the theorem statement. Hence, we have our conclusion for the test accuracy.For the train time, since the following holds, we conclude T(J) warm < T (J) cold when J ≥ 2: j∈[J] N (j,0) warm = T (J) warm ≤ Jn < nJ(J + 1) 2 = j∈[J] N (j,0) cold = T (J) cold .Theorem 3.6. For any experiment J ≥ 2, the following holds:ACC L (J) cold = ACC L (J) ideal , T (J) warm < T (J) ideal < T (J) coldProof of Theorem 3.6. Recall that the ideal algorithm works by forgetting memorized data points while retaining previously learned features. In other words, at the initial step of the (j + 1)-th experiment, we have L(j+1,0) ideal = L (j) ideal . Additionally, g(v; T 1:j+1 , N (j+1,0) ideal ) = h(v; L (j+1,0) ideal ) holds for all v ∈ S since M (j+1,0) = ∅.We will show that L(J) cold = L (J)ideal holds for all J ≥ 1 by using induction.When J = 1, by applying Lemma D.2, it holds since L(1) cold = L (1) ideal . Suppose L (J-1) cold = L (J-1)ideal for some J ≥ 2 and we will prove that L (J) cold = L (J) ideal . We have the following at the first step of the J-th experiment for all v ∈ S:g(v; T 1:J , N (J,0) ideal ) = h(v; L (J-1)ideal ) For the cold-starting method in the J-th experiment, by Lemma D.2, let s be the step at which the model first finishes learning features in L (J-1) cold . Then, at the (s + 1)-th step for all v ∈ S:g(v; T 1:J , N (J,s) cold ) = h(v; L (J-1) cold ) Since we assumed h(v; L (J-1) cold ) = h(v; L (J-1)ideal ), the cold-starting method starts to behave identically to the ideal method from the (s + 1)-th time step onwards, by Lemma D.2, resulting in L In this section, we provide detailed training algorithms for our proposed learning framework. Algorithm 2 outlines the standard training method within our learning framework. Subsequently, we compare the Cold-starting, Warm-starting, and Ideal methods using the given abstract algorithm in the following algorithms.Require:• L: Set of learned features • T 1:J : Training dataset • γ: Threshold for learning features • τ : Threshold for the number of learned features a data point needs to be considered well-classified 1: Initialize: L (0) ← ∅ M (0) ← ∅ 2: for j in 1 : J do 3: L (j) , M (j) ← TrainingProcess(L (j-1) , M (j-1) , T 1:j , γ, τ ) 4:Initialize: N ← {(x, y) ∈ T : |V(x) ∩ L| < τ ∧ (x, y) / ∈ M} s ← 0 3: while N ̸ = ∅ do 4: s ← s + 1 5: g(v; N ) ← 1 |T | (x,y)∈N 1(v ∈ V(x)) for v ∈ S 6: v s ← arg max u∈S\L g (u; N ) break ties arbitrarily 7: if g(v s ; N ) ≥ γ/ |T | then 8: L ← L ∪ {v s } 9: N ← {(x, y) ∈ N : |V(x) ∩ L| < τ } 10:L (j) ← ∅ 5: M (j) ← ∅ 6: end for 7: return L (j)Algorithm 4 Warm-Starting until J-th Experiment Require:• T 1:J : Training dataset • γ: Threshold for learning features • τ : Threshold for the number of learned features a data point needs to be considered well-classified 1: Initialize: L (0) ← ∅ M (0) ← ∅ 2: for j in 1 : J do 3: L (j) , M (j) ← TrainingProcess(L (j-1) , M (j-1) , T 1:j , γ, τ ) 4: end for 5: return L (j) Algorithm 5 Ideal-Starting until J-th Experiment Require:• T 1:J : Training dataset • γ: Threshold for learning features • τ : Threshold for the number of learned features a data point needs to be considered well-classified 1: Initialize:L (0) ← ∅ M (0) ← ∅ 2: for j in 1 : J do 3: L (j) , M (j) ← TrainingProcess(L (j-1) , M (j-1) , T 1:j , γ, τ ) 4: M (j) ← ∅ 5: end for 6: return L (j)NeurIPS Paper Checklist 


Decision: This paper introduces Direction-Aware SHrinking (DASH), a novel method to mitigate plasticity loss in warm-started neural networks under stationary data distributions (i.e., IID distributions). The approach is well-grounded and supported by experimental results, although these results are only on relatively small, low-resolution, image datasets.

**Strengths:**
- **Novel Contribution:** DASH presents a unique approach to addressing the warm start problem, grounded in a solid theoretical framework.
- **Experimental Validation:** The method shows promising results on benchmark datasets.

**Weaknesses:**
- **Clarity and Theory:** The connection between the theoretical model and the practical implementation of DASH lacks clarity, particularly regarding the role of cosine similarity.
- **Evaluation Scope:** The evaluation is limited to toy datasets, raising concerns about DASH's scalability and performance on larger datasets like ImageNet-1K.
-**Efficiency:** While efficiency is one of the paper's central focuses, it is unclear how efficient the method is. While the number of steps (gradient updates) is reported, timing experiments are not detailed (although some timing information was provided during the rebuttal). Moreover, they only study SGD and SAM, rather than optimizers often used for training deep neural networks in the current era (e.g., Adam, AdamW).
- **Applicability:** The broader applicability of DASH, especially in non-stationary settings like class incremental learning, remains unclear. The authors did provide information during the rebuttal period, showing that their method worked poorly for class incremental learning.

**Recommendation: Accept.** The authors are encouraged to incorporate reviewer feedback and include the issues regarding non-stationary distributions in the limitations section.

Reviews: review: paper_summary: paper_summary: This paper investigates the reasons why warm-starting a neural network by pre-training it on a subset of the full training set leads to suboptimal performance when compared to training it on the full dataset from scratch. In particular, it proposes an abstract combinatorial model of feature learning that according to the authors captures the essence of this phenomenon. In this model, warm-starting leads to overfitting on feature noise that hinders learning new features when fine-tuning on the full distribution when compared to cold-starting. The authors then show that there exist an idealized learning algorithm in this abstract setting that can perform as well as cold-starting by forgeting the memorized noise prior to fine-tuning. Based on this theoretical intuition, they then propose Direction-Aware SHrinking (DASH) a technique that shrinks the learning step per-neuron based on its alignment with previous warm-starting epochs. DASH seems to outperform other techniques in a synthetic warm-starting benchmark.
strength_weakness: strength_weakness: 1. **Unclear details of DASH**: The description of DASH in section 4.5 is not clear and the algorithm listing does not fully explain the algorithm (see questions). This makes it hard to assess the complexity of implementing DASH and its practical relevance.
2. **Limited evaluation of DASH in online learning**: The authors argue that one of the main motivations of studying the decrease of performance under warm-starting is the fact that many networks are trained in an online setting with new data being collected sequentially. However, the only evaluation of DASH is a synthetic warm-starting benchmark that only tests a narrow and unrealistic setting. I am not against this particular benchmark, but I believe the paper would have benefited from further investigations of DASH in a more realistic online setting (including comparing it against strong baselines in that space).
3. **Reasons for improved performance of DASH over cold-starting are not explained**: As the authors highlight, DASH seems to outperform cold-starting in their benchmarks. This phenomenon, however, cannot be explained by their theoretical model and thus it is a weakness of the model itself.
questions: questions: I would appreciate a deeper description of how DASH works:
- Is the running average of gradients computed at every step or every epoch? 
- Is the shrinking performed after every step, every epoch, every experiment?
- Can DASH be combined with any off-the-shelf deep learning optimizer? If so, how?
- What is the memory complexity of DASH?

If these questions are addressed and the explanations are clear and made part of the final manuscript I would be open to increase my score.
limitations: limitations: The authors briefly mention mention some of the limitations of their work (such as the unexplained outperformance of DASH over cold-starting) or specificity of their results to the stationary setting.
paper_summary: paper_summary: Warm-starting neural networks may lead to poor generalization performance or loss of plasticity, likely due to overfitting. This paper presents a framework that hypothesizes that noise memorization is the primary cause of loss of generalization in warm-starting settings. The authors then present an algorithm motivated by their framework that improves the generalization and plasticity of the neural networks. The algorithm works on a per-neuron level and measures the cosine similarity between the negative gradient of the input weight to each neuron and its weights. This cosine similarity is used as a utility measure to allow for more shrinking for the input weights with low cosine similarity. The authors showed the effectiveness of their method on a wide range of problems and compared it against other baselines demonstrating superior performance.
strength_weakness: strength_weakness: - The main problem of this work is that the link between the framework and the algorithm is not clear. For example, it’s unclear why different parts of the algorithm are designed this way, especially the cosine similarity part. Why is the momentum direction assumed to be the direction of features? 
- The evaluation may be unfair and statistically insignificant. In multiple experiments (e.g., figure 1),  the results are generated based on three seeds. This is a very low number to have statistically significant results. Additionally, the reported methods use the same hyperparameter across multiple methods, which may be unfair to the baselines since they are not well-tuned compared to DASH. The results should be based on the best hyperparameter of each method. This might explain why the authors found approaches such as L2 Init, resetting, layer normalization, SAM, or reviving dead neurons to be less effective. In contrast, the only effective one was S&P, which uses a hyperparameter (shrinking) similar to the DASH method.
questions: questions: - In Figure 4, the authors presented an arrow with a feature direction. What does that mean exactly in a rigorous way?
- What is the definition of a feature? The word is used in the paper many times without a proper definition, leading to further confusion. In neural networks, the learner always improves its set of features, so they are always changing.
limitations: limitations: No. I think the main limitation of the method is that the cosine-similarity metric seems to be heuristic-based and not motivated by the introduced framework. It is unclear if such a metric is ideal or if it reflects what is suggested by the theoretical framework.
paper_summary: paper_summary: The paper examines warm-start and the loss of plasticity, identifying that noise memorization from warm-start impairs generalization in stationary data distributions. It proposes a method called DASH, which selectively forgets previously memorized noise while preserving learned features.
strength_weakness: strength_weakness: - **Compute Overhead**: Gradient-based approaches typically incur significant compute overhead. While DASH is claimed to achieve faster convergence, it remains unclear if its shrinking process slows down the overall training. Quantitative comparisons in terms of FLOPs and training time would be insightful. Furthermore, DASH has not been compared with baselines regarding compute and memory overhead. The training convergence based on the gradient norm does not consider the compute and memory overhead associated with DASH’s gradient calculations.
- **Assumption: Gradient norm as a proxy for the number of training steps**: The assumption that the number of training steps can be reliably approximated by the initial gradient norm is insufficiently supported by the data presented in Figure 10 of the paper. The correlation between training steps and the initial gradient norm does not appear to be linear, particularly for the Warm start. 
- **In-depth Analysis**: The experiments do not demonstrate a direct link between learned/memorized data and the cosine similarity of gradients and weights. Conducting ablation studies might provide further insights into this relationship. Additionally, the proposed method appears to be a variant of the shrink and perturb approach.
- **Evaluation and Scalability**: Evaluation is performed exclusively on toy datasets, leaving DASH’s scalability to larger and more challenging datasets, such as ImageNet-1K, unknown.
- **Generality**: In SOTA settings, as shown in Table 7, DASH does not consistently outperform the compared methods, particularly warm initialization. This raises questions about the wider applicability and robustness of DASH. Its efficacy seems to diminish when using augmentations and weight decay.
questions: questions: - Does the apparent non-linearity in the correlation between initial gradient norms and training steps in Figure 10 change how Figure 1(b) should be interpreted?
- In Figure 2, the test accuracy drops with longer pre-training when the model is pre-trained on 50% of the training data and then fine-tuned on the full dataset. Is this phenomenon universal or does it only occur with the CIFAR dataset? Could the authors provide similar results for high-dimensional datasets, such as ImageNet-1K (224x224)? If the network learns the same set of semantic categories during the pre-training and fine-tuning stages, the fine-tuning stage only differs in terms of the number of samples. Is this phenomenon related to the size of the training data? How much do pre-training and fine-tuning datasets differ in terms of distribution shifts? Is the pre-training data class balanced? How does DASH perform in a similar setting?
- Does DASH need access to the entire dataset (all chunks), including current and previous datasets/chunks, for computing gradients and shrinking weights during every training step?
- In *Line 116*, the set `S_y` is undefined, and may be corrected to `S_c`.
- In *Line 134*, `Tau < K` may be corrected to `Tau <= K`, where the number of learned features can also equal the set of learnable class-dependent features, K.
limitations: limitations: The paper provides little to no detail on the limitations of this work. While it mentions the assumptions of a discrete learning process and the lack of consideration for a hypothesis class, it does not address whether DASH can be extended to larger problem settings. If such limitations exist, they should be explicitly stated by the authors.
paper_summary: paper_summary: This paper examines a setting in which a model is learned with an expanding dataset, such that new data is introduced after the model hits a training accuracy threshold. In particular, the new data comes from the same distribution as the existing data (or sometimes has the exact same statistics), making this a stationary learning problem (unlike the non-stationary problems commonly explored in plasticity loss settings). After demonstrating that plasticity loss occurs in this setting, the paper proposes a new method, DASH, to tackle the problem. DASH shrinks weights based on the cosine similarity between the weight vector and the negative gradient of the training loss. There are a series of experiments which generally show DASH outperforming other plasticity loss prevention approaches, with more in the appendix.
strength_weakness: strength_weakness: Before noting weaknesses of the paper, which I generally feel come down to clarity issues. I am open to increasing my score in light of discussion based on the author's response to my highlighted weaknesses and questions.

- I found a lot of the theory quite confusing, and couldn't follow section 3 ('Comparison Between Warm-Start and Cold-Start'). Similarly, I found some explanations about the method, and the training process quite confusing. Perhaps demonstrating with examples could be useful, but it is currently quite hard to parse exactly what the problem setting is.
- This work has not been studied in a particularly standard learning process. Instead, the focus is on a setting where features are learned based on their frequency of appearance in data, rather than their signal strength. This is quite an artificial setting, and it is not exactly clear to me that DASH scales to more conventional learning frameworks.
- Appendix A.3 feels slightly burrowed away, but considers the SOTA setting for CIFAR-10 and CIFAR-100, in which DASH underperforms random initialisation. I think this is quite a crucial result, and arguably outweights the results from Table 1 in which a more naive approach is taken. While it is good that the authors include these results in the appendix, it would be good to try to include this in a more upfront way (perhaps including discussion about how DASH is helpful in a naive setting and helps reduce the barrier to entry with less studied datasets?)
questions: questions: - Considering a different learning setting, where data is continuously added rather than in predictable batches, would DASH still be able to perform (given at the moment, it is applied whenever new data is provided)? Similarly, how is its performance when we are not adding data from a stationary distribution, given this is possibly more realistic?
- How do the authors think DASH could work in the more generic learning problem (i.e., where we are not learning features based on their frequency)?
- What is actually meant by the frequency of features - can you offer an example please? I find it hard to determine exactly what is meant by the frequency of these discrete features?

- In table 8, the authors write CIF1R rather than CIFAR.
- What is meant in line 325: 'This allows weights to easily change its direction.'?
limitations: limitations: - In the appendix, there are examples where DASH does not outperform other methods, or is within the error margin. However, there is clearly a strong signal that DASH on average provides benefit to the training in all but the SOTA setting. The authors are upfront about this.
- I believe the authors are generally upfront about the limitations of their work, though I feel perhaps highlighting that they are working in a slightly artificial setting, and demonstrating how DASH could be implemented to a typical machine learning pipeline, would be useful.

Comments: title: Thank you for your response
Thank you for your rebuttal
comment: Thanks for your response. We acknowledge that the logical flow between the framework and the algorithm is weak in our current draft, which could lead to misunderstandings or make it difficult for readers to follow. To address this, we plan to reorganize the paper as follows:

- Add the main intuition behind our theoretical framework which included in our global response at the beginning of Section 2 to make it easier for readers to understand and follow our framework.
- Move 'Section 4.1 Motivation: An Idealized Method' to the end of Section 3, which mainly addresses the theoretical framework. This will create a clearer separation between the results on the theoretical framework and the real-world scenarios. We believe this change will help prevent any confusion by clearly distinguishing between the discussion of these two distinct settings.
- At the beginning of Section 4, add an additional subsection to provide motivation for DASH by presenting the connection between DASH and the idealized method considered in the theoretical framework. This will help clarify how the theoretical findings relate to our practical proposal, DASH.

We believe this reorganization will improve the paper.

If you have further concerns or comments, we would be happy to hear them.
Thank you for your feedback and for reconsidering the score.

- While we currently don't have the time to fine-tune hyperparameters for DASH in the SOTA setting, we intend to address this in our next revision. In the upcoming update, we will perform more comprehensive hyperparameter tuning, tailored specifically to each method, and incorporate these improved results into our findings.
- Thank you for your additional comments on our feature learning framework. We now understand the concerns you raised in W2 and Q2. As you pointed out, the learning of features can also be influenced by the strength of the features, especially when distinct features vary in strength. Our theoretical results and framework can be directly extended to this scenario by slightly altering our approach to consider $(\text{frequency}) \times \text{(strength)}$ as learning criteria instead of just frequency. Additionally, we believe our analysis can be extended to scenarios where feature strength varies across data by treating the set of features as a multiset, where multiple instances of the same element are allowed. Since the analyses in these cases are nearly identical to the one we have considered and follows the same underlying principles, we have assumed all features have identical strength for notational simplicity, without loss of generality. We hope these discussions adequately adress your concerns. We plan to include these discussions as a remark in the next revision to provide a clearer explanation to readers of our learning framework.

We welcome any further questions or comments you may have.
I thank the authors for their detailed response. After reading the response and other reviews, I think my initial assessment has been confirmed. I think major restructuring is needed to build a logical flow between the framework and the algorithm. While the authors explained the intuition, it still lacks some rigor, and judging a completely rewritten introduction and method sections requires another review process. Therefore, I maintain my original score.
Thank you for your feedback and for reconsidering the score. We are glad to hear that our response addressed your concerns. We would also be happy to hear if you have any additional thoughts or suggestions.

Best regards,

Authors
I thank the authors for answering my questions and providing a thorough response to all reviewers. After reading the other reviews and the authors's responses I have decided to increase my score to 7: Accept as the authors have clarified the main 
 details that were unclear in the implementation of DASH and provided new insightful experiments in a different online learning scenario (I apologize for my confusion in my previous review and acknowledge that their previous setting was also realistic for a data accumulation setup). I would encourage the authors to improve the description of DASH in a potential camera-ready version in any case.
Dear authors,

Firstly, let me apologise for my delay in responding to your rebuttals. I will attempt to remain more active in the coming days to enable active discussion, if necessary.

I still hold a few concerns regarding the performance of DASH:
- It is quite difficult to make claims about the significance of DASH given the performance in the SOTA setting. Despite the author's beliefs that this may improve with hyperparameter tuning, I would need to see this run experimentally to believe this is the case (I respect that you did not have the time to run this experiment over the rebuttal period due to the strict time constraints).
- I am still not certain I believe in even the intuition of the feature learning framework; as I say, this ignores the fact that different signals provide varying levels of signal for learning, and thus the frequency of features is not the only factor that determines learning.

That said, additional results provided in the global response have strengthened your paper in my opinion. I also agree that it would be interesting to see how DASH could be applied to nonstationary settings like RL, and would suggest adding this as a proposal for obvious future work. 

I have raised my score to recommend borderline acceptance.
Dear Reviewer rFKW,

We sincerely appreciate your time in reviewing our work. Your feedback has helped us improve our research. We understand that you may have a busy schedule, but we'd be grateful if you could take a moment to look over our responses and ensure we've addressed your concerns adequately. If you have any additional questions or suggestions, please don't hesitate to let us know. We're ready to address any further points you may have. 

Thank you again for your valuable contribution to our research.
Dear Reviewer t732,

Thank you for your time and effort in reviewing our work. We greatly appreciate your valuable and constructive feedback. Given your busy schedule, we would be grateful if you could review our responses to ensure we've adequately addressed your concerns. If you have any further questions or suggestions, please let us know, and we'll be happy to address them.

Thank you again for your valuable contribution to our research.
Dear Reviewer LBum,

Thank you for taking the time to review our work and for providing such insightful and constructive feedback. We understand that you may have a busy schedule, but we wanted to follow up to ensure that our responses have sufficiently addressed your concerns. If you have any further questions or comments, we would be glad to hear them.
Thank you for your response. We are glad to hear that our explanations were helpful and adequately addressed your concerns. We also appreciate your suggestions for improvement and plan to incorporate them into our next revision.

Best regards,

Authors
I appreciate the authors for providing additional clarifications regarding the IID/CIL settings and the forgetting phenomenon, which overlap in continual learning and the shrinking process. 

The authors have adequately addressed my major concerns, presenting solid contributions toward resolving warm-starting issues and the loss of plasticity. Therefore, I have raised my rating and believe this paper could be accepted for publication.

I suggest the authors clarify these points in the main paper and release the code to ensure reproducibility and transparency.
**References**

[1] Nikishin et al., The Primacy Bias in Deep Reinforcement Learning, ICML 2022

[2] Zhou et al., Fortuitous forgetting in connectionist networks, ICLR 2022

[3] Ash and Adams, On warm-starting neural network training, NeurIPS 2020
### **3-1. Explanation of DASH on Online CIL Setting**

We acknowledge that Class Incremental Learning (CIL) is an important research topic in continual learning where data distribution is non-stationary. Following your suggestion, we conducted experiments to explore DASH's effectiveness in a CIL setting, which contrasts with our original setup.

We divided CIFAR-10 into 10 chunks, each containing data from a single class. In this setting, at the start of each experiment, a chunk with a specific label is introduced, and the model is trained using only this new chunk without access to previous data. We trained CIFAR-10 on ResNet-18 with five random seeds.  Please let us know if you have a different setup in mind.

We first conducted experiments with warm-starting as a baseline in this setting. The results showed that the model overfits the data introduced in each experiment, predicting only the corresponding label, resulting in a test accuracy of around 10% for all experiments. In this online learning scenario, it seems unlikely that any method manipulating the initialization at the beginning of every experiment, including warm-starting, DASH, or S&P, would be effective. In fact, both DASH and S&P performed no better than random guessing, also achieving only 10% test accuracy, simillar to  warm-starting. Instead, we might need to modify the model architecture or employ other training techniques e.g. replay buffer or regularization.

Despite this, it's worth considering why DASH performs as bad as warm-starting. The primary reason is the limited overlap between class-dependent features across different data chunks. If we apply DASH at the beginning of each experiment, even neurons that learned features from previous chunks would have low cosine similarity with the negative gradients of the new chunk. This leads to significant shrinking and forgetting. It's important to note that our algorithm was originally designed for stationary settings. As we mentioned in our response to rFKm, non-stationary cases require different considerations. We view this as an important direction for future work.

### **3-2. CIL with “Data Accumulation” Setup**

The setting described above is not suitable for evaluating DASH's applicability, as even warm-starting performs no better than random guessing. Given that even a warm-started neural network struggles in a standard CIL setting, we conducted additional experiments in a modified CIL environment. Similar to the previous setup, we divided CIFAR-10 into 10 chunks, each containing data from a single class. This time, instead of discarding data from previous experiments, we combined newly introduced data with the existing training data (i.e., the same as the “data accumulation” setup considered in our paper). This approach aligns more closely with our main focus while still maintaining the core concept of non-stationarity you mentioned. During testing, we evaluated the model only on the labels it had encountered during training.

The model trained with warm-starting achieved a final accuracy of $59.55\% (\pm 0.53\%)$ in this setting, with an average test accuracy of $72.39\% (\pm 0.53\%)$ across 10 experiments. In contrast, when applying DASH, we achieved a final accuracy of $83.74\% (\pm 0.45\%)$ with an average test accuracy of $87.86\% (\pm 0.18\%)$. 

These results show that DASH can be effective in certain non-stationary settings, though our modified setup doesn't fully represent a truly non-stationary environment. While we can't entirely explain DASH's strong performance in this context with our abstract theoretical framework (Section 2), we attribute it to two key factors. 

The first factor is our experimental approach. By incorporating data from previous experiments, we allow this information to continue contributing to the learning process. This is why warm-starting tends to yield better test accuracy in this scenario. Another key factor is DASH’s algorithm design. Even when DASH shrinks all weights to their lower bound $\lambda$ because of low cosine similarity, it would likely perform similarly to S&P.

### **4. Regarding IID Setting**

You're correct in noting that we focused on the IID setting. We aim to address why warm-starting underperforms in this IID setting (i.e., the loss of plasticity) and to propose a solution to this issue. Thus, we would like to emphasize that the significance of our research in the IID setting remains important and should not be overlooked.

### **5. Regarding S&P in Figure 3 of the attached PDF**

Due to the computation-intensive nature of the experiments on ImageNet-1K, we couldn’t make a direct comparison with S&P at this time. We appreciate your suggestion and plan to include this comparison in the revised version of our paper.


---

We hope that our response clarifies your concerns. We would appreciate it if you could reconsider your assessment.
Thanks for your response. We are glad to hear that some of your concerns have been addressed. We want to respond to your additional comments regarding our supplementary experiment in the online learning setting.

### **1. Clarification on “Catastrophic Forgetting”**

Before directly answering the questions one at a time, we want to clarify what we meant when discussing the reduction of “catastrophic forgetting” in an IID setting. In our experiment setup, we considered an online learning scenario where all data chunks are drawn from an identical distribution. While catastrophic forgetting is typically less severe in this IID setting as you pointed out, the inability to access previously introduced data in later experiments can still lead to significant forgetting issues. This is particularly problematic with shrinking-based methods, as they intentionally force the model to forget previously learned information through the shrinking process [3].
We considered the inability to retain learned information as a form of catastrophic forgetting. We apologize for any confusion this may have caused. To clarify, when we referred to "catastrophic forgetting" in our previous response to Reviewer LBum, we meant to highlight that shrinking-based methods inherently face this “forgetting” issue. Our point was that applying DASH and S&P for every chunk doesn't actually solve this problem. Instead, we attempt to mitigate the issues associated with shrinking-based methods by applying shrinking with some intervals.

### **2. Degradation of DASH and S&P in Figure 8**

Every point at which degradation of test accuracy occurs in DASH and S&P coincides with the shrinking point of each method. The reason for different shrinking cycles is that we report the best learning curve (in terms of the last test accuracy) out of all $3\times3=9$ hyperparameter configurations (the interval between shrinkages $\in \\{10, 15, 20\\}$ & the shrinkage factor $\lambda \in \\{0.05, 0.1, 0.3\\}$) for each method. The performance drops immediately after applying parameter modification (i.e., resetting or shrinking) and subsequent performance recovery is common in the literature of resetting techniques [1,2]. 

Additionally, in our online learning setting, we only used a newly introduced chunk of data for training at each experiment, rather than training on all previously seen data. This resulted in slower accuracy recovery due to the limited amount of training data. Moreover, when applying shrinking (DASH or S&P) too frequently, the performance degradation caused by shrinking outweighs the learning effect from the limited number of new data points, leading to a decrease in overall test accuracy.
I thank the authors for their detailed responses, clarifications, and additional experiments. Some of my concerns are addressed by the authors.

I have a few more questions and concerns. Although the concern regarding online learning is raised by reviewer LBum, I also need clarification regarding this.

- Regarding Figure 8 in the attached PDF, could you explain why the test accuracy of DASH and S&P sometimes decreases and is less smooth compared to the "warm" method? Additionally, can you confirm that the model is only trained on new data or the new chunk, rather than on all previously seen chunks?

- In your online learning experiments (Figure 8), I assume each chunk contains almost all classes from the CIFAR-10 dataset. I’m interested in understanding whether there is any distribution shift between subsequent chunks. In continual learning, there is a setting called class incremental learning (CIL), where chunks or batches of data are mutually exclusive, meaning classes do not overlap across chunks. For example, if CIFAR-10 is divided into 10 chunks, each chunk would contain only one unique class. If DASH were applied in this setting, how effective would shrinking by cosine similarity be? Could you explain this in terms of learned features and noise?

- In contrast, there’s another setting in continual learning called IID (independent and identically distributed), which is the opposite of CIL. In the IID setting, chunks or batches contain overlapping classes, leading to minimal distribution shift, unlike in CIL where the shift is significant. Consequently, catastrophic forgetting is significant in CIL and negligible in IID. I assume all your experimental settings in the paper and rebuttal are based on IID, is that correct?

- My primary question is: Is DASH only applicable in scenarios where subsequent chunks/batches/experiments contain the same/ overlapping classes (IID setting)? If so, it cannot be claimed that DASH addresses catastrophic forgetting.

It would also be helpful to include S&P in Figure 3 of the attached PDF. I suggest incorporating the additional experiments and results into the paper and clarifying the applicability of DASH in response to my questions.