Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Authors: Sangyeon Park
Isaac Han
Seungwon Oh
KyungJoong Kim
Venue: ICML 2025 poster (2025)
Keywords: loss of plasticity
plasticity
continual learning
reinforcement learning
Primary area: deep_learning
TL;DR: We introduce AID (Activation by Interval-wise Dropout), a novel method inspired by Dropout, designed to address plasticity loss by applying different dropout probabilities across each preactivation interval.

Abstract: Plasticity loss, a critical challenge in neural network training, limits a model's ability to adapt to new tasks or shifts in data distribution. While widely used techniques like L2 regularization and Layer Normalization have proven effective in mitigating this issue, Dropout remains notably ineffective. This paper introduces AID (Activation by Interval-wise Dropout), a novel method inspired by Dropout, designed to address plasticity loss. Unlike Dropout, AID generates subnetworks by applying Dropout with different probabilities on each preactivation interval. Theoretical analysis reveals that AID regularizes the network, promoting behavior analogous to that of deep linear networks, which do not suffer from plasticity loss. We validate the effectiveness of AID in maintaining plasticity across various benchmarks, including continual learning tasks on standard image classification datasets such as CIFAR10, CIFAR100, and TinyImageNet. Furthermore, we show that AID enhances reinforcement learning performance in the Arcade Learning Environment benchmark.

Activation by Interval-wise Dropout A Simple Way to Prevent Neural Networks from Plasticity Loss
Sangyeon Park
Isaac Han
Seungwon Oh
Kyung-Joong Kim
Abstract
X Sps
M pos
X Xpos
M p00
Plasticity loss, critical challenge in neural net- work training, limits a model's ability to adapt to new tasks or shifts data distribution. This pa- per introduces AID (Activation by Interval-wise Dropout), novel method inspired Dropout, de- signed address plasticity loss. Unlike Dropout, AID generates subnetworks by applying Dropout with different probabilities on each preactivation interval. Theoretical analysis reveals that AID reg- ularizes the network, promoting behavior analo- gous to that of deep linear networks, which do not suffer from plasticity loss. We validate the effec- tiveness of AID in maintaining plasticity across various benchmarks, including continual learn- ing tasks on standard image classification datasets such as CIFARIO, CIFAR 100, and TineemageNet. Furthermore, we show that AID enhances rein- forcement learning performance the Arcade Learning Environment benchmark.
2.5
13 14
4.0
Xnog
Mnex
Xxng
M
3.0
0
4.0
Figure I. AID Architecture for Simplified Version. We apply Dropout a rate of 1 p for positive values and p for negative values, where Mies Ber(p) and mngg ~ Ber(1 - p). Unlike ReLU activation, AID allows utilizing negative preactivation and regularizing toward linear network, effectively mitigating plasticity
loss.
the reinforcement learning domain, caused by its inherent non-stationarity (Sokar et al., 2023; Kumar et al., 2020). However, recent study suggests that this phenomenon is prevalent not only in non-stationary domains, but also in stationary domains (Lee et al., 2024b; Shin et al., 2024).
1. Introduction
Several factors have been proposed causes of plasticity loss. Earlier studies have identified issues such dead neu- rons (Sokar et al., 2023) and large weight norms (Lyle et al., 2023) as potential contributors to this phenomenon. More recent research suggests that non-stationary serves as the primary trigger for plasticity loss, subsequently leading to effects like dead neurons, linearized units, and large weight norms, which constrain the network's capability and desta- bilize training (Lyle al., 2024). Other factors have also been implicated, including feature rank (Kumar et al., 2020), the sharpness of the loss landscape (Lyle et al., 2023), and noise memorization (Shin et al., 2024).
Loss of plasticity refers to the phenomenon in which a neural network loses its ability to learn, which has been reported in recent studies (Lyle et al., 2023; Dohare et al., 2021). This phenomenon has been observed in models pre-trained on datasets where labels are assigned randomly or input images are randomly permuted. When trained on a new task, these models exhibit degraded learning capabilities than freshly initialized networks. Loss of plasticity is a critical issue that arises across various domains, making it an essential problem to address. There have been evidence that plasticity loss caused by non-stationarity (Lyle al., 2024). While these results are from supervised domain like continual learning, plasticity loss also has been observed in
Based on these insights, various approaches have been pro- posed to address plasticity loss. These include adding penalty terms regularize weights (Kumar et al., 2023; Gogianu et al., 2021). resetting dead low-utility neu- rons (Dohare et al., 2021; 2023; Sokar et al., 2023), shrink- ing the network to inject randomness (Ash & Adams, 2020; Shin et al., 2024), introducing novel architectures (Lee et al., 2024b) or novel activations(Abbus et al., 2023; Lewandowski et al., 2024), and developing new optimiza-
DDeaartment of AI Convergence, Gwangju Institute of Science and Technology (GIST), Gwangju, South Korea, Correspondence to: Joong Kim <kjkim @ggisttc..r>..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
tion strategies (Elsayed& & Mahmood, 2024).
loss. They demonstrated that plasticity loss also occurs under a stationary distribution, as in warm-start learning scenario where the model is pretrained on a subset of the training data and then fine-tuned on the full dataset.
Widely used methods such L2 regularization and Lay- erNorm (Ba et al., 2016) have proven to be effective in addressing plasticity loss (Lyle et al., 2024). Several studies (Ma et al., 2023; Lee et al., 2024b) have also shown that data augmentation alleviates plasticity loss, suggesting a possible connection between overfitting and plasticity loss. However, these approaches do not tackle the underlying cause of plas- ticity loss. Furthermore, recent study (Dohare et al., 2024) found that Dropout, which is commonly used to prevent overfitting, is not effective in mitigating plasticity loss. This raises the question: why is Dropout ineffective against plas- ticity loss? We first analyze why Dropout fails to address plasticity loss.
Most existing studies have focused on only one of the fol- lowing challenges: trainability, generalizability, or reinforce- ment learning. However, in this study, we validate our AID method across all three aspects, demonstrating its effective- ness in each scenario.
2.2. Activation Function
Our AID method is stochastic approach similar to Dropout while also functioning we aim to discuss previously proposed probabilistic activa- tion functions. Although the field of probabilistic activation functions has not seen extensive research, two noteworthy studies exist. The first is the Randomized ReLU (RReLU) function, introduced the Kaggle NDSB Competition (Xu, 2015). The original ReLU function maps all negative val- ues to zero, whereas RReLU maps negative values linearly based on random slope. During testing, negative values are mapped using the mean of the slope distribution. Their ex- perimental results suggest that RReLU effectively prevents overfitting. Another example of a probabilistic activation function is DropReL.U (Liang et al., 2021). DropRel.U randomly determines whether a node's activation is pro- cessed through a ReLU function linear function. The authors claim that DropReLU improves the generalization performance of neural networks. The fundamental distine- tion between these probabilistic activation functions and our method lies in the generality of our approach. Unlike simple probabilistic activation functions, our method encompasses techniques such as Dropout and ReLU, providing a more comprehensive framework.
activation function. Therefore,
Building on this analysis, we propose new method in- spired by Dropout, called AID (Activation by Interval-wise Dropout), to tackle plasticity loss. Unlike Dropout, AID applies interval-specific Dropout probabilities, making it functions as a nonlinear activation. Through experiments, we demonstrate that AID's masking strategy is highly ef- fective in addressing plasticity loss. Recent studies have shown that plasticity loss does not occur in deep linear networks (Lewandowski et al., 2023; 2024; Dohare et al., 2024). We theoretically prove that AID maintains plasticity by regularizing activations to close to those of a linear net- work. Since AID also functions as an activation function, we theoretically prove its compatibility with He initialization, demonstrating that AID can seamlessly replace the ReLU function.
2. Related Works
2.1. Plasticity in Neural Networks
In recent years, various methods have been proposed to address plasticity loss. Several works have focused on main- taining active units (Abbas et al., 2023; Elsayed & Mah- mood, 2024) or initiaalizing dead units (Sokar et al., 2023; Dohare et al., 2024). Other studies have explored limiting deviations from the initial statisties of model parameters (Kumar et al., 2023; Lewandowski et al., 2023; Elsayed et al., 2024). Additionally, some methods rely on architec- tural modifications (Nikishin et al., 2024; Lee et al., 2024b; Lewandowski et al., 2024). Plasticity loss also occurs in the reinforcement learning due to its inherent non-stationary. Nikishin et al. (2022) proposed resetting the model, while www.eeuuggggeo
Another related approach involves activation functions de- signed to address plasticity loss. (Abbas et al., 2023) pro- posed the Concatenated Rectified Linear Units (CReLUU, which concatenates the outputs of the standard ReLU ap- plied to the input and its negation. This structure prevents the occurrence of dead units, thereby improving plasticity. Additionally, trainable activation functions have also been shown to effectively mitigate plasticity loss in reinforce- ment learning (Delfosse et al., 2024). Specifically, they introduced a trainable rational activation function that pre- vents value overfitting and overestimation in reinforcement learning.
Asadi et al. (2024) suggested resetting the optimizer state.
As noted by Berariu et al. (2021), loss of plasticity can be divided into two distinct aspects: a decreased ability of networks to minimize training loss on new data (trainabil- ity) and a decreased ability to generalize to unseen data (generalizability). While most previous works focused on trainability, Lee et al. (2024b) addressed generalizability
3. Why is Dropout Ineffective for Maintaining Plasticity?
In this section, we investigate the ineffectiveness of Dropout in mitigating plasticity loss. Dropout operates by randomly
2

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Trainability
Subnetwork trainability
Generalizability
1.0
1.0
0.65
Vanilla
0.8-
0.8
Test Accuracy
0.60
Drapost
ATD
Accuracy
0.6-
0.4
Dropest 1
Dropazipea
Umograll pm)) 2
Uropgal pmn)
Emogial pm0
Accuracy
0.4
0.6
Despout pe0.2
Tropert p-
Bropert y-6
Dripert
0.55
0.50
Cald-itatt
Warn Start
0.45
0.2
0.2
0.40
40
40
100
Number of tasks
Number of tasks
Epach
Figure 2. Left. Random label MNIST experiment using an 8-layer MLP. Higher dropout probabilities result in significant trainability loss. Middle. Accuracy of the subnetworks trained on random target. Each subnetworks are sampled from original network after each epoch. Subnetworks of the Dropout also experience trainability loss. Right. Warm-start scenario of Resnet-18 18 model with CIFAR 100 datasct. Dropout improves generalization performance; however, the reduction in accuracy compared to the cold-start scenario is nearly identical to that of the vanilla model.
deactivating specific nodes during training, effectively gen- erating random subnetworks. This mechanism fosters ensemble effect, where the network learns more generalized features across these subnetworks. However, our hypothesis suggests that this process does not address plasticity loss:
ability degradation, reinforcing our hypothesis that Dropout creates multiple subnetworks, each of which suffers from plasticity loss. In contrast, our proposed method, AID, effec- tively maintained trainability across all tasks, demonstrating its robustness against plasticity loss.
"Dropout merely creates multiple subnetworks, all of which suffer from plasticity loss."
3.2. Generalizability Perspective
To evaluate generalizability, we conducted a warm-start learning experiment inspired by Lee et al. (2024b). Specif- ically, we pre-trained a RESNET-18 model on 10% of the training data for 1,000 epochs before continuing training the full dataset. We repeated this experiment for mod- els trained with vanilla settings, Dropout, and AID. The results are shown in Figure 2 (right). Dropout appeared to improve overall performance, with both warm-start and cold-start models showing better accuracy compared to the vanilla setting. However, we argue that this improvement arises from enhanced model generalization rather than im- proved generalizability. Dropout fails to reduce the accuracy gap between warm-start and cold-start training (10.1 %%)) as vanilla setting does (10.9%p). If Dropout were primarily ad- dressing plasticity loss, we would expect disproportionately higher performance improvements in warm-start models, as cold-start models inherently maintain perfect plasticity. In contrast, the performance gap with AID was significantly smaller (3.3%p compared to the vanilla model's 10.9%p). This observation suggests that AID effectively mitigates plasticity loss, as warm-start models trained with AID retain higher degree of plasticity compared to those trained with Dropout. Extended results regarding generalizability can be found in Appendix G.2.1.
In other words, because each subnetwork independently experiences plasticity loss, the combined treated as an ensemble- also suffers from degraded plasticity.
To validate this hypothesis, we conducted experiments on two forms of plasticity loss: trainability and generalizability. Details of the experimental configurations are provided in Appendix F.1.
3.1. Trainability Perspective
To examine the relationship between Dropout and trainabil- ity, we trained 8-layer MIP on the randomly labeled MNIST dataset. For each task, labels were shuffled, and the network was trained for 100 epochs. Following prior studies (Lyle et al., 2023; Kumar et al., 2023), trainability degradation was quantified as the accuracy drop relative to the first task. Figure 2 (left) demonstrates that higher Dropout probabilities lead more pronounced trainability degradation. This observation aligns with prior findings that Dropout exacerbates plasticity loss (Dohare et al., 2024). As Dropout probabilities increase, the network is divided into progressively smaller subnetworks, reducing the ex- pected number of parameters available for learning. This is consistent with the results of Lyle et al. (2023), which show that smaller networks are more vulnerable to plasticity loss. To further explore this phenomenon, we measured the trainability of individual subnetworks. At the end of each task, we sampled 10 subnetworks and trained them new tasks. As shown in Figure 2 (middle), subnetworks with higher Dropout probabilities exhibited more severe train-
4. Method
4.1. Notations
We denote r(-) to be ReLU (Rectified Linear Unit) activa- tion function, which means that r(x) max(x,, 0) for input
3

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
data x. For ease of notation, define a negative ReLU, F(x: min(x, 0). Furthermore, for a real number > 0, we define modified leaky ReLU, To function that scales the positive part of the input by a, and the negative part by
Algorithm 1 Activation by Interval-wise Dropout (AID)
Given: Predefined intervals {41, 12...., Jah. Dropout
probabilities (p1.P2..... .Ps}
Input: Preactivation values x E TR."
1 - a, formally given as ra (2) = _ + (a- - [x] (e.g- ri (2) = r(z), To(2) = F(x). and (1)2(2) = x). Note that, ro differs from leaky ReLU, since its slope on the positive side may not be 1.
Output: Postactivation values y E R" if training phase then Initialize y I2" as empty vector for each z; E x do Identify interval 15 such that x; E 1 Initialize binary variable m ~ Bernoulli(1 - Pp))
4.2. Activation by Interval-wise Dropout (AID)
Apply mask: y: = J; m end for else if test phase then Initialize y I2 as empty vector for each z; E x do
Activation by Interval-wise Dropout (AID) applies differ- ent Dropout probabilities to distinct intervals of preactiva- tion values, functioning as non-linear activation. Specifi- cally, given & predefined intervals and their corresponding Dropout probabilities {PI. PP......... }. distinct Dropout masks. Each mask is applied only to the values falling within its corresponding interval, while other intervals are unaffected by that mask. Formally, let the pre- defined intervals be {41 42.... _ where I = [,,, uy) for j ..2,...., ks The union of these intervals covers the entire range of real values, i.e., U I R. Moreover, the intervals are disjoint, meaning there is no overlap between any two intervals, i.e., 4; n = 0 for all i # j. AID op- erates as a non-linear activation function when the Dropout probabilities vary across intervals, i.e., when p. * Py for some # j. If all Dropout probabilities are identical = pk), AID reduces to a standard Dropout mechanism. The output of the AID mechanism for an input vector x is given by: (xD = x ④ m. Where m is the Dropout mask vector defined as: m, = 0 with probability Pj if I; E /j. and m. = 1 otherwise. This interval-wise Dropout mechanism enables the model to handle varying ranges of preactivation values, effectively functioning as nonlinear activation function while applying stochastic regularization with different probabilities across intervals.
AID generates k
Identify interval I, such that v, E 1/ Apply scaling: yi = 24 . (1 - Pt) end for end if return
range of intervals, as well as the Dropout probabilities. To reduce the hyperparameter search space and improve computational efficiency, we propose the Simplified AID, Specifically, we define p,)) as a function that applies Dropout rate of 1 to positive and p to negative values. This definition is intuitive since ATD, behaves like ReLU when 1 and a linear network when p = 0.5. Addition- ally, we will demonstrate that this simplified version has an effect that regularizes to linear network and an advantage when He initialization is applied, as discussed in Section 4.4. Next, we explore property of ATD, that useful for its practical implementation.
(p1 = p2
Property 1. Applying A.D, is equivalent to applying r(t) with probability p. and F(-) with probability 1 1-p.
During the testing phase of traditional Dropout, each weight is scaled by the probability 1 1-2 p. which is the chance that a node is not dropped. Similarly, in AID, as each interval has a unique Dropout probability, the preactivation values within each interval I are scaled by 1 1-22 Py for testing. The pseudo-code for AID is presented in Algorithm 1.
We provide a brief proof in Appendix A. Using Property 1, the implementation of AID becomes straightforward. An im- portant consideration is whether scaling up is performed dur- ing training. Unlike the inverted implementation of Dropout, which behaves the identity function during the test phase, AID is intended to operate as activation function and thus should retain its nonlincarity at test time at test time. To ensure this, we adopt the standard implementation, follow- ing the approach of Srivastava et al. (2014). Specifically, AID utilizes (.) (=)p which is the same as passing the average values from the training phase. To clarify the pro- cess, we provide the pseudo-code for the AID activation in Algorithm 2.
AID generalizes concepts from ReLU, standard Dropout, and DropReLU, We provide a detailed discussion of these relationships in Appendix C.I. Since defining each interval and its corresponding Dropout probability requires explor- ing a vast hyperparameter space, Algorithm demands extensive hyperparameter tuning. To address this, we intro- duce a simplified version of AID in the next section.
Algorithm 2 demonstrates that AID can be easily applied by replacing the commonly used ReLU activation. More- over, it shares the same computational complexity as stan- dard Dropout. These properties enable AID to integrate
4.3. Simplified Version of AID
Section 4.2 shows that this methodology offers infinitely many possible configurations depending on the number and
4

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Algorithm 2 Simplified Version of AID
the approach taken by Liang al. (2021). Theorem 4.1 shows that the lower bound of Equation (1)-the the loss under AID-can be decomposed into two terms: the loss under modified leaky ReLU (Lp), and regularization term (Rp) that encourages linearity in the network. As the hyperparam- eter p approaches 0.5, AID imposes stronger regularization toward linear behavior. Prior work (Lewandowski et al., 2024) has shown that linear networks do not suffer from trainability issues, suggesting that the linearity-induccin effect of AID directly contributes to mitigating plasticity loss, On the other hand, simply adding a Dropout layer with ReLU does not yield the same effect, and the corresponding lemma is provided in Appendix C.2.
Given: ReLU r, negative ReLU F, modified leaky ReLU Ta and one vector R
Input: Preactivation values x E R,", coefficient p
Output: Postactivation values y E IR"
if training phase then
M + Generate binary masks from Bernoulli(p) y + M@ r(x)= + (1 - M) @ f(x)
else test phase then
y +- (x)
end if
return y
Since AID functions as an activation, is important to ex- amine its compatibility with existing network initialization methods designed for activations. Therefore, AID ensures the same stability as the ReL.U function when used with Kaiming He initialization. The following property illustrates this strength:
seamlessly with various algorithms and model architectures, regardless of their structure. For reproducibility, we pro- vide the Py PyTorhh implementation of the AID module in Appendix H. Throughout this paper, AID is implemented as described in Algorithm 2, unless specified otherwise. In the next section, we study the factors that contribute to the effectiveness of AID.
show that
Property 2. Replacing ReLU activation with AI ensures the same stability during the early stages of training when using Kaiming He initialization.
4.4. Theoretical Analysis
Property guarantees that the output and gradient will not explode or vanish, thereby supporting the learning stability of the network during its initial training. The outline of the explanation builds on He et al. (2015)'s formulation, utiliz- ing the fact that replacing ReLU with AID does not alter the progression of the equations. A detailed explanation is provided in Appendix D.
Recent studies (Dohare et al., 2024; Lewandowski et al., 2024) have found that linear networks do not suffer from plasticity loss and provided theoretical insights into this phenomenon. This highlights that the properties of linear networks play a crucial role in resolving the issue of plastic- ity loss.
From an intuitive perspective, AID shares properties with deep linear networks in that, it provides the possibility of linearly passing all preactivation values and having gradients of 1. This behavior contrasts with activations such as ReLU, RReL.U, and leaky ReL.U, which tend to pass fewer or no values for negative preactivation. Building on this intuition, the following theoretical analysis establishes that AID as regularizer, effectively constraining the network to exhibit properties of linear network.
5. Experiments
In this section, we empirically evaluate the effectiveness of the proposed method across various tasks. First, we compare AID with previous methods in a range of continual learning tasks to assess its impact on mitigating plasticity loss in Section 5.1. Second, we investigate whether AID provides tangible benefits in reinforcement learning in Section 5.2. Finally, we demonstrate that AID improves generalization in standard classification tasks in Section 5.3.
Theorem 4.1. For a 2-layer network, AID has a regular- ization effect that constrains the model behave like lin- ear network. Formally, let weight matrices be W1, W₂ e Baxx" the input and target vectors be x,y E IR",, and the coefficient of AID be p. Then, for learnable parameter #:
5.1. Continual Learning
We compare AID with various methods that were proposed to address the challenges of maintaining plasticity. In addi- tion, Randomized ReLU and DropReLU are included due to their structural similarity our method. Refer to Appendix E for the description of each baseline.
LAID((0) = [[[[WAIID, (Wix) -
(0) = ||2rr (W -yl1
(0) = [2(, 1 Wix) - -Werp
(W (2r,(Wxx)]B
(0) > L (0) + 4p(1 - p) (2pp 1)2 (0).
Regularization: Dropout (Srivastava et al., 2014), L2 regularization (L.2) (Krogh & Hertz, 1991) and L2 regularization to initial value (L2 Init) (Kumar et al., 2023)
We provide the detailed proof in Appendix B, following
5

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Vanilla
Full Reset
Fourter
CReLU
1.0
1,0
1.0
0.7
0.7
0.7
0.7
DropReLU
ReDo
CBP
S&P
1.0
1.0
0.7
0.7
1.2
1.2 Init
Dropout
AID (Ours)
1.0
1.0
1.0
1.0
Myy
0.7
0.7
0.7
0.7
PermutedMNIST SCD
RandomLabelMMIIST SCD
PermutedMNIST Adam
Randoml abeloNSST Adam
Figure 3. Results for Trainability Experiments. We plot the train accuracy comparisons across different optimizers on Permuted MNIST and Random Label MNIST for each method, with the x a-axis representing tasks. For visibility, we did not show the region using standard deviation. Notably, AID consistently achieves high accuracy across all conditions, demonstrating its robustness in maintaining trainability.
Re-initialization: Shrink & Perturb (S&P) (Ash & Adams, 2020), Recycling Dormant neurons (ReDo)
accuracy of vanilla model gradually declines for both SGD and Adam optimizers. In contrast, methods specifically de- signed address plasticity loss-such as Fourier, CReLU, ReDo, and CBP demonstratee improved plasticity under most conditions. However, these methods fail to retain ac- curacy in certain conditions. particular, S&P and L2 Init often struggle to maintain plasticity when trained with low learning rate (see Appendix G.1.1), suggesting that their effectiveness may be sensitive learning rate choices. No- tably, Dropout fails to mitigate plasticity loss in most cases compared to the vanilla model, supporting the results of Dohare et al. (2024). In contrast, DropReLU and AID con- sistently achieve high accuracy across all settings, which can attributed to the regularizing effect of the linear network, as discussed in Section 4.4. We provide detailed experimen- tal results with various learning rates and methods, as well additional metrics relevant to plasticity dormant neuron rate, effective rank, and average unit sign entropy-for for Vanilla, Dropout, and AID in Appendix G.I.
(Sokar et al., 2023) and Continual Backprop (CBP)
(Dohare et al., 2024)
Activation function: Concatenated ReLU (CReLU) (Abbas et al., 2023), Randomized ReLU (RReLU) (Xu, 2), deep Fourier features (Fourier) (Lewandowski et al., 2024), and DropReLU (Liang et al., 2021)
5.1.1. TRAINABILITY
We start by validating the AID's ability to maintain train- ability. Lee et al. (2024a) categorized the trainability of neural networks into two cases: Input trainability and Label trainability'. Input trainability denotes the adaptability to changing of input data, and label trainability denotes the adaptability to evolving ontut- relationships. Under the concept, we conducted permuted MNIST experiment for input plasticity by randomly permuting the input data and random label MNIST experiment for label plasticity by shuffling the labels at each task. Detailed experimental settings are shown in Appendix F.2.
-nclluiing
1.2. GENERALIZABILITY
In our earlier experiments, we confirmed that AID effec- tively maintains ability to adapt new data. However, while adaptability is important, it is even that the model retain the ability generalize well to un- seen data. Therefore, generalizability is critical aspect related to plasticity loss. We evaluated the generalizabil- ity of AID across three benchmark settings: continual-full, continual- and class-incremental. For this evalu-
Figure shows the training accuracy for each method, cat- egorized by optimizer and experimental settings. We used learning rates of 3e-2 for SGD and 1e-3 for Adam opti- mizer. The results reveal several key findings. First, training
critical to ensure
In Lee et al. (2024a), the terms input plasticity and label plasticity were originally used. For consistency with study, instead adopt the terms input trainability and label trainability.
6

---

Figure 5 presents the difference in test accuracy between Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
CIFAR1O (CNN)
CIFAR1 00 (Resnet-18)
TinylmageNet (VGG-1 6)
0.6
Test Accuracy
0.7
Test Accuracy
0.5
Test Accuracy
0.4
0.3
0.5
0.4
0.0
0
100
200
300
100
500
600 700
800
9001000
100
200
300
400
500
600 700 800 9001000
100
200
300
400
500
600
700
800
9001000
Epoch
Epoch
Epoch
CIFAR1O (CNN)
CIFAR1 00 (Resnet-18)
TinylmageNet (VGG-16)
0.5
Test Ancuracy
0.65
Test Test Accuracy
Test Accuracy
0.3
0.60
0.2
0.55
0.3
0.1
0.45
0.1
0.0
9001000
100
300
9001000
100
200
300
400
500
600 9001000
Epoch
Epoch
Epoch
Vanilla
S-EWC
S&I
RReLU
Pourier DrupReLU
AID (Ours)
Full Reset
Dropout
CBP
DASH
CReLU
Figure 4. Results on Continual Full & Limited Settings. The dataset was divided into 10 chunks, with models trained over 10 stages of 100 epochs each. The settings are distinguished by whether access previously seen data is allowed (top row, continual full) not (bottom row, continual limited). AID consistently outperforms baseline methods, highlighting its effectiveness in mitigating plasticity loss across both settings.
ation, we utilized three datasets: CIFARIO, CIFAR100 (Krizhevsky al., 2009), and TinylmageNet (Le & Yang, 2015).
pending on the specific task and dataset. However, AID demonstrates a noticeable performance advantage over other methods throughout the entire training process. This result highlights that plasticity can be maintained without the need to re- re-initialize certain layers or neurons, which carries the risk of losing previously learned knowledge. Notably, this trend persists even in the continual limited setting. Interest- ingly, methods belonging to the Dropout family (Dropout, DropReLU, and AID) also exhibit strong performance in this setting, aligning with previous findings that suggest Dropout can help mitigate catastrophic forgetting (Mirzadeh et al., 2020). We evaluated the sensitivity of AID to its hyperparameter in the continual full setting with results confirming its robustness provided Appendix G.2.2.
Continual Full & Limited We investigated two contin- ual learning settings inspired by previous works (Lee et al., 2024b; Shen et al., 2024). In the continual full setting, an extended version the warm-start framework, training progress is divided into 10 phases, with 10% of the dataset introduced at each phase while maintaining access all previously seen data throughout training. In contrast, the continual limited setting imposes stricter constraints by re- stricting access to past data, reflecting practical challenges such as privacy concerns and storage limitations. In our experiments, we exclude L2 Init and ReDo, as they have been shown to have no improvement on generalizability, and instead include the recently proposed Direction-Awart SHrinking (DASH) (Shin et al., 2024) as baseline. Addi- tionally, incorporate Streaming Elastic Weight Consoli- dation (S-FWC) (Kirkpatrick et al., 2017; Elsayed & Mah- mood, 2024) on continual limited, which has demonstrated effectiveness in mitigating forgetting under constrained set- tings.
addition to these continual learning settings, we also conducted experiments on class-incremental Learning, inspired by previous works (Lewandowski et al., 2024; Dohare al., 2024). Unlike the warm-start framework, this approach partitions the training process into 20 phases based on class labels, incrementally introducing new classes at each phase. Like continual full setting, the model trained on all available data, including previously introduced class data. Further details regarding the experimental setup can be found Appendix F.3.
Class-nncremental
The results of the Continual setting are presented in Figure 4. Several methods- Suchhods-uuchaao DASH and achieve comparable or even superior performance to full reset, de-
7

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
CIPAR100 (Resee--8))
Finylaageee (VGG-111 (VGG-16)
1QM Normalized Score
0.5
Relative Ancuracy
to Full Reset (%)
Vanilla
0.4
Drepout
AID
-10
-13
0.3
0.2
Number Classes
Number Classes
0.1
Vandla
12
CaP
DASSI
Chtaali
DeopoaLL AID (Ourst
Pull Reset
Dropout
S&P
RRe1
Fourtee
0.0
2
6
10
Figure 5. Results on Incremnncellttall Setting, The tigure shows the relative accuracy compared to a model trained from scratch. AID consistently outperforms the full reset approach, maintaining its advantage throughout the entire training process.
Number of Frames (in millions)
Figure 6. Results on Reinforcement Learning. We train DQN model with replay ratio of I. The plot presents the IQM Human Normalized Score 17 Atari games, with shaded regions repre- senting the 95% confidence intervals across 5 random seeds. The results indicate that AID significantly improves sample efficiency compared to the vanilla model.
model trained with full reset as each class is introduced. Methods- succ as Dropout and DASH-initially demon- strate an accuracy advantage when full reset is applied dur- ing training. However, as training progresses, this advantage diminishes, highlighting their limitations in long-term plas- ticity retention. In contrast, AID stands out the only method that consistently maintains its advantage over full reset throughout the training process. These results suggest that AID effectively mitigates plasticity loss across various experiments evaluating generalizability, reinforcing its ro- bustness in continual learning scenarios. Additional figures showing overall test accuracy are provided in Appendix G.3.
to several factors (Hausknecht & Wagener, 2022). Addi- tionally, as demonstrated in Section 5.1.1, its limitation in mitigating plasticity loss in non-stationary environments may further explain this issue. We note that the slight per- formance improvement observed with a small Dropout rate appears to result from the ensemble effect of Dropout, rather than from mitigation of plasticity loss. In contrast, AID achieves performance gains by effectively addressing plas- ticity loss itself. Further discussion and additional results on this point are provided in Appendix G.4.
5.2. Reinforcement Learning
A high replay ratio is known to offer high sample efficiency in reinforcement learning. However, increasing the replay ratio often leads to overfitting to early data and results in plasticity loss, as highlighted in previous studies (Kumar et al., 2020; Nikishin et al., 2022). In this section, we demonstrate that using AID can effectively mitigate these challenges by maintaining plasticity while benefiting from high sample efficiency.
5.3. Standard Supervised Learning
The preceding experiments have demonstrated that AID effectively addresses the loss of plasticity and exhibits its effectiveness in non-stationary problems. In this section, we investigate whether AID also provides advantages in classical deep learning scenarios without non-stationarity.
For comparison, utilized L2 regularization and Dropout, which are commonly employed in supervised learning, as baselines. The model was trained for a total of 200 epochs using the Adam optimizer with a learning rate of 0.001. The learning rate was reduced by a factor of 10 at the 100th and 150th epochs. The final test accuracy is shown in Table 1.
The experimental setup is as follows: Unlike standard DQN (Mnih al., 2015) use RR- 0.25, train with RR = on several Atari (Bellemare al., 2013) tasks, using the CleanRL environment (Huang et al., 2022). We use the 17 games to train and hyperparameter settings from Sokar et al. (2023). Due to the high computational cost associated with a high RR, we followed the settings of Sokar et al. (2023); Elsayed et al. (2024), where the models are trained for 10 millions frames.
Table 1. Results on Generalization with Learning Rate Decay. The final test accuracy after training 200 epachs. The learning rate initialized with 0.001 and divided by 10 at epoch 100 and 150.
The IQM Human Normalized Score results for the 17 Atari games are presented in Figure 6, with shading indicat- ing stratified bootstrap 95% confidence intervals. The re- sults demonstrate that AID significantly improves sample efficiency compared to the vanilla model. Traditionally, Dropout has been rarely used in reinforcement learning due
CIFARIO (CNN)
CIFAR 100
TinyImageNer (VGG-16)
Method
(Resnet-18) 18)
Vanilla
0.660 1: 0.0058
0.581 1: 0.0029
0.430 1: 0.0009
L2 Dropout
0.742 # 0.0038
0.374 + 0.0017
0.417 + 0.00-1
0.733 1 0.0037
0.651 1 0.0028
0.466 1 0.0040
AID
0.748 0.0021
0.690 + 0.0014
0.519 + 0.0043
8

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
The results presented in Table demonstrate that AID is effective not only in addressing plasticity loss but also in im- proving generalization performance in classical supervised learning settings. Interestingly, AID reduces the generaliza- tion gap more effectively than standard methods commonly used to addressing overfitting. We report the corresponding learning curves in Appendix G.5.
Impact Statement
The proposed method, Activation by Interval-wise Dropout (AID), improves neural network adaptability by mitigating plasticity loss-a critical challenge in continual and rein- forcement learning. This enhancement enables better gener- alization in dynamic environments, making AID applicable across both domains. These advancements contribute to more robust Al systems capable of efficiently handling real- world tasks, such as autonomous systems and personalized user experiences.
6. Conclusion
In this paper, we analyzed why Dropout fails to maintain plasticity. From this observation, we proposed AID (Acti- vation by Interval-wise Dropout) and established through theoretical analysis that acts as a regularizer by penaliz- ing the difference between nonlinear and linear networks. Through extensive experiments, we demonstrated the ef- fectiveness of AID across various continual learning tasks, showing its superiority in maintaining plasticity and en- abling better performance over existing methods. Moreover, we validated its versatility by showing that AID performs well even in classical deep learning tasks, highlighting its generalizability across diverse scenarios.
References
Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, pp. 220-636. PMLR, 2023.
Asadi, K., Fakoor, R., and Sabach, S. Resetting the opti- mizer in deep rl: An empirical study. Advances in Neural Information Processing Systems, 36, 2024.
Despite its effectiveness, several limitations remain in the current implementation of AID. Our experiments primarily focused on configurations with a simplified version, leaving the potential impact of more complex configurations under- explored. While we have provided a theoretical explanation for the high trainability of AID, its relationship to the ob- served improvements in generalizability remains uncertain. Future work will address this limitation by optimizing AID's scalability, exploring diverse configurations and adapting it to new architectures. Additionally, investigating the con- nection between trainability and generalizability would be promising direction for future research.
Ash, and Adams, R.
On warm-starting neural network training. Advances in neural information processing sys- lems, 33:3884-3894 2020.
Ba, J. L.., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv: 160:: 607,06450, 2016.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation plat- form for general agents. Journal of Artificial Intelligence Research, 47:253-279 2013.
Berariu, T., Czarnecki, W., De, S., Bomnsceen, J., Smith, S., Pascanu, R., and Clopath, C. A study on the plasticity of neural networks. arXiv preprint arXiv:2 106.00042, 2021.
Acknowledgements
This work supported by Institute of Information & com- munications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) 2019-00--0-011 0--2 Artificial Intelligence Graduate School Program (GIST)).
Delfosse, Q.. Schramowski, P., Mundt, M. Molina, A., and Kersting, K. Adaptive rational activations to boost deep reinforcement learning. In ICLR, 2024.
Dohare, S., Sutton, R. S., and Mahmood, A. R. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint ar 100.0625, 2021.
This work was supported by the GIST-MIT Research Col- laboration grant funded by the GIST in 2025.
We appreciate the high-performance GPU computing sup- port of HPC-AI Open Infrastructure via GIST SCENT.
Dohare, S., Hernandez--Gariia, J. F., Rahman, P., Mahmood, A. R., and Sutton, R. S. Maintaining plasticity in deep continual learning. arXiv preprint 230.v2230..1381. 2023.
This research was financially supported by the Ministry of Trade, Industry and MOTTE) and Korea Institute for Advancement of Technology(KIAT) through the "Interna- tional Cooperative R&D program" (Grant No. PO028435)
Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature 632(8026):768-7 2024.
9

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Elsayed, M. and Mahmood, A. R. Addressing loss of plas- ticity and catastrophic forgetting in continual learning. arXiv preprint arXiv:2404.00781, 2024.
Kumar, S., Marklund, H., and Van Roy, B. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308. 11958, 2023.
Elsayed, M.,
Lan, Q Lyle, C., and Mahmood, A. R. Weight clipping for deep continual and reinforcement learning. arXiv preprini ar arXiv:2407.011 2000000777.74 2024.
Le, Y. and Yang. X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., Yun, S.-Y., and Yun, C. Plastic: Improving input and la- bel plasticity for sample efficient reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024a.
Gogianu, F., Berariu, T., Rosca, M. C., Clopath, C., Busoniu, L., and Pascanu, R. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, pp. 3734- 3744. PMLR, 2021.
Hausknecht, M. and Wagener, N. Consistent dropout for policy gradient reinforcement learning, arXiv preprint arXiv: 1122222.11111 11818, 2022.
Lee, H., Cho, H., Kim, H., Kim, D., Min, D., Choo, J., and Lyle, C. Slow and steady wins the race: Maintain- ing plasticity with hare and tortoise networks. In Inter- national Conference on Machine Learning, pp. 26416 26438. PMLR, 2024b.
He, K., Zhang. X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE inter- national conference on computer vision, pp. 1026-1034, 2015.
Lewandowski, A., Tanaka, H., Schuurmans, D., and Machado, M. C. Curvature explains loss of plasticity, 2023.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Lewandowski, A., Schuurmans, D., and Machado, M. C. Plastic learning with deep fourier features. arXiv preprint arXiv:2410.2066 2024.
Liang, S., Khoo, Y., and Yang, H. Drop-activation: implicit parameter reduction and harmonious regularization. Com- munications on Applied Mathematics and Computation, 3:293-311,
Huang. S., Dossa, F. Ye, C.,, Braga, J., Chakraborty, D., Mehta, K., and AraAšjo, J. G. Cleanrl: High-quality single-file file implementations of deep reinforcement learn- ing algorithms. Journal of Machine Learning Research, 13 2022.
Lyle, C., Rowland, M., and Dabney, W. Understanding and preventing capacity loss in reinforcement learning, arXiv preprint arXiv:2204.09566 2022.
Huber, P. J. Robust estimation of location parameter. In Breakthroughs in statistics: Methodology and distribu- fion, pp. 492-518. Springer, 1992.
Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural net- works. International Conference on Machine Learning, pp. 23190-23211, PMLR, 2023.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Baabhka-arrwiskka A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 11((13)352113355
Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pas- canu, R., Martens, J., and Dabney, W. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762. 2024.
2017.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Ma, G., Li, L., Zhang, S., Liu, Z., Wang, Z., Chen, Y.. Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. arXiv preprint arXiv:2310.0744 2023.
Krogh, A. and Hertz, J. A simple weight decay can im- prove generalization. Advances in neural information processing systems, 4, 1991.
Mirzadeh, S. I., Farajtabar, M., and Ghasemzadeh, H.
Kumar, A., Agarwal, R., Ghosh, D., and Levine, S. Implicit under- under-parametie pattetttraattiizziiioonn inhibits data-efficient deep rein- forcement learning. arXiv preprint Xiiv:2010.14490 2020.
Dropout as an implicit gating mechanism for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 232-233, 2020.
10

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Mnih, V., Kavukeuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540): 529-533, 2015.
Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-I.,, and Courville, A. The primacy bias in deep reinforcement learning. In International conference machine learn- ing, pp. 18828-16847. PMLR, 2022.
Nikishin, E., Oh, Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., and Barreto, A. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems, 36, 2024.
Shen, M., Yin, H., Molchanov, P., Mao, L., and Alvarez, J. M. Step out and seek around: On warm-start training with incremental data. arXiv preprint arXiv:2406.. 244.044000044444 2024.
Shin, B., Oh, J., Cho, H., and Yun, C. Dash: Warm-starting neural network training in stationary settings without loss of plasticity. Advances in Neural Information Processing
Systems, 37:43300-43344 2024.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv: 1409, 1556, 2014.
Sokar, G., Agarwal, R., Castro, S., and Evei, U. The dor- mant neuron phenomenon deep reinforcement learning. In International Conference Machine Learning, pp- 32145-32168. PMLR, 2023.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-198, 2014.
Xu, Empirical evaluation of rectified activations in con- volutional network. arXiv preprint arXiv:1505.008 2015.

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
A. Proof of Property 1
Proof. Consider an input vector x = (Ø1, ,2,..., In) E IR". After applying AID, to x, the i-th element is expressed as:
AxD,(x); = Piz: 3i 2 0,
-1)ps) 22 0,
where p; a random variable sampled from Ber(p).
To enable vectorized computation, we introduce a mask matrix M E Raxx, satisfying ATD,(x) Mx. Specifically, the diagonal elements mis,s) of M are sampled follows:
m(.,)) (ep 1)H (1)
where H(-) is the step function, mapping positive values and negative values Then, the matrix M can written as:
M = (2P (x) + (I PP
(2)
where I the identity matrix of size n x n and diag (P1, P2.... Pp........... ,r)).., where diag(-) the function that creates a matrix whose diagonal elements are given by the corresponding vector.
Using equation 2, the AID operation be expanded as:
A1D Mx
[(2P IxH(() (I P)]x
(2P -))(()) + (I- -[--))
(3)
-(2P - -I)(() + - (I
- rx() + +r(x))
Pr(x)) + (I- P)(x r(x))
= Pr(x) + (I-P) PX)(()
[Pr+ (I-P)f]x
Thus, applying AID is equivalent applying ReLU with probability p. and negative ReLU with probability p.
12

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
B. Proof of Theorem 4.1
Proof. We clarify that we follow a similar procedure to the one demonstrated during the proof by Liang et al. (2021), which employs a comparable methodology.
Let the input data points and true labels be X1, x X and V2........3. (x,, Yr R*). For simplicity, let weight matrices be W1, 1, W2 E Rax= and they do not have bias vectors. If we assume that this model uses AlDp. p. using notation in Equation (3), the prediction of the model ýi can be expressed as below:
y: W2 P + (2P 1)r) (Wix.)
For testing, using E[P] = pI, we get y; = W2((1- p)1 + (2p - identity matrix, Bernoulli mask, ReLU and modified leaky ReI.U function, respectively, as mentioned Property 1.
Here, I, P. r()) and ry(+) refer to the
Then, the training process can be formally described as below:
min W.
W2 -I- (P I)r) (W xx)) - y:lle
For simplicity, the input vector and label will be denoted as (x,y) in the following proof. Let D = diag((Wis 0)) be the diagonal matrix, where (Wrz xx >> 0) is 0-1 mask vector, which means r(Wix) DW, x. To reduce the complexity of the equations, let us denote S, Sp and as below:
S = - + (2P I)D
Sp = (1 -P)I+ (2p 1)D
v = Wix
Note that S and Sp are diagonal matrix. Let the be trace of matrix and vec(-) be the function that makes elements of diagonal matrix to the vector. Then, expand the objective function below:
E W2 (I- P (2P 1)r) (Wix) -yl W - y
(4)
= TWt 2u((WS,vyy) + trrlyy
where, E Wt((WSyvTTSW))
El (r(vec(S))eec(S) diag(v) W Wzdiag(v))
(( diag(v)) Wzdiag(v))
(r(B(yec(S,))vec(5/// ve))
vec(S,)vee((.) diaag( (iiiag(v))o W, Wzdiag(v))
= (e(eec((S,)vee(SS,)) + diag (EE((1 - pt + (2p: - 1)d4) (1 - + (2p - 1) 1)d4) dia)))))))) Wzdiag(v))
(r((ve)(S,))cc(SS,) T + diag((p(1 p)((1 (da))))")))ddaa((\)))) Wzdiag(v))
(r((vec(S,)veo((S))) T p(1 p) (DDD)))ddavvvVV))))) Wzdiag(v)).
Here, p; and d; are (i, element P and D, respectively. Similarly, we expand the objective function when the activation function is modified leaky ReLU, r.))):
W2r, (2r,WWix Wix) - -y = ||2(( - I (2) - y11
(5)
= W y
-tr(W2S,vv' S, W. To
2tr T)(
13

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
S where, tWW2S, S, S, = vv W2)
tr(diag(v) vec(S,) diag(v) W2 W2) (r(vec(S))vec(S))) diag(v) W2diag(v)).
To determine the relationship between the two objective functions (4) and (5), we subtract one equation from the other:
E - P y1
1|
p)1 (2p LWD) (W - y
E(r(W,Sw''SW:)) - - -
Wzdiag(v))
= p(1 -rrprr (I 2D) 2D) diag(v) T W.)
= p(1 p)] |2 - 2D)diag(( W
p(1 - p)
W2 2D)
(6)
p(1 p)
W2 -2222 2W2rp(Wxx)) (Wxx)
(7)
n(2p- 1)2
4p(1 p)
W2 Wix Werp (Wix)
m(2p- - 1)
To derive Equation (7), we utilize the definition of Sp introduced at the beginning of the proof:
S, = =1 - p)l + (2p 1)D
1
(Sp - (1 - p)I)
2p
I- 2D = I
S - p)
2p
(2p I
28p 2(1 - I
2p -
I 2Sp
2p
Since r-()) = SSp)) get Equation (7). Finally, get the relationship between these two objective functions:
E W[12 yl13]
W2rp (Wix)
y
4p(1 - p) (2pp 1)2
W2 Wix
W2rp (Wix) w
This means that objective function of AID gives the upper bound of the optimization problem when leaky ReLU and penalty term. The penalty term means that weight matrices W1, W₂ are trained to closely resemble the behavior of the linear network. As p approaches 0.5, the regularization effect becomes stronger, which intuitively aligns - yl - 0) -
modified
with the fact that AID equivalent to a linear network when = 0.5. Since A/D,,,
= W2 wxx) = W (1 Wix), we confirm that AID has regularizing effect that constrains the network to behave like a linear 2
We identified made by Liang al. (2021) while deriving Equation (6), and correcting with incquality, the claim becomes weaker compared the original proof, However, we still assert that regularization effect remains valid.
14

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
C. Additional Studies
C.1. Exploration of AID in Relation to ReLU, Dropout and DropReLU
We explore that when the number of the interval is two around zero (i.e., = and 01 = 12 - 0), AID encompasses several well-known concepts, including ReL.U activation, Dropout, and DropReI.U. This relationship is formalized the following property:
Property 3.
Let p.Dp. be the function that dropout rate p for positive and for negative values. Then, in training process, ReLU,, Dropout, and DropRel.U can be expressed by AID formulation.
For simplicity, without derivation, we describe the relationships between the equations below. We assume that Bernoulli sampling generates an identical mask within the same layer. Let r(-) be the ReLU function, Dropout,() (-) be the Dropout layer with probability p. and DropReLU, be an activation function that applies ReLU with probability and identity with probabiltiy - p. Then, during training process, for input vector x, those functions can represented by AID as below:
r(x) 1x)
Dropout,(x)) px) p)
(r(x)) AID p. (.)p)(()))))))))) (x)/(1 p)
(x)) (x)
Property 3 indicates that using AID guarantees performance least for the maximum performance of models using ReLU activation alone, in combination with dropout, or with DropReLU, Analysis and investigation of the AID function under varying hyperparameters such as intervals and probabilities are left as future work.
C.2. Effect of Theorem 4.1 for ReLU and Standard Dropout
We analyze the changes in the formulation when applying dropout layer to the ReLU function instead of AID. Our findings show that this approach does not achieve the same regularization effect toward linear network as AID. We discuss this in Corollary C.1 below.
Corollary C.1. Similar setting with Theorem 4.1, combination of ReLU activation and Dropout layer does not have effect that regularize linear network.
Proof. We same notation with Theorem 4.1. We can write the prediction of the model that use Dropout layer with probability p after ReLU activation, y; below:
1 (I- (I-)
y W2
1-
where r is ReLU function. On test phase, y; W2r(Wix Then, to simplify the equations, let us denote S, Sp and v as follows:
S = 1 1-
(I (IDP)D
S, = D
= W
Then, following the same process in Theorem 4.1, derive the expansion for the relationship between [1 vIl and || - yl We note that the detailed derivation is omitted as it largely overlaps with the previously established proof.
15

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
W2 2
1
-
P)r(Wix)--X) - yyl
|2 (Wix) -yl6
P
D2 diag W Wediag(v) v)
1
tr( ur(Wadiag v DD diag(v) W2)
W W2Ddiag v)ll
1 P
2
p
|22r W (
n11 p)
E
1 -I- (-P))(Wix)- - -y13 W
2 [er((Wix) - -yl + n21 -
P
[ WWr(WWix))
Corollary C.1 demonstrate that the simultaneous use of Dropout and ReLUU does not achieve the same regularization effect on the linear network as AID. This suggests that, rather than completely discarding all negative values ReLU does, allowing portion of negative values to pass through with dropout plays crucial role in effect that regularizes to linear network.
16

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
D. Details of Property 2
At the forward pass case, the property the ReLU function used in the derivation equations He et al. (2015)'s work as follows:
E [(((v))) 2 Ely"
2 (Elv²) - [Fly)))
Var(y)
where r(-) is ReLU activation, y represents preactivation value which is random variable from zero-centered symmetric distribution. We show that substituting ReLU with AID in this equation yields the same result:
B((AID(()))
(1- = px 1 2
1 V Var(y) 2
+ (1- p) +
Varly).
Additionally, during the backward pass, He al. (2015) utilize that derivative of ReLU function, r' is zero with the same probabilities. We have same condition that is random variable from zero-centered symmetric distribution. Applying this to AID, we can confirm that it yields the same result:
P(AID(()) = = P(AID) = 1ly 2 0)P(y 2 0) P) 1/y < <) < 0)
=px 1 + (1-p) 1
2
P(AID)()) 0) P(AID) 0ly 2 0yP(y 2 0) P(ATD,)))))) = Oly < 0) <)
1 1
=1
+px 2
2
2
Therefore, under the given conditions, replacing the ReLU function with AID during the derivation process yields identical results. Consequently, can be concluded that Kaiming He initialization is also well-suited for AID.
17

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
E. Baselines
Dropout (Srivastava et al., 2014). Dropout randomly deactivates a fraction of neurons during training, effectively simulating an ensemble of sub-networks by preventing specific pathways from dominating the learning process. This approach improves generalization by reducing overfitting and encouraging the model to learn more robust and diverse representations. We applied Dropout layer next to the activation functions.
L. Regularization (Krogh & Hertz, 1991). Also known as weight decay, 1.2 regularization (1.2) adds penalty proportional to the squared magnitude of the model weights to the loss function. This encourages smaller weights, which can lead to better generalization.
1.2 Init Regularization (Kumar et al., 2023). 1.2 Init introduces a regularization term that penalizes deviations of parameters from their initial values, aiming to preserve plasticity in continual learning. Unlike standard L2 regularization, which penalizes large weight magnitudes, it specifically focuses maintaining the parameters near their initialization.
Streaming Elastic Weight Consolidation (Elsayed & Mahmood, 2024). Streaming Elastic Weight Consolidation (S-HWC) is a technique used in continual learning to prevent catastrophic forgetting. It identifies critical weights for previously learned tasks and penalizes changes to those weights during subsequent task learning.
Shrink & Perturb (Ash & Adams, 2020). Shrink & Perturb (S&P) is method that combines weight shrinking, which reduces the magnitude of the weights to regularize the model, with perturbations that add noise to the weights. This approach is particularly effective in warm-start scenarios. As done in prior study (Lee et al., 2024b), we use single hyperparameter to control both the noise intensity and the shrinkage strength. Specifically, given @ as the learnable parameter, d as the initial learnable parameter, and X as the coefficient of S&P, then the applying S&P is defined as: 0 x (1 -)0 + A06.
DASH (Shin et al., 2024). Direction-Aware SHrinking (DASH) selectively shrinks weights based on their cosine similarity with the loss gradient, effectively retaining meaningful features while reducing noise memorization. This approach improves training efficiency and maintains plasticity, ensuring better generalization under stationary data distributions. We applied DASH between the stages of training for generalizability experiments.
ReDo (Sokar et al., 2023). Recycling Dormant Neurons (ReDo) is technique designed to address the issue of dormant neurons in neural networks by periodically reinitializing inactive neurons. This method helps maintain network expressivity and enhances performance by reactivating unused neurons during training. We applied ReDo at the end of each task.
Continual Backprop (Dohare et al., 2024). Continual Backpropagation (CBP) is a method that enhances network plasticity in continual learning by periodically reinitializingg a fraction of low-utilization neurons. This targeted reinitialization preserves adaptability to new tasks, effectively mitigating the loss of plasticity while preserving previously learned knowledge.
Concatenated ReLU (Abbas et al., 2023). Concatenated ReLU (CReLU) is variant of the ReLU activation function that combines the outputs of both the positive and negative regions of the input. This technique enhances plasticity on Streaming ALE environment.
Randomized ReLU (Xu, 2015). Randomized ReLU (RReLU) behaves like a standard ReLU for positive inputs, while allowing negative inputs to pass through with a randomly determined scaling factor. This stochasticity can help improve model generalization by introducing diverse non-linearities.
DroP (Liang et al., 2021). DropReLU operates by probabilistically switching the ReLU activation function to an linear function during training. This method effectively achieves generalization while remaining compatible with most architectures.
Deep Fourier Feature (Lewandowski et al., 2024). Deep Fourier Features (Fourier) effectively address plasticity loss by leveraging the concatenation of sine and cosine functions, which approximate the embedding of deep linear network properties within a model.
Full Reset. To compare performance against models trained from scratch, we included full reset as a baseline. At each stage, we reset the model's parameters to their initial values before continuing training.
For baselines like CReLU and Fourier, which inherently double the output dimensionality, we ensured fair comparisons during trainability experiments by matching their number of learnable parameters to that of the vanilla model. In all other cases, we halved the output dimensionality to approximately halve the total number of parameters, thereby climinating any advantage arising solely from increased parameter counts.
18

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
F. Experimental Details
F.1 Loss of plasticity with Dropout
F.1.1. TRAINABILITY FOR DROPOUT
We employed an 8-layer MLP featuring 512 hidden units and trained it on 1400 samples from the MNIST dataset. The model trained for 50 different tasks, with each task running for 100 epochs. To evaluate subnetwork plasticity, we extracted 10 subnetworks the conclusion of each task, training these on new tasks for an additional 100 epochs and then calculated the mean final accuracy. Adam optimizer was utilized for model optimization.
F.1.2, WARM-START
We used the ResNet-18 architecture described in Appendix F.3. In the warm-start scenario, the model was pre-trained on 10% of the CIFAR100 dataset for 1,000 epochs and continued training on the full dataset for 100 epochs, with the optimizer reset at the start of full dataset training. In the cold-start scenario, the model was trained on the full dataset for 100 epochs. Adam optimizer with learning rate 0.001 was utilized.
E.2 Trainability
Permuted MNIST. We followed the setup of (Dohare et al., 2024). It consists of a total of 800 tasks that 60,000 images are fed into models only once with 512 batch size. In the beginning of each task, the pixel images are permuted arbitrarily. We trained fully connected neural networks with three hidden layers. Each hidden layer has 2,000 units and followed by ReLU activation function.
Random Label MNIST. We conducted a variant of (Kumar et al., 2023). It consists of a total of 200 tasks that 1,600 images are fed into models with 64 batch size. In the beginning of each task, the label class of images are changed to other class arbitrarily. We trained fully connected neural networks with three hidden layers 100 epochs per each task. Each hidden layer has 2,000 units and followed by ReLU activation function.
F3. Generalizability
We conducted experiments on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and TinyImageNa (Le & Yang, 2015) datasets using a 4-layer CNN, ResNet-18, and VGG-16, respectively, to evaluate the effectiveness of AID across different model architectures and datasets. We provide detailed model architecture below:
We employed a convolutional neural network (CNN), which is used in relatively small image classification. The model includes two convolutional layers with a 5 x 5 kernel and 16 channels and max-pooling layer is followed after activation function. The two fully connected layers follow with 100 hidden units.
CNN:
Resnet-18 (He et al., 2016): We utilized Resnet- 18 to examine how well AID integrates with modern deep architectures featuring residual connections. Following (Lee et al., 2024b), the stem layers were removed to accommodate the smaller image size of the dataset. Additionally, a gradient clipping threshold of 0.5 was applied to ensure stable training.
VGG-16 (Simonyan & Zisserman, 2014): We adopted VGG- 16 with batch normalization to investigate whether AID adapts properly in large-size models. The number of hidden units of the classifiers was set to 4096 without dropout.
In addition, we replaced the maxpooling layer with average pooling layer for methods such as Fourier activation, DropReLU, and AID, where large values may not necessarily represent important features. Next, we provide the detailed experimental settings below.
Continual Full. Similar to the setup provided by (Shen et al., 2024), the entire data is randomly split into 10 chunks. The training process consists of 10 stages and the model gains access up to the k-th chunk in each stage k. In each stage, the dataset fed into models with a batch size of 256. trained the model for 100 epochs per each stage, and we reset the optimizer when each stage starts training.
19

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Continual Limited. This setting follows the same configuration continual full, with the key distinction that the model does not retain access to previously seen data chunks. At each stage, the model is trained only the current chunk, without revisiting earlier data, simulating real-world constraints such memory limitations and privacy concerns.
Class-Incremental. For CIFAR100 and TinyImageNet, the dataset was divided into 20 class-based chunks, with new classes introduced incrementally at each stage. Unless otherwise specified, test accuracy is evaluated based on the corresponding classes encountered up to each stage, The rest of the setup, including the batch size, and training epochs per stage, follows the Continual Full setting.
F4. Reinforcement Learning
To evaluate whether AID enhances sample efficiency on reinforcement learning, we conducted experiments on Atari games from the Arcade Learning Environment (Bellemare et al., 2013), selected based on prior studies (Kumar et al., 2020; Sokar al., 2023). We trained DQN model following Mnih al. (2015) using the CleanRL framework (Huang et al., 2022). The replay ratio was set adopted in Sokar et al. (2023); Elsayed al. (2024). We followed the hyperparameter settings for environment from Sokar et al. (2023), with details provided in Table 2.
Table 2. Hyperparameters used in reinforcement learning environment.
Parameter
Value
Optimizer
Adam (Kingma, 2014)
Optimizer:
1.5e 4
Optimizer: Learning rate
6.25e 4
Minimum for training
0.01
Evaluation
0.001
Discount factor Y
0.99
Replay buffer size
106
Minibatch size
32
Initial collect steps
20000
Training iterations Training environment steps per iteration
10
250K
Updates per environment step (Replay Ratio)
Target network update period
2000
Loss function
Huber Loss (Huber, 1992)
F.5. Standard Supervised Learning
For the same model architecture and dataset used in the generalizability experiments, we trained with Adam optimizer for 200 epochs, applying learning rate decay at the 100th and 150th epochs. The initial learning rate was set to 0.001 and was reduced by factor of 10 at each decay step.
20

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
F.6. Hyperparameter Search Space
We present the hyperparameter search space considered for each experiment in Table 3. Without mentioned, performed sweep over different seeds for all experiments, except for VGG- 16 model on the TinyImageNet dataset, where used only 3 seeds due to computational cost. Tables to 10 shows the best hyperparameter set that we found in various experiments.
Table 3. Hyperparameter search space for every experiment.
Experiment
Method
Hyperparameters
Search Space
Warm-Start
Dropout
P
0.1,0.3, 0.5
AID
p
0.7,0.8,00. 0.9
Trainability
Adam
learning rate
le 10-3, 3, le -4
SGD
learning rate
3e 2,3e 3
L2
À
le 2, le -,
1.2 Init
À
le 2e - 5,
Dropout
P
0.01, 0,,.0.05,....1.3 00110000..00101111122202..1,,,11 0.1, 0.5 0.6 0.0, 1.0, 50.0
S&P
A
ReDo
threshold (r)
CBP
replacement rate(p)
le - 1,10 1, 4,1e-
maturity threshold
100
RReLU
lower
0.0625, 0.125, 0.25
upper
0.125, 0.25, 0.333, 0.5
DropReLU
p
0.1,0.2, 0.4, 0.5, 0.6, 0,6, 01.00.,,03,0.4,0000 0.5, 0,6, 0.9,
AID
P
Generalizability
Adam
learning rate
1e-3,
S&P
À
0.2, 0.4,0 0.6, 0.8
CBP
replacement rate(p)
le 4.1
maturity threshold
100, 1000
DropReLU
P
0.7,0.8,0.9 0.9
L2
A
le 2, 3, le - 4, 1e
Dropout
P
0.1,0.3,0.5
RReLU
lower
0.125
upper
0.333
AID
P
0.7, 0.7,0.8,0.9
Continual Full
DASH
0.1,0.3
0.05, 0.1,0.1,0.3
Class-nncremental
DASH
0.1,
X
0.05, 0.1
Continual Limited
DASH
a
0.,.
X
0.3
S-EWC
X
100, 10,10.10001100
Standard SL
L2
A
le - 2.
4. -
Dropout
P
0.1,0.3,0.3 0.5
AID
P
0.8,
Reinforcement Learning
Dropout
P
0.01, 0.001
AID
P
0.99, 0.999
21

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Table 4. Hyperparameter set of each method on permuted MNIST.
Method
Optimizer
Learning Rate
Optimal Hyperparameters
Bascline
Adam
1e-3
Dropout
Adam
1e-3
0.05
1.2
Adam
1e-3
À = le-5
L2 Init
Adam
1e-3
À = le-4
Shrink & Perturb
Adam
le-3
À = 0.2
ReDo
Adam
1e-3
T 50
Continual Backprop
Adam
1e
P le-4
CReLU
Adam
1
RReLU
Adam
3e-3
lower 0.0625, upper 0.333
DUup
Adam
3
0.99
AID
Adam
1e-3
= 0.99
Baseline
Adam
1e-4
Dropout
Adam
1e-4
0.05
L2
Adam
1e-4
À 1e-5
L2 Init
Adam
le-4
À = le-4
Shrink & Perturb
Adam
le-4
À = 0.1
ReDo
Adam
le
Continual Backprop
Adam
le-4
P le
CReLU
Adam
1e-4
RReLU
Adam
le-4 4
lower 0.0625, upper 0.333
DropReLU
Adam
1e-4 4
0.99
AID
Adam
1e-4
= 0.99
Baseline
SGD
3
Dropout
SGD
3
p 0.25
L.2
SGD
3e-2
À 1e-5
L2 Init
SGD
3e-2
À = 1e-3
Shrink & Perturb
SGD
3e-3
À = 0.1
ReDo
SGD
3e-2
T = 5
Continual Backprop
SGD
3e-2
P 1e-3
CReLU
SGD
3e-2
RReLU
SGD
3e-2
lower 0.0625, upper 0.333
DropReLU
SGD
3e-2
0.99
AID
SGD
3e-2
0.99
Baseline
SGD
3c-3
Dropout
SGD
3e-3
p 0.01
L2
SGD
3e-3
À le-5
L2 Init
SGD
3c-3
X = 1e-3
Shrink & Perturb
SGD
3e-3
À = 0.1
ReDo
SGD
3e-3
T =
Continual Backprop
SGD
3e-3
P = le-5
CReLU
SGD
3e-3
RReLU
SGD
3e-3
lower 0.0625, upper 0.333
DropReLU
SGD
3e-3
p 0.99
AID
SGD
3e-3
0.99
22

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Table 5. Hyperparameter set of each method on random label MNIST.
Method
Optimizer
Learning Rate
Optimal Hyperparameters
Bascline
Adam
1e-3
Dropout
Adam
1e-3
0.15
1.2
Adam
1e-3
X = le-4
L2 Init
Adam
1e-3
À = le-3
Shrink & Perturb
Adam
le-3
À = 0.8
ReDo
Adam
1e-3
T 0
Continual Backprop
Adam
1
P le-3
CReLU
Adam
1
RReLU
Adam
3e-3
lower 0.0625, upper 0.125
DUup
Adam
3
0.9
AID
Adam
le-3
= 0.9
Baseline
Adam
1e-4
Dropout
Adam
1e-4
0.25
L2
Adam
1e-4
À = le-3
L2 Init
Adam
le-4
À = le-4
Shrink & Perturb
Adam
1
À = 0.7
ReDo
Adam
lc
Continual Backprop
Adam
le-4
P le-4
CReLU
Adam
1
RReLU
Adam
le-4 4
lower 0.25, upper 0.333
DropReLU
Adam
1e-4 4
0.99
AID
Adam
1e-4
= 0.99
Baseline
SGD
3
Dropout
SGD
3
p 0.15
L.2
SGD
3e-2
À le-3
L2 Init
SGD
3e-2
À = le-3
Shrink & Perturb
SGD
3e-3
À = 0.2
ReDo
SGD
3e-2
T ==1
Continual Backprop
SGD
3e-2
P le-5
CReLU
SGD
3e-2
RReLU
SGD
3e-2
lower 0.0625, upper 0.125
DropReLU
SGD
3e-2
0.99
AID
SGD
3e-2
0.99
Baseline
SGD
3c-3
Dropout
SGD
3e-3
p 0.01
L2
SGD
3e-3
À le-2
L2 Init
SGD
3c-3
X = 1e-3
Shrink & Perturb
SGD
3e-3
À = 0.1
ReDo
SGD
3e-3
T =
Continual Backprop
SGD
3e-3
P = le-3
CReLU
SGD
3e-3
RReLU
SGD
3e-3
lower 0.0625, upper 0.25
DropReLU
SGD
3e-3
p 0.99
AID
SGD
3e-3
0.99
23

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Table 6. Hyperparameter set of each method continual full setting.
Dataset (Model)
Method
Optimal Learning Rate
Optimal Hyperparameters
CIFARIO
Baseline
1e-4
(CNN)
Full Reset
1e 4
L2
le-3
X le-2
Dropout
le-4
p = 0.3
RReLU
le-4
CReLU
1e-4
Fourier
1e-4
S&P
1e-4
À 0.8
CBP
1e_4
P 1c 5, maturity threshold 100
DASH
le
0.1, À 0.3
DropReLU
1e-4
p 0.9
AID
le-3
p 0.9
CIFAR100
Baseline
le-3
(Resnet-18)
Full Reset
le-3
L2
le-3
X le-2
Dropout
1e-4
P = 0.3
RReLU
1e-3 3
CReLU
le-3
Fourier
1
S&P
le 3
À 0.8
CBP
1c 3
P 1c 5, maturity threshold 1000
DASH
le-4 4
0.1, X 0.05
DropReLU
1e-3 3
p = 0.8
AID
le-3
p = 0.7
TinyImageNet (VGG-16)
Baseline
le-3
Full Reset
1e-3 3
1.2
le- 3
X le-4
Dropout
le-4
p = 0.1
RReLU
1e-4
CReLU
1e-3
Fourier
1 4
S&P
le-3 3
X 0.8
CBP
le 3
p 1e 4, maturity threshold 100
DASH
le-4 4
0.3, X 0.1
DropReLU
1e-4 4
p 0.7
AID
le-4
p = 0.7
24

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Table 7. Hyperparameter set of cach method continual limited setting.
Dataset (Model)
Method
Optimal Learning Rate
Optimal Hyperparameters
CIFARIO
Baseline
1e-4
(CNN)
L2
1e-4 4
À 1c-5
Dropout
le-3
p 0.5
S-EWC
le-4
À 0.01
RReLU
le-4
CReLU
1e-4
Fourier
1e-4
S&P
1e-4
À 0.2
CBP
1e_4
P 1c 4, maturity threshold 100
DASH
1
0.1, À 0.3
DropReLUU
1e_4
p 0.9
AID
1e-3
p 0.8
CIFARI00
Baseline
le-3
(Resnet-18)
L2
le-3 3
À le-5
Dropout
le-4
p 0.1
S-EWC
1e-3
À = 1
RReLU
1e-3 3
CReLU
le-3
Fourier
1
S&P
le 3
À 0.2
CBP
1c 3
P 1e 5, maturity threshold 1000
DASH
le 4
0.1, X 0.3
DropReLU
le
p = 0.7
AID
le-3
p = 0.8
TinyImageNet
Baseline
le
(VGG-16)
L2
le 3
À le-4
Dropout
1e-4
p = 0.1
S-EWC
le-3
À 100
RReLU
le 4
CReLU
1e-3
Fourier
1
S&P
le 3
X 0.4
CBP
le 4
P 1e 4, maturity threshold 1000
DASH
le-3 3
0.1, X 0.3
DropReLU
1e-4 4
p 0.8
AID
le-4
p = 0..
25

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Table Hyperparameter set of each method on class-incremental setting.
Dataset (Model)
Method
Optimal Learning Rate
Optimal Hyperparameters
CIFARI00
Baseline
le-3
(Resnet-18)
Full Reset
1e-3 3
1.2
1e-3
X le-5
Dropout
1e-3
p = 0.3
RReLU
le-3
CReLU
le-3
Fourier
1e-3
S&P
le-3
X 0.4
CBP
l 3
P 1e 4, maturity threshold 1000
DASH
le-4 4
0.1, À 0.05
DropReLU
1 4
p 0.8
AID
1e-3 3
p = 0.7
TinyImageNet (VGG-16)
Baseline
1e 3
Full Reset
1c-3 3
L,2
le 3
X le-5
Dropout
le
p = 0.1
RReLU
10 4
CReLU
1e-3 3
Fourier
1e 4
S&P
le 3
X 0.4
CBP
1c 3
P 1c 50-5, maturity threshold 1000
DASH
l 3
0.3, X 0.3
DropReLU
1e-4 4
p 0.8
AID
le-4
p 0.7
26

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Table Hyperparameeer of each method on reinforcement learning setting.
Game
Dropout (p)
AID (p)
Seaquest
0.01
0.999
DemonAttack
0.01
0.99
Spacelnvaders
0.01
0.99
Qbert
0.01
0.999
DoubleDunk
0.01
0.99
MsPacman
0.01
0.999
Enduro
0.001
0.99
BeamRider
0.01
0.99
WizardOrWor
0.01
0.999
Jamesbond
0.01
0.99
RoadRunner
0.01
0.999
Asterix
0.01
0.99
Pong
0.01
0.999
Zaxxon
0.01
0.999
YarsRevenge
0.01
0.99
Breakout
0.01
0.99
IceHockey
0.01
0.99
Table 10. Hyperparameter set each method on standard supervised learning setting.
Dataset(Model)
Method
Optimal Hyperparameters
CIFARIO
L2
= 1e-2
(CNN)
Dropout
p = 0.3
AID
p 0.95
CIFARI00
L2
N = 1e-5
(Resnet-18)
Dropout
p = 0.1
AID
p = 0.8
TinylmageNet
L.2
X = le-5
(VGG-16)
Dropout
p 0.1
AID
p 0.9
27

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
G. Omitted Results
G.1. Additional Results for Trainability
G. 1.1.1. EXTENDED RESULTS FOR VARIOUS CONDITIONS
Vanilla
Identity
Full Reset
RReLU
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
Sigmoid
Tanh
Fourier
CReLU
1.0
1.0
1.0
1.0
0.8
0.8
0.6
0.6
DropReLU
ReDo
CBP
S&I
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
L2
L2 Init
Dropout
AID (Ours)
1.0
1.0
1.0
1.0
0.8
0.8
M
0.6
0.6
MNIST SGD PermuledMNIST Adam
RandomLabelMNIIST SGD RandomLabe/MNIST Adam
High LR
Low LR
Figure 7. Additional Results for Trainability Various Conditions. We plot the train accuracy comparisons across different optimizers and learning rates on Permuted MNIST and Random Label MNIST for each method. We train SGD with learning rate 3e 2, 3e 3 and Adam with learning
1e 3, 1e
4.
conducted additional trainability experiments on various activation functions, including Identity, RReLU, Sigmoid, and Tanh, as presented Figure discussed in Dohare et al. (2024); Lewandowski et al. (2024), linear networks do suffer from plasticity loss. Interestingly, stochastic activation functions such RReLU, DropReLLU, and AID demonstrated strong trainability across different settings. This observation suggests that studying the properties of these activation functions may contribute to preserving plasticity, providing a promising direction for future research.
28

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
G.1.2. ANALYSIS OF METRICS CONTRIBUTING PLASTICITY Loss
The precise cause of plasticity loss remains unclear, but several plausible indicators have been proposed assess its impact.
Among them, the dormant neuron ratio (Sokar al., 2023) and effective rank (Kumar et al., 2020; Lyle al., 2022) have been widely studied key metrics. Recently, average sign entropy (Lewandowski et al., 2024) of preactivation values has been introduced as an additional measure. Each metric is computed as follows:
Dormant Neuron Ratio, Sokar et al. (2023) introduced dormant neurons as a measure for the reduced expressivity of the neural network. A neuron considered 7-dormant, its normalized activation score is lower than т. The normalized activation score is computed as follow:
EXED [x)
T EKE(Hy EXED (x)]
where st is normalized activation score for i-th neuron in /-th layer. E D is sample from input distribution, H' is the number of neurons in [-th layer, and (.) post-activation function. Note that I does not include final layer of the network. The dormant neuron ratio is computed as the proportion neurons with T = 0.
Average Unit Sign Entropy. Lewandowski et al. (2024) proposed unit sign entropy generalized metric encompass- ing unit saturation (Abbas et al., 2023) and unit linearization (Lyle et al., 2024). It is defined as follows:
Unit Sign Entropy (r) = Pp (xp()))
where p(x) is a distribution of inputs the network, h(-) preactivation values of according unit, and sgn(.) is sign function. To obtain the final metric, compute the mean across all units, yielding the average unit sign entropy.
Effective Rank. Previous studies (Kumar et al., 2020; Lyle et al., 2022) present the effective rank the output matrix after the penultimate layer closely related to plasticity loss. While several methods exist for computing effective rank, we follow srank (Kumar al., 2020):
sranks ( min Σ ( Σド= 0
2-8
where + is feature matrix, and (o)())) singular values of + sorted in descending order. We set 0.01 as default.
29

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Random Label MNIST
Dormant Neuron Ratio
Average Sign Entropy
Effective Rank
1.0
1.0
2000
0.8
0.8
1500
0.6
0.6
Vanilla
Dropoul:
0.4
0.4
500
0.0
0.0
30
100
150
200
100
150
200
50
100 Task
150
200
Task
Task
Permuted MNIST
Dormant Neuron Ratio
Average Sign Entropy
Effective Rank
0.6
0.8
1500
0.4
Vanilla
1000
Dropout
0.4
AID (Ours)
0.2
500
0.2
0.0
0.0
Task
Task
Task
Figure 8. Metries for Mcasuring Plasticity Loss.
This figure presents the Dormant Neuron Ratio, Average Sign Entropy, and Effective Rank across tasks, comparing Vanilla, Dropout, and AID random label MNIST (Top) and permuted MNIST (Bottom). Notably, AID maintains key metrics where Dropout fails, demonstrating its ability to mitigate plasticity loss and sustain network adaptability.
We evaluate these metrics for vanilla, Dropout, and AID under experimental settings characterized by severe plasticity loss. Specifically, we analyze permuted MNIST and random label MNIST with the Adam optimizer (Ir = 0.001). The results are presented in Figure 8. Interestingly, Dropout had no effect on random label MNIST and provided only a slight improvement in permuted MNIST. While both Dropout and Vanilla models cause a significant portion neurons to become dormant, AID effectively prevents neuron saturation. The Average Sign Entropy further supports this observation, as AID maintains consistently higher entropy, indicating a more diverse and active neuron distribution. Lastly, the Effective Rank plot demonstrates that AID preserves representational capacity throughout training, whereas both Dropout and Vanilla models experience rapid decline. These findings highlight AID's ability mitigate plasticity loss and sustain model adaptability over training.
30

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
G.1.3. ANALYSIS ON PREACTIVATION DISTRIBUTION SHIFT OF DROPOUT AND AID
Dropout
AID
AID vs Dropout at T=30
QG Q1
Q2
Frequency
8000
T=10
T=1
Frequency
8000
T=1
T=10
Q1
Frequency
8000
AID
Dropout
Q1/03
6000
6000
6000
Q1
4000
4000
4000
2000
2000
2000
0
0
0
-15
-10
55
-15
-10
55
5000
-2500
-100
0
Preactivation Value
Preactivation Value
Preactivation Value
Figure 9. Left & Middle. Visualization of the preactivation distribution shift for Dropout and AID at the first and tenth tasks. QI, Q2, and Q3 represent the first, second (median), and third quartiles, respectively. Right. Comparison of the preactivation distributions of AID and Dropout at the 30th task.
In this section, we present additional experimental results related to Section 3. Specifically, we analyze the preactivation distribution of Dropout and AID. The results presented in this section follow the Random label MNIST experimental setup as those in Section 3, with further details provided in Appendix F.1.1. We trained an 8-layer MLP on the random label MNIST and visualized the preactivation of the penultimate layer in Figure 9.
Figure 9 (left and middle) illustrate the preactivation distributions of Dropout and AID, for the first and tenth tasks, respectively. According to prior research (Lyle et al., 2024), preactivation distribution shift is one of the primary causes of plasticity loss in non-stationary settings. Dropout, which suffers from plasticity loss, exhibits a drastic preactivation shift, aligning well with previous findings. However, AID, despite maintaining plasticity effectively, undergoes similarly significant preactivation shift Dropout. This observation contradicts existing results, which indicate that the initial preactivation shift during training may not necessarily be the primary cause of plasticity loss.
Although both Dropout and AID experience preactivation distribution shifts, the extent of the shift in Dropout becomes more severe over the course of training compared to AID. Figure 9 (right) compares the preactivation distributions of Dropout and AID in the 30th task. As training progresses, Dropout's 's distribution shifts considerably, with the Q2 value approaching -2500, whereas AID maintains a distribution much closer to zero. These results indicate that while both Dropout and AID undergo distribution shifts, AID exhibits a bounded shift, preventing excessive plasticity loss. In contrast, Dropout experiences an unbounded distribution shift, leading substantial plasticity degradation. This finding suggests that simply applying different Dropout probabilities across different stages, in AID, can help mitigate preactivation distribution shifts.
G.2. Additional Results for Generalizability
G.2.1. GENERALIZABILITY COMPARISON: DROPOUT VS. AID
To further validate the findings in Section 3.2, conduct extended comparison between Dropout and AID in terms of generalizability under the continual full setting. While Dropout is known to improve generalization performance, our results suggest that it does not enhance generalizability. To strengthen this conclusion, we perform a controlled experiment using the same model architecture, dataset, and learning rate search range.
For each method (Dropout and AID), we evaluate two variants: one where the model is reset after each data increment and one where is not. For both settings, we identify the hyperparameter that yields the highest final accuracy at the last epoch. We then compute the generalizability gap by subtracting the accuracy of the non-resetting variant from that of the resetting one (i.e., (Cold-Start Acc.). (Warm-Star Start Acc.)). A smaller value indicates better generalizability, as it implies that the method retains performance even without full retraining.
Figure 10 demonstrates that AID consistently maintains a lower generalizability gap across training epochs, indicating its superior generalizability. While Dropout exhibits a smaller gap compared to the vanilla model, it fails to prevent the gradual increase over time. Investigating the theoretical foundations behind AID's ability to preserve generalizability would be a valuable direction for future research in this area.
31

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
CIFARIO (CNN)
CIFAR100 (Resnet- 18)
TinylmageNet (VGG-16)
0.5
0.7
0.6
Tost Accuraoy
Test Accuracy
Test Accuracy
0.4
0.5
0.4
0.3
0.4
0.2
0.1
0.2
200
400
600
800
1000
200
600
800
1000
200
400
600
800
1000
Epoch
Epoch
Epoch
Vanilla
Dropout
AID
Cold-Start
Warm-Start
CIFAR10 (CNN)
CIFARI 00 (Resnet-15
TinylmageNet (VGG-16)
Generalizablig Glap (%op)
Ceneralizan ceneralizabiitty Cap (%p)
Conaralizabliity Cap (%)p
15
Drapost
AID
10-
2-
5
-2
200
400
600
800
1000
200
400
600
800
1000
2000
400
600
800
1000
Epoch
Epoch
Epoch
Figure 10. Generalizability Comparison. Test accuracies of cold-start and warm-start variants for vanilla, Dropout, and AID (top row). To facilitate direct comparison, we report the generalizability gap, computed as the accuracy difference between cold-start and warm-start (bottom row).
G.2.2. HYPERPARAMETER SENSITIVITY
In this section, we present an additional study on the hyperparameter sensitivity of AID in the most representative setting, the continual full setup described in Section 5.1.2. We evaluate the effect of varying the hyperparameter in AID using the same models and datasets in the main experiments. Specifically, report the test accuracy the final epoch for p values in [0.6, 0.7, 0.8, 0.9), and learning rates 0.001 and 0.0001.
CIFAR10 (CNN)
CIFAR100 (Resnet-18)
TinylmageNet (VGG-16)
0.9
73.00
75.48
58.85
60.05
0.9
45.41
43.77
(0:35)
00:0)
(2.72)
48
40
0.8
(0:49)
74.56
68
(0/29)
(0:05)
80
0.8
(0.20)
(0:21)
-4
88
66.69
69.3 .7
63.82
64.03
50.77
43.81
00.530
(2,42)
(0.16)
10.191
(0:0-)
(0:39)
38
57. 60 (0.38)
60.86
56.06
58.14
43.81
33.73 (0:12)
98
0444
(0:37)
(0:18)
(0:10)
0.0001
0.001
0.0001
0.001
0.0001
0.001
In
Figure 11. Hyperparameter Sensitivity for AID, Heatmaps show the final epoch test accuracy and standard deviation for different values of p and learning rate on continual setting. lines in colorbars indicate haseline accuracy of full reset. Values above threshold are displayed in bold. Standard deviation is shown in smaller font below the mean accuracy.
The results are visualized heatmaps in Figure 11, where each cell shows the mean accuracy (with standard deviation in smaller font). The red line on each colorbar indicates the baseline performance of the full reset model. Accuracy values above this baseline are highlighted in bold. Overall, AID achieves higher performance than the full reset baseline across most hyperparameter configurations, demonstrating the robustness of AID to the choice of and learning rate.
32

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
G.3. Additional Results for Class-Incremental
CIFAR100 (Resnet-18) 18)
TinylmageNet (VGG-16)
0.9
0.7
0.8
Test Accuracy
0.7
Test, Accuracy
0.5
0.6
0.4
0.5
0.4
0.3
0.2
0.2
0.1
250
500
750
1000
1250
1500
1750
2000
0
250
500
750
1000
1250
1500
1750
2000
Epoch
Epoch
Vanilla
1.2
CBP
DASH RReLU
CReLU Fourier
DropReIU AID (Ours)
Full Reset
Dropout
S&P
Figure 12. Test Accuracy for Learned Classes in Class-Incremental Setting, As new classes are introduced, the task complexity increases, leading to gradual decline in accuracy.
CIFAR100 (Resnet-16 188)
TinylmageNee (VGG-16)
0.5
0.6
Test Accuracy
0.5
Test. Accuracy
0.4
0.4
0.3
0.3
0.2
0.1
0.0
0.0
250
500
750
1000
1250
1500
1750
2000
250
500
750
1000
1250
1500
1750
2000
Epoch
Epoch
Vanilla
L2
CBP
DASH RReLU
CReLU
DropReLU AID (Ours)
Full Reset
Dropout
S&P
Fourier
Figure 13. Test Accuracy for Total Classes in
Setting.
In the main section of the paper, primarily presented results for the difference from full reset in class-increment experiment. To provide more detailed analysis, we include additional plots in Figures 12 and 13, illustrating both the accuracy for previously learned classes and the accuracy across all classes. These results further highlight that AID consistently outperforms the other proposed approaches, demonstrating its effectiveness maintaining plasticity while mitigating performance degradation in plass-incrementee learning.
33

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
G.4. Additional Results for Reinforcement Learning
G.4.1. ATARI 2600 RAW SCORES BY GAMES
Seaquest
DemonAttack
SpaceInvaders
Qbert
Average RAW Score
2500
Average RAW Score
RAW Score
Average IAW Score
4000
2000
1.500
3000
1500
1000
1000
2000
500
500
1000
DoubleDunk
MsPacman
Enduro
BeamRider
Average RAW Scooe
Averuge RAW Score
800
Average RAW Score
5000
-10
2000
600
4000
RAW
1500
3000
-15
1000
400-
200
300
1000
WizardOFWor
Jameshond
RoadRunner
Asterix
Average RAW Scoze
Average RAW Score
3000
2000
4.00
RAW
3,0000
1000
Average
1000
200
200000
1000
1.0000
Pong
Zaxxon
YarsRevenge
Breakout
Average RAW Scorre
60
Average EUW Score
6000
10
Average RUA
4000
Average RAW
88000
2000
100000
200
5000
IceHockey
Vanilla
Dropout 0,01
Average RLW
Dropout 0.001
AID 0.99
0.999
Figure 14. Average RAW Scores across 17 Atari Games. We train DQN model over 10 million frames with replay ratio 1. The comparison includes vanilla DQN, Dropout and Shaded regions indicate standard deviation across runs.
In the reinforcement learning experiments, we sweep over relatively high coefficient for AID [0.99, 0.999], consistent with previous trainability experiments where a high coefficient was found to be optimal for continuously reducing training loss. For Dropout, we sweep over smaller coefficients [0.01, 0.001], since Dropout can cause large variance in reinforcement learning. Despite its lower trainability in some games, Dropout showed modest improvements in few cases, possibly due to its model ensemble effect. However, in three games-Asterix Asterix, BeamRider and DemonAttack- -wiih are known to suffer from plasticity loss (Sokar et al., 2023), Dropout failed to improve performance, supporting our argument that Dropout is ineffective against plasticity loss. While AID demonstrated advantages in most games, we observed slight performance drop in certain cases, such as Pong and Qbert, where plasticity loss was not a significant issue even at high replay ratios. Further investigation is needed to understand this behavior, including extended training on these specific games. Additionally, our experiments were limited high replay ratios; future research could explore alternative RL algorithms, different environments, long-term learning settings gain further insights.
34

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
G.4.2. IMPACT AID ON EFFECTIVE RANK
Asterix
BeamRider
DemonAttack
Effective Rank
500
Effective Rank
500
Effective Rank
Vanilla
500
Dropout
AID
460
480
400
420
460
2
10
0
10
10
Number Frames (in millions)
Number of Frames (in millions)
Number Frames (in millions)
Figure 13. Effective Rank 3 Games. The AID method maintains a higher effective rank compared to the vanilla and Dropout model throughout training.
Following the approach Section G.1.2, analyze the feature rank for three games-Asteris Asterix, BeamRider, and DemonAt- tack-where AID demonstrated significant performance improvements. As shown in Figure 15, the feature rank of the vanilla and Dropout model steadily declines throughout training, indicating loss of representational diversity. In contrast, AID consistently maintains a higher feature rank, suggesting that it helps preserve network plasticity.
G.S. Learning Curves for Standard Supervised Learning
CIFAR1O (CNN)
CIFAR100 (Resnet-18)
TinylmageNet (VGG-16)
1.0
1.0
1.0
0.9
0.8
Accuracy
Accuracy
0.8
Accuracy
0.6
0.4
0.7
0.6
0.2
0.0
50
100
150
200
50
100
1.50
200
50
100
150
200
Epoch
Epoch
Epoch
Veaill
L2
Drupout
AID (Ours)
Train
Test
Figure 16. Learning Curves on Standard Supervised Learning. The figure presents training and test accuracy curves for CIFAR10 (CNN), CIFARI00 (Resnet-188 18), and TinylmageNet (VGG-16) We divide learning rate by 10 at 100th and 150th epochs. AID not only effectively mitigates plasticity loss but also reduces the generalization gap, demonstrating its robustness in both continual and standard supervised learning settings.
The Figure 16 present training and test loss of experiment in Section 5.3. We observe that AID more effectively closes the generalization gap compared to L2 regularization and Dropout, both of which are commonly used to address overfitting. This result suggests that AID is not merely technique for mitigating plasticity loss but more enneral-purposee methodology that enhances model robustness across various tasks.
35

---

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
H. Implementation Details of AID
import torch
import torch. nn as nn
class AID (nn Module)
def
init _init_ (self, p-0.9):
super (AID, self) salff._inin initiii 0 0 self.p Eloat (p)
if 0.0 or p 1.0: raise ValueError dropout probability has to be between and 1, but got (p)")
def forward(self, x): if self.training:
mask torch. bernoulli (torch. full fuke_ (x, self.p)) bbol
return torch. where (mask, torch. relu (x), relu
else:
return torch. where (x >= 0, self.p (1 self.p) x)
Listing 1: PyTorch implementation of the AID.
We provide the PyTorch implementation the simplified version of AID in Listing corresponding Algorithm 2. Note that the negative ReLU function F(x) is implemented -r(-r), allowing for efficient expression using standard ReLU operations. For all supervised learning experiments, we simply replaced all ReLU activations in the network with the AID module above. More extensive experiments are required, but intuitively, since AID can preserve negative preactivation values unlike ReLU, we recommend replacing maxpooling layers with average pooling layers in model, as discussed in Section F.3.
In the reinforcement learning experiments described in Section 5.2, an additional consideration necessary. In general, when using dropout during policy rollouts, maintaining the model in training mode (i.e., with dropout active) can promote exploration by producing stochastic Q-values for the same state. However, in our setting, found that this behavior introduced instability during training. Therefore, we set the model to evaluation mode when selecting greedy actions during rollouts, ensuring deterministic Q-value predictions. This strategy was applied consistently to both Dropout and AID, Investigating how such design choices affect the behavior of tropout-based methods in reinforcement learning remains an interesting direction for future work, All other aspects of the training procedure followed the standard DQN framework.
36


Reviews:
[{"review_id": "ia2lycz94b", "reviewer": "Reviewer_AVaP", "summary": "The proposed method in this paper is motivated by the characteristic of dropout, poor trainability and good generalizability. By enhancing the trainability of dropout in AID, the model can have both good trainability and generalizability. The main point of AID is separating the neurons into positive and negative part, and using different masks to utilize both positive and negative neurons. As a result, the network can obtain better trainability than using naive dropout. In the experiment, AID outperforms most of the baselines in warm-start scenario, and is also effective on reinforcement learning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. First, we clarify a potential misconception: **Dropout does not significantly improve generalizability.**\n\n> “The proposed method in this paper is motivated by the characteristic of dropout, poor trainability and **good generalizability**.”\n> \n\nIn the literature, generalization performance in deep learning and generalizability [1] in plasticity have different meanings. This is addressed in Section 3.2 and illustrated in Figure 2 (right). While Dropout slightly improves performance, it does not reduce the performance gap between cold-start and warm-start, which signals loss of generalizability. In contrast, AID narrows this gap, indicating better generalizability. Thus, we show that Dropout is limited in both trainability and generalizability, whereas AID improves both. We appreciate the opportunity to clarify this important distinction and will revise the wording in the camera-ready version to avoid confusion.\n\n---\n\nWe address the reviewer’s concerns below.\n\n> 1. [...] authors only carried out the experiment on warm-start scenario [...] it is hard to figure out whether the trainability issues are really occur in this scenario. [...]\n> \n\nWhile the reviewer notes that our experiments focus on the warm-start scenario, our paper evaluates both **trainability** and **generalizability**, as shown in Section 5.1.1 and Section 5.1.2, respectively. Additional trainability results are included in Appendix G.1. Importantly, trainability issues typically arise in **non-stationary settings** (e.g., permuted or random-label MNIST) [5], while generalizability issues occur in **stationary continual settings** such as warm-start and class-incremental learning [2,3]. Prior work [1,4,5] has established these tasks as **standard benchmarks** for studying each form of plasticity loss, and we follow the same practice in our experiments.\n\n> 1. [...] initial test accuracy of AID is much higher than naive Dropout [...] To make the comparison be fair, all the methods have same generalizability at the learning phase [...]\n> \n\nWe understand the reviewer’s concern about fairness in comparison. The initial accuracy difference stems from hyperparameter choice. All methods were tuned based on final accuracy, reflecting typical practice. For instance, we found the Dropout setting with similar initial accuracy to AID, but their final accuracy gets lower (e.g., continual full, ResNet-18, table below). This highlights AID’s strength in consistently retaining generalization throughout training.\n\n| Epoch | 100th | 1000th |\n| --- | --- | --- |\n| Dropout p=0.1 | 0.255 ± 0.005 | 0.5395 ± 0.003 |\n| Dropout p=0.3 | 0.136 ± 0.023 | 0.5398 ± 0.005 |\n| AID p=0.7 | 0.289 ± 0.005 | 0.638 ± 0.001 |\n\n> 2.  [...] and only **additional treatment are applied when the number of training data increases.**\n\nThis is a valuable point. We distinguish two classes of methods for generalizability in warm-start:\n\n- Applied only as data comes in: S&P [3], head reset, DASH [2]\n- Apply continuously: H&T [1], Fourier activation [4], AID\n\nAID belongs to the second category. These methods are advantageous as they do not require task boundary information [6], making them more practical—especially in settings like RL where task boundaries are implicit or unknown. They can also complement transition-based ones, making AID broadly applicable.\n\n> 3. If AID is applied only when the experiment transition occurs [...] does AID still achieve better performance than others? [...]\n\nSince AID is implemented as an activation function, it is structurally unsuitable for being applied only at task transitions. \n\n> 4. I wonder this theorem is actually aligned with the experiment results in warm-start scenario.\n\nWe agree this is a subtle point. Theorem 4.1 supports AID’s ability to improve trainability by regularizing toward linear behavior. It does not directly explain the generalizability improvements observed in warm-start settings.\n\nHowever, prior work [4] has empirically shown that encouraging linearity can also lead to improved generalizability in several continual learning scenarios. Similarly, AID’s ability to inject linearity appears to yield benefits not only for trainability but also for generalizability in our experiments. This contrasts with other methods for trainability, which often fail to improve generalizability [1]—further supporting this interpretation.\n\n---\n\n[1] Lee et al., Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks, ICML 2024\n\n[2] Shin et al., DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity, NeurIPS 2024\n\n[3] Ash et al., On Warm-Starting Neural Network Training, NeurIPS 2020\n\n[4] Lewandowski et al., Plastic Learning with Deep Fourier Features, ICLR 2025\n\n[5] Lyle et al., Disentangling the Causes of Plasticity Loss in Neural Networks, arXiv 2024\n\n[6] Aljundi et al., Task-Free Continual Learning, CVPR 2019", "claims_and_evidence": "The main claims are not quite persuasive. The motivation experiments are not related to the results in the experiment section.", "methods_and_evaluation_criteria": "The evaluation criteria does make sense.", "theoretical_claims": "The theoretical results are not aligned to the empirical results.", "experimental_designs_or_analyses": "Experiment designs are valid.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n\n1. In the warm-start scenario experiment, AID achieves much higher accuracy than other baselines. Furthermore, AID is also effective on reinforcement learning\n\n\nWeaknesses\n\n1. As mentioned in this paper, the main disadvantage of using dropout is its low trainability in non-stationary target scenario, and the authors show the degradation of the trainability in permuted MNIST and random label MNIST experiment. However, in the experiment section, the authors only carried out the experiment on warm-start scenario, and it is hard to figure out whether the trainability issues are really occur in this scenario. If it is not, does the performance increase are from enhanced trainability? or just from the generalization capability of AID? Since the initial test accuracy of AID is much higher than naive dropout, it is hard to know that AID can actually resolve the trainability issue.\n\n2. It may not be a fair comparison in warm-start scenario between AID and other baselines since the initial test accuracy of AID is much higher than other baselines. To make the comparison be fair, all the methods have same generalizability at the learning phase, and only additional treatment are applied when the number of training data increases.\n\n3. If AID is applied only when the experiment transition occurs similar to DASH [1] (i.e. apply AID when the number of training data increases), does AID still achieve better performance than others? If it does not increase the performance, we can think that applying AID is just giving generalization capability similar to other generalization techniques (e.g. data augmentation, or batch normalization) which are always effective regardless of the learning scenario such as warm-start or non-stationary target experiments.\n\n4. As in theorem 4.1, the authors said applying AID can inject the linearity which can mitigate the plasticity loss (i.e. trainability). However, I wonder this theorem is actually aligned with the experiment results in warm-start scenario.\n\n[1] Shin et. al., DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity, NeurIPS, 2024", "other_comments_or_suggestions": "Already mentioned in the above section.", "questions_for_authors": "Already mentioned in the Strengths And Weaknesses section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9di2WHZiHs", "reviewer": "Reviewer_KPJu", "summary": "The paper proposes AID, a novel activation function that generalizes Dropout to intervals, where it can be applied with different probabilities to different intervals of the activations. In its simplified version, it has 2 intervals (positive and negative) and can be interpreted as an interpolation between ReLU and a linear activation.\nIt claims that\n1) AID effectively tackles plasticity loss in continual learning and reinforcement learning, unlike Dropout.\n2) This is achieved by a regularization towards a linear regime.\n3) AID also improves generalization in supervised learning.\n\n## update after rebuttal\nAll my concerns have been addressed. It also seems that the other reviewers' concerns have been addressed (apart from apparently Reviewer  AVaP, with whom we can engage in discussions if they feel otherwise). I maintain my score from the end of the rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback and thoughtful questions. We appreciate your recognition of the motivation, experimental setup, and analysis, and we address your concerns below.\n\n---\n\n> 2. b) Reinforcement learning [...] I would expect to see the vanilla method collapse [...]\n> \n\nReinforcement learning settings do not induce non-stationarity as severely as random-label or permuted MNIST. Indeed, prior studies with similar setups [1,2] also do not report full IQM score collapse. Instead, they demonstrate the presence of plasticity loss in reinforcement learning by showing that addressing it—through improvements in various indicators of plasticity such as dormant neurons and effective rank—leads to enhanced performance. Our experiments use the same setting—four times higher replay ratio (RR) than in typical DQN—which has been shown to accelerate plasticity loss [1,2,3], and is widely used in recent plasticity research [1,2,4].\n\nGame-specific results provide additional insight: AID outperforms the baseline in **DemonAttack**, **SpaceInvaders**, and **Asterix**, where the vanilla model struggles—suggesting plasticity loss. In contrast, for games like **Pong** and **Qbert**, where the plasticity loss is not the main bottleneck for the performance , AID does not benefit from mitigating plasticity loss. While other factors may contribute to performance,  the results of the trainability and high RR experiments support our interpretation that AID mitigates plasticity loss.\n\n---\n\n> 3. “Theorem 4.1. The notation is not clear, and the interpretations of the results do not help in supporting the claim.”\n> \n\nWe appreciate the reviewer’s feedback. We acknowledge that the current notation and interpretation in Theorem 4.1 could be clearer, and we will revise these in the camera-ready version.\n\nTheorem 4.1 shows that the lower bound of the loss function under AID, denoted as L_{\\text{AID}*p}, can be decomposed into the loss under a modified leaky ReLU, L*{r_p}, and a regularization term  that encourages **linearity** in the network. As the hyperparameter p approaches 0.5, AID imposes stronger regularization toward linear behavior.\n\nPrior work [5] has shown that **linear networks do not suffer from plasticity loss**, suggesting that the linearity-inducing effect of AID (as formalized in Theorem 4.1) directly contributes to mitigating plasticity loss.  In contrast, naive dropouts cannot induce the regularization term associated with linearity (Appendix C.2), which may explain its ineffectiveness at preserving plasticity.\n\n---\n\n> Figure 4. Vanilla does not seem to collapse; how are the benefits in plasticity isolated from the benefits in generalization?\n> \n\nAccuracy collapse is the indicator of trainability. The experiment in Figure 4 focuses on **generalizability**, evaluated by comparing the performance of the vanilla model to that of a fully reset model. A performance gap here indicates **loss of generalizability**, a form of plasticity degradation.\n\nA method with poor generalizability but good generalization may initially perform well, but lose its advantage over time. In contrast, AID maintains consistent performance throughout training. As shown in Section 3.2, Dropout enhances generalization performance but does **not** improve generalizability. AID, by narrowing the performance gap, demonstrates its ability to preserve generalizability.\n\n---\n\n> - Only performance is used to diagnose plasticity loss. […]\n> \n\nIn Appendix G.1.2, we evaluate multiple **plasticity metrics**, including **dormant neuron ratio** [1], **average unit sign entropy** [5], and **effective rank** [2], comparing vanilla, Dropout, and AID. In Appendix G.1.3, we also analyze **pre-activation distribution**, which is one of the causes that induces plasticity loss [6].\n\nThese analyses show that:\n\n- Dropout does not improve plasticity indicators relative to the vanilla baseline.\n- AID shows favorable trends across all metrics.\n- AID causes significantly less change to pre-activation distributions than Dropout.\n\nThus, our claims are supported by a combination of **performance metrics** and **plasticity metrics**.\n\n---\n\n[1] Sokar et al., *The Dormant Neuron Phenomenon in Deep Reinforcement Learning*, ICML 2023\n\n[2] Kumar et al., *Implicit Under-Parameterization Inhibits Data-Efficient Deep RL*, ICLR 2021\n\n[3] Nikishin et al., *The Primacy Bias in Deep Reinforcement Learning*, ICML 2022\n\n[4] Elsayed et al., *Weight Clipping for Deep Continual and Reinforcement Learning*, RLC 2024\n\n[5] Lewandowski et al., *Plastic Learning with Deep Fourier Features*, ICLR 2025\n\n[6] Lyle et al., *Disentangling the Causes of Plasticity Loss in Neural Networks*, arXiv 2024", "claims_and_evidence": "The claims are supported by evidence, which I do not find fully sufficient.\n\n1.a) continual learning - The claim is supported by Figure 2 showing the benefits of AID on plasticity loss and generalizability, unlike dropout.\n\n1.b) Reinforcement learning - Figure 6 does not show benefits on plasticity in my understanding. I would expect to see the vanilla method collapse but AID not to isolate plasticity as a cause. Otherwise, it's not clear that the benefits of AID come from better plasticity.\n\n2.) This is addressed in Theorem 4.1. The notation is not clear, and the interpretations of the results do not help in supporting the claim.\n\n3.) The claim is supported by Table 1.", "methods_and_evaluation_criteria": "The methods and evaluations in the paper are very well chosen and performed.\nThe paper categorizes and compares AID to multiple SoTA methods proposed to mitigate plasticity loss.\nIt uses supervised learning, continual learning, and reinforcement learning experiments.\nAll the experiments have enough statistical significance.", "theoretical_claims": "I did not check the proofs of the theoretical claims which are in the Appendix.", "experimental_designs_or_analyses": "Figure 4. Vanilla does not seem to collapse; how are the benefits in plasticity isolated from the benefits in generalization?", "relation_to_broader_scientific_literature": "The paper provides a good discussion of the broader scientific literature relating to plasticity and activation functions.", "essential_references_not_discussed": "The paper references the important literature combining dropout and ReLU and other attempts at mitigating plasticity loss.", "other_strengths_and_weaknesses": "+The explanation of why dropout does not prevent plasticity is beneficial.\n-Only performance is used to diagnose plasticity loss. The paper would be stronger by incorporating other metrics like feature rank, weight norm, activation norms, etc.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "No other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gDnS7aubQX", "reviewer": "Reviewer_PypX", "summary": "This paper presents a new method for addressing loss of plasticity. The authors investigate why Dropout doesn’t help with loss of plasticity, although it helps with generalization. Through empirical work, the authors pinpoint the causes and introduce AID, an improvement over Dropout that maintains plasticity and improves generalization. The authors show their method's effectiveness in a wide range of experiments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and their positive assessment of the paper. We appreciate the recognition of our motivation, empirical scope, and clarity, and we respond below to the concerns raised.\n\n---\n> It’s not clear how AID is sensitive to its hyperparameter. The authors need to be transparent about this information.\n\nWe agree and will include a sensitivity analysis in the camera-ready version. Below, we report the mean and standard deviation of final accuracy for various AID hyperparameter settings (*p*) in the **continual full** setup, which we consider representative. We use the same seeding procedure as in the main paper. We report $p=0.6\\sim 0.9$, as $p=0.5$ yields a linear network and $p=1$ gives ReLU. We exclude $p<0.5$ due to symmetry around 0.5.\n| *p* (Hyperparameter) | 0.6 | 0.7 | 0.8 | 0.9 | Best Baseline |\n| --- | --- | --- | --- | --- | --- |\n| CIFAR10 (CNN) | 0.601 ± 0.004 | 0.700 ± 0.004 | 0.746 ± 0.001 | **0.755 ± 0.004** | 0.717 ± 0.003 |\n| CIFAR100 (Resnet-18) | 0.5814 ± 0.002 | **0.640 ± 0.002** | 0.631 ± 0.003 | 0.600 ± 0.003 | 0.578 ± 0.005 |\n| TinyImageNet (VGG-16) | 0.438 ± 0.002 | **0.508 ± 0.0004** | 0.494 ± 0.004 | 0.454 ± 0.002 | 0.435 ± 0.001 |\n\nAs shown, AID mostly outperforms the baselines across a range of hyperparameters, though some variance is expected. Additionally, as shown in Table 3, we used a very small hyperparameter search space for the **generalizability** and **RL** experiments—where AID still performs robustly. This suggests AID works well without extensive tuning.\n\n---\n> The comparison with dropout in the RL experiment is missing (Figure 6).\n\nDropout is not commonly used in deep RL due to its high variance and unstable training behavior. We also found that Dropout offered no consistent advantage over the vanilla model, which is why it was excluded from Figure 6. However, given that the paper discusses Dropout’s role in plasticity, we agree that including its RL results would be valuable.\n\nBelow, we report the mean and standard deviation of final **human-normalized scores** for three games identified in Appendix G.3.2 as suffering from plasticity loss:\n| Game\\Method | Vanilla | Dropout | AID |\n| --- | --- | --- | --- |\n| Asterix | 0.115 ± 0.007 | 0.180 ± 0.025 | **0.318 ± 0.054** |\n| BeamRider | 0.107 ± 0.021 | 0.124 ± 0.009 | **0.242 ± 0.052** |\n| DemonAttack | 0.122 ± 0.039 | 0.155 ± 0.052 | **0.714 ± 0.245** |\n\nWhile Dropout shows minor gains over vanilla in some cases, the improvements are negligible and likely reflect ensembling effects rather than plasticity preservation. In contrast, AID demonstrates consistent and substantial improvements, reinforcing its plasticity-related benefits.\n\n---\n> There is room to improve the introduction…\n\nWe believe this refers to the paragraph beginning *“Widely used methods…”* on page 2. We agree that including the relationship between the overfitting and plasticity loss can be made clearer, and including data augmentation as an example—as the reviewer suggests—would enhance the explanation. We will revise this in the camera-ready version.\n\n---\n> The appendix refers to the method DIA instead of AID, which needs to be fixed.\n>\n\n> References with missing years [...]\n\nThank you for pointing these out. We will correct these issues in the final version.\n\n---\n> “Shouldn’t the scaling be 1/(1−p) for Dropout...?”\n\nWe appreciate the reviewer’s attention to detail here. The confusion likely stems from different ways in implementing Dropout. To clarify:\n\n- **Standard implementation** [1]: no scaling at train time; multiply by $1 - p$ at test time.\n- **Inverted implementation** (e.g., PyTorch’s `nn.Dropout`): scale by $1/(1 - p)$ at train time; no scaling at test time.\n\nIn our case, applying **inverted Dropout** to AID would make it act as an identity function during inference, removing non-linearity. Thus, we adopt **standard Dropout** way, which allows AID to maintain its activation function properties at test time via a modified leaky ReLU $r_p$.\n\nAlgorithm 2 may appear not to scale, but the scaling is implicitly applied via $r_p$ during inference. We hope this helps in understanding how AID works.\n\n---\n> How would AID perform under streaming supervised or RL settings?\n\nWe thank the reviewer for the interesting question. Though we haven’t explicitly tested AID in streaming, our results show that AID effectively preserves plasticity in several challenging settings. This suggests it may also be helpful in streaming contexts.\n\nHowever, streaming learning often involves **catastrophic forgetting** in addition to plasticity loss. Since our work does not focus on forgetting mitigation, the effectiveness of AID in streaming scenarios would likely depend on the relative impact of forgetting versus plasticity loss in those settings. We believe this is a promising direction for future investigation.\n\n---\n[1] Srivastava et al., Dropout: a simple way to prevent neural networks from overfitting, JMLR 2014", "claims_and_evidence": "The paper is well-written and easy to follow. I like how the authors decided to show the problem first with Figure 1 and tried to convince the reader first before introducing the algorithm itself. The results show the effectiveness of AID on performance. The authors also complement their empirical work with some theoretical analysis of why AID is able to address loss of plasticity.", "methods_and_evaluation_criteria": "The authors considered standard tasks for plasticity like input/output permuted MNIST, continual CIFAR10/CIFAR100/TinyImageNet. The authors also included challenging settings like incremental class learning with CIFAR100 and TinyImageNet in addition to reinforcement learning with high-replay-ratio DQN.", "theoretical_claims": "I didn't check the correctness of the proofs of the theoretical claims.", "experimental_designs_or_analyses": "The authors use standard benchmarking tasks that are widely used and accepted by the research community. I don’t see any issues in their experimental design or analysis.", "relation_to_broader_scientific_literature": "The paper provides a comprehensive comparison of many methods addressing loss of plasticity. In addition, they related their method to similar ideas in the literature, namely DropReLU, CRELU, and RRELU. I think that answering the question of why Dropout doesn’t address loss of plasticity would interest a significant number of researchers.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is well-written and well-motivated. The performance gain when using AID seems to be significant and superior to other methods, even though it is a simple method. The approach is easy to implement, which is a plus. All in all, I would like to see this paper published. There are a few issues that can be addressed easily by the authors. If the authors made efforts towards fixing the issues, I’m going to raise my score to reflect the changes.\n\nThe main weaknesses I see in the paper:\n- It’s not clear how AID is sensitive to its hyperparameter. The authors need to be transparent about this information.\n- The comparison with dropout in the RL experiment is missing (Figure 6).", "other_comments_or_suggestions": "- There is room to improve the introduction. Overfitting and loss of plasticity are often correlated, thus, most algorithms addressing overfitting would also address loss of plasticity. However, dropout (a method that addresses overfitting) does not address loss of plasticity. Thus, it’s interesting to understand why.\n\n- The appendix refers to the method DIA instead of AID, which needs to be fixed.\n\n- References with missing years:\n  - (Shin et al.)\n  - (Delfosse et al.).", "questions_for_authors": "> “During the testing phase of traditional dropout, each weight is scaled by the probability $1−p$, which is the chance that a node is not dropped. Similarly, in AID, as each interval has a unique dropout probability, the pre-activation values within each interval $I_j$ are scaled by $1−p_j$ for testing”\n\n> “Dropout layer with probability p scales the value by $1/(1−p)$ at training phase, so it acts as an identity function during the test phase. However, since AID functions as a non-linear activation, we scale the value at test phase”\n\nShouldn’t the scaling be $1/(1-p)$ for dropout in the first paragraph and by $1/(1-p_j)$ for AID in the second paragraph?\n\nAdditionally, can the authors share why they are not scaling in Algorithm 2? Also, why the scaling is $1-p_j$ in Algorithm 1 instead of $1/(1-p_j)$\n\\\n\\\n\\\nAnother question: how would AID perform under the streaming supervised learning setting or the streaming reinforcement  learning setting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a variation of dropout that leads to modest improvements in trainability and generalizability for neural networks.
Overall, there was minimal but positive enthusiasm for the work.
Decision: Accept (poster)