Activation Function Design Sustains Plasticity in Continual Learning
Authors: Lute Lillo
Nick Cheney
Venue: ICLR 2026 Poster (2026)
Keywords: loss of plasticity
continual learning
lifelong learning
continual reinforcement learning
activation functions
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Activation design—guided by simple first-principles rules—yields drop-in choices that keep models plastic across sequences and generalize better under distribution shift in continual supervised learning and RL.

Abstract: In independent, identically distributed (i.i.d.) training regimes, activation functions have been benchmarked extensively, and their differences often shrink once model size and optimization are tuned. In continual learning, however, the picture is different: beyond catastrophic forgetting, models can progressively lose the ability to adapt—loss of plasticity—and the role of the non-linearity in this failure mode remains underexplored. We show that activation choice is a primary, architecture-agnostic lever for mitigating plasticity loss. Building on a property-level analysis of negative-branch shape and saturation behavior, we introduce two drop-in nonlinearities—Smooth-Leaky and Randomized Smooth-Leaky—and evaluate them in two complementary settings: (i) supervised class-incremental benchmarks and (ii) reinforcement learning with non-stationary MuJoCo environments designed to induce controlled distribution and dynamics shifts. We also provide a simple stress protocol and diagnostics that link the shape of the activation to the adaptation under change. The takeaway is straightforward: thoughtful activation design offers a lightweight, domain-general way to sustain plasticity in continual learning without extra capacity or task-specific tuning.

Published as a conference paper ICLR 2026
ACTIVATION FUNCTION DESIGN SUSTAINS PLASTICITY IN CONTINUAL LEARNING
Lute Lillo & Nick Cheney
Department of Computer Science University of Vermont Burlington, VT 05401, USA
oll,lopp, ncheney y @uvm, edu
ABSTRACT
In independent, identically distributed (i.i.d)) training regimes, activation functions have been benchmarked extensively, and their differences often shrink once model size and optimization are tuned. In continual learning, however, the picture is different: beyond catastrophic forgetting, models progressively lose the ability adapt (referred to as loss of plasticity) and the role of the non-linearity in this failure mode remains underexplored. We show that activation choice plasticity loss. Building on a property-leve analysis of negative-brance shape and saturation behavior, we introduce two drop-in nonlinearities (Smooth-Lee Leaky and Randomized Smooth- Leaky) and evaluate them in two complementary settings: (i) supervised class-in dlas-incremmnntl benchmarks and (ii) reinforcement learning with non-stationary MuJoCo environments designed to induce controlled distribution and dynamics shifts. We also provide simple stress protocol and diagnostics that link the shape of the activation to the adaptation under change. The takeaway is straightforward: thoughtful activation design offers lightweight, Momain-genera way to sustain plasticity in continual learning without extra capacity or task-speciff tuming.
primary, sichitecture-agnosie
lever for mitigating
1 INTRODUCTION
Continual learning requires neural networks to acquire new knowledge over time without erasing previously learned information. This poses a fundamental challenge: maintaining a balance between plasticity- -the ability to adapt to new data- and stability the ability to retain prior knowledge. While catastrophic forgetting refers poor performance on previously learned tasks when they are no longer explicitly trained on, loss of plasticity is a distinct phenomenon: networks might retain past capabilities but become increasingly incapable of learning new ones. Despite growing interest, loss of plasticity remains less understood and underexplored, particularly in reinforcement learning (RL) settings where the agent's 's evolving policy changes the distribution data it encounters, making difficult to disentangle learning ability from environmental exposure.
Recent work documents symptoms associated with plasticity loss in deep RL, including reduced gradient magnitudes Abbas et al. (2023), increasing parameter norms Nikishin et al. (2022), rank-deficient curvature Lyle et al. (2022): Lewandowski al. (2023), and declining representation diversity Kumar et al.) (2020) 2023); Dohare et al. (2044). Yet no single factor explains its onset across settings. Lyle et al. (2024) propose "Swiss cheese" view: multiple, partly independent mechanisms e.g. pre-activation distribution shift, uncontrolled parameter growth, and the scale of bootstrapped value targets in -iiferre-difference learning- can each contribute. Mitigation strategies range from architectural refresh (Continual Backprop's senerate-and-tes.

---

Published as a conference paper ICLR 2026
replacement of low-utility units Dohare et al. [202]] regularization that targets plasticity retention Lyle et al. (2022) :: Kumar et al. (023); see Klein et al. (2024) for survey.
We argue that more fundamental knob hiding in plain sight: the activation function. Differences among activations often shrink in i.i.d. training once model size and optimization are tuned, but under continual, non-stationary data they can matter substantially. This motivates a property-level study of how activation shape, especially negative-side responsiveness and saturation, affects plasticity Previous work has shown that alternative activations such as CReLU CReI.USSaagg et al. (2066); Abbas et al. (2023); Lee et al. (2023) or rational functions Molina et al. (2019); Delfosse et al. (202 lb a) improve performance under -tatioaa conditions. Section 2 closes with a simple IID vs. class-incremental (C-IL) comparison using a shared setup illustrate this contrast and motivate the case studies that follow.
Our contributions are as follows:
Comparison of activations function performance across supervised continual learning and non- stationary RL benchmarks (Sec. Tab. AI).
Analysis of activation function properties identifying moderate, non-zero negative-side respon- siveness ( ('Goldilocks zone') and small dead-band width as key predictors of sustained plasticity (Secs.
Two drop-in activation functions SSooSSmooooeeeeo Leaky and Randomized Smooth-Leaky that preserve non-zero derivative floor and target the moderate-leak regime with C transition (i.e., having a continuous first derivative) between its linear and non-linear regions efficiently improving continual adaptation. (Secs.
2 ACTIVATION FUNCTIONS AND PLASTICITY IN CONTINUAL LEARNING
Activation functions are the first gatekeepers of gradient information. Their slope near zero, negative input behavior, and degree of saturation jointly determine how much learning signal survives backpropagation, a critical factor once data distributions change. We provide a comparative overview of commonly used activation functions, highlighting their potential to either exacerbate or mitigate plasticity loss.
Rectifiers. ReI.U is efficient but prone to the dead-unit problem (Maas et al. 2013): neurons that output O receive no gradient and thus become inactive permanently. Continual-learning studies confirm rising fraction of dormant units over time (dormant neurons phenomenon (Sokar et al. (2023)), shrinking gradient norms and eventually freezing learning (Abbas et al. 2023 Dohare et al.) 2024). ReLU networks often show an increase in parameter norms during training (as they drive outputs with ever-larger weights) and drop in the number of effective directions in weight space that can reduce error Lewandowski et al. (2023). Leaky-Rei ReLL alleviates this with constant negative slope, while PReI.U (He al. (015) and CR (Xu ( al. 2015) postpone dormancy by making the negative slope learnable or by randomly sampling it during training.
Saturating sigmoids. Sigmoid and Tanh map inputs bounded ranges; when units saturate, derivatives shrink toward zero and gradients can vanish, slowing learning Glorot & Bengio (2010). This is relevant for
continual learning, which requires sustained adaptation under shift and where loss of plasticity has been repeatedly observed Dohare et al. (2024); Abbas el al. (2023) Juliani & Ash 2024). (See Sec. evidence.)
4 for empirical
Smooth non-monotonic. Swish (Ramachandran et al. 2017) and GeLU (Hendrycks & Gimpel 2016) are smooth, weakly non-monotonic rectifiers that preserve small but non-zero- non-zero-gradientt for inputs near and below zero. This mitigates "dying ReLU" behavior (Maas et al., 2013). so units remain trainable when pre- activation distributions drift under shift. In continual settings, this responsiveness supports ongoing adaptation;
gradients
Coode available at: http:////w.tttteo..eeeeeee/////a
2

---

Published as a conference paper ICLR 2026
empirically, non-monotonic/smoodl rectifiers have shown advantages in both supervised and RL domains (Ramachandran et al. 2017, Hendrycks & Gimpel 2016
Elfwing et al. 2018). and in our experiments (Secs. D they exhibit lower dead-unit fractions and stronger late-cycle adaptation than zero-floor rectifiers.
Barron 2017) reduce bias shift via negative branch self-normalizes activations toward zero mean/unit variance. Under continual, non-i.i.d. data, where batch-statistics can drift and hurt retention, such built-in stabilization helps maintain trainable scales across tasks (Toffe 2017
Exponential variants. ELU/CELU (Clevert et al. 2015
(CELU is C²), while SELU (Klambauer et al., 2017)
Pham et al. 2022).
Why focus on continual learning? Under identical models and training budgets on Split-CIFAR-100, activation function rankings compress in i.i.d. joint training but separate sharply in class-incremental (C-IL) settings (Van de Ven al., (202)) (see Table [T.. This motivates probing how negative-brance behavior affects plasticity under shift. Unless noted, all case studies use the same 4-layer CNN backbone, the Adam
optimizer (Kingmal 2014), and training budget (full details in App. B isolating activation effects from architectural or optimization confusion.
ReLU
LReLU
RReLU
PReLU
Swish
GeLU
CeLU
eLU
SeLU
Tanh
Sigmoid
72.11
72.00
73.71
71,43
73.16
72.51
72.66
72.64
72.12
66.49
58.78
i.dd.
(0.40)
(0.63)
(0.24)
(0.52)
(0.55)
(0.8)
(0.42)
(0.33)
(0.78)
(0.59)
(0.48)
24.41
28.57
32.95
22.71
24.43
20.91
22.79
27.59
27.49
26.44
25,47
C-II
(0.75)
(0.64)
(0.12)
(0.93)
(0.86)
(0.64)
(0.25)
(0.89)
(0.18)
(0.64)
(0.69)
Table 1: Values for SPli--CIAAR-100 are reported as average accuracy (standard deviation) for independent runs with identical architecture, optimizer, and budget. Performance differences across activation functions are modest under i.i.d. joint training but widen under ctass-incremental learning (C-IL). BReLU attains the top mean in both settings; the C-IL improvement is statistically significant (all p < 0.05), whereas i.i.d. differences are not.
3 CASE STUDY I: NEGATIVE-SLOPE 'GOLDILOCKS ZONE'
We now test whether negative-side responsiveness drives plasticity under shift. Using the shared setup from Sec. 2 (T. we sweep the negative branch for three families: piece-wise linear (Leaky-Rel.U, RReL.U), smooth-tailed activations (Swish, GeLU, ELU/CHLU/SHLU) and adaptive (PReLU at global/layer/neurro scopes). Our goals are: (i) test if there exists a consistently good value of the negative-side slope across activation functions; (ii) test whether smooth tails approximate the effective slope (8) of optimal linear regimes; and (iii) assess whether adaptively learned slopes discover optimal values without extra guidance.
'Goldilocks zone' for negative slopes (with shape-matched comparison). Performance of activations with constant begative-branch slope (Leaky-ReLU, RReLU), reliably peaks for moderate leak 0.6 < $ 0.9 and degrades once 3 2 0.9 (Fig. [ Tab. D suggesting a 'Goldilocks zone' for negative slopes. To compare smooth-tailed functions on common scale, we project their negative-branch behavior onto an effective slope axis, ş = Ex-coo B (x)), representing the average derivative for negative inputs. When matched by s, these smooth tails still underperform linear leaks within the 'Goldilocks zone' and exhibit higher dead-unit fractions (Fig. IB), approaching (but not surpassing) the linear-leak peak only for 3 >>> 1.
Failure Modes for Slope Magnitude. We identify two distinet mechanisms defining the zone's boundaries. As $ - 0, dead-unit regime dominates (== 45% inactive). We implement dead-uni here as effective inactivity, identifying units either stuck in saturation platcaus or contributing negligible magnitude relative to the layer's s scale. This inactivity strongly correlates with accuracy loss (Pearson r= 0.51, 2-11,x10 28), and weakly but also significant with respect to final scaled gradient norms (Pearson r= 0..1,, p=0029 (App. C. Conversely, as s ++1 1, performance degrades despite minimal dead units. Fig. [C-D reveals this coincides
3

---

Published as a conference paper ICLR 2026
with optimization instability: sharp spikes in principal curvature (Amax) and effective rank (App. B.7 Thus, sustaining plasticity requires trade-off: avoiding gradient starvation (low 8) without inducing landscape stiffness (high 8).
Leaky RetU
CELU
ELU
SEW
SWISH
Aug Principal CUrvatura (mmax)
Average accuraty (8)
Performance
boy Dead fracion PS)
Dead Units
Effective Rank
Principel Curvature (mmax
0.0
200
Avg Effesivee Rank
25
0.6
30
20
0.4
20
13
0.2
50
10
0.0
Effective Negative Slope
Effective Negative Stepe
Effective Negative Stope 8
Effective Negative Stope
Figure 1: A: Final accuracy vs. effective negative slope 8. B: Dead-unit fraction vs. 3. Linear-leak families peak for & [0.6, 0.9]. Smooth-tailed activations are plotted on the same axis; they underperform within the 'Goldilocks zone' and only approach the linear-leak peak when 8> 1, reflecting concentrated near-zero responsiveness and vanishing tails. C: Effective rank of the gradient Gram matrix. D: Dominant Amax- Smooth-tailed activations show spikes at large 5, while constant-slope leaks remain comparatively stable.
Current adaptive, learnable slopes fail and need constraints to stay in-band. Under non-stationarity, the "right" negative-side responsiveness is not uniform across units or tasks. We therefore asked whether learnable randomized slopes find and maintain their specific 'Goldilocks zone'. Per-neuron PReLU (PReLLUN)
drifts below the band (= 0.3 0.6) over training and attains 30. 1% ACC (see Figs... C.2) layer/global scopes drift even further (Figs. CB
Thus, adaptivity is relevant-it offers robustness when single fixed leak cannot serve all units- unconstrained adaptation does not reliably remain in the pre-defined 'Goldilocks zone', which explains why learned slopes may sit outside it despite good (but suboptimal) ACC.
4 CASE STUDY 2: DESATURATION DYNAMICS UNDER SHOCKS
Case Study] 3 showed that negative leak (a non-zero derivative floor is necessary but not sufficient: after a distributional shift, many pre-activations can be pushed deep into an activation's 's tail, where gradients are effectively zero. We hypothesize that the time it takes a network to desaturate (how quickly gradients reopen after shock) a key determinant of adaptation delay in lifelong settings. Therefore, we subject the network to a protocol which isolates how quickly different activation families reopen gradients and regain performance after controlled shocks, complementing the steady-state results from Case Study 3
Every C2=10 epochs we apply a one-epoch scaling shock by multiplying all pre-activations by 7 (1.5, 0.5, 0.25, 2.03. then revert to 7=1 (App. D.1. Large 7 (e.g.. 2.0) pushes = into negative tails or positive plateaus (saturation); small (e.g., 0.5) produces the mirrored event. Each activation uses its best negative-slope setting from Case Study 3 (Table B4p. A unit is saturated at a step if (x)) < 10-3, We report: (i) Peak SF: the maximum saturated fraction immediately after each shock: (ii) AUSC: the area under the saturation curve over the recovery window (lower is better); (iii) Recovery time Tas: steps needed to regain 95% of pre-shock performance (.4..
D.4]
Derivative-floor rule. Activations with a strict non-zero derivative floor (Leaky-ReLLl RReLU, PReLU) achieve the lowest AUSC and near-zero non-recovery rates (<5%) even under the strongest shocks (Fig. middle/right). In contrast, zero-floor types (ReLU, Sigmoid, Tanh) show the largest AUSC and very high non-recovery across all y (Fig. left).
We call activation zero-floor if inf, 1 [z)] 0 (c.g., ReLU, and Sigmoid/Tanh whose derivatives approach 0 in the tails). We call it non-zero-floor if there exists such that of (a) 2 on the negative branch (e.g..
4

---

Published as conference paper ICLR 2026
Activation Function
Recovery Effactive
Zero Floor
Non Recovery vs.
Moan Epochs Hall-terooee [Suceedd RecOSAATION Only!
100
0.25
Non Rec--eeyyeryyy Ratee for SF
0.20
60
0.15
0.10
0.05
0.00
0.25
0.25
Germm
Garmia
Garmia
Figure 2: Desaturation under scaling shocks y. Left: mean AUSC (lower better). Middle: SF recovery time
(epochs to halve the saturated fraction after the shock; successful recoveries only). Right: SF non-recovery rate (%). Groups: Zero-floor ReLU, Tanh, Sigmoid: Non-zero-floor Leaky-ReLUU, RReLU, PReLU; Effective non-zero-floor ELU, CELU, SELU, GELU, Swish. See App. D.2 for details.
Two-sided penalty, Activations that saturate on both sides (Sigmoid, Tanh) suffer the worst shocks: they show the largest peak saturated fraction and AUSC (Fig. left/right,, and fail to desaturate in roughly half of runs (49.8%). One-sided (Kink non-zero-floor rectifiers (Leaky-Rel.U PReLU, RReLU) recover more reliably (non-recovery x 13.3%) One-sided (Smooth) (ELUU CELU, SELU) sit between these extremes: when they recover, they do so quickly (Fig. 3 middle), but failures still occur frequently at strong shocks. Overall, single hard saturation boundary less harmful than two; maintaining non-zero derivative floor on the negative side remains the most protective.
Peak Saturation Fraction
SF Time- ko-Haal--Reeoueee
AUSC
0.8
10
Area Under Sat. Curve (AUS)
1.2
Hall-Reecoo Recover
1.0
0.6
0.8
0.6
0.4
0.2
0.2
0.0
0.0
One Sided (Kink)
One Sided (Smoothh
TWo Sided
One Sided (Kink)
One Sided (Smooth)
One Sided (Kink)
One Sided (Smeoth)
Two: Sided
Two
Figure 3: Sidedness effects under shocks. Left: Peak saturated fraction during the shock (higher = more units saturated). Middle: Saturation Fraction (SF) time-to-half-rccon (epochs; successful recoveries only; lower is better). Right: AUSC (lower is better). Groups: One-sided (kink) = Reaky-ReLU,, PReLU, RReLU; One-sided (smooth) ELU, CELU, SELU; Two-sided (saturating) Sigmoid, Tanh. See App. D.3 for details.
The width of the dead band predicts shock sensitivity. Beyond the findings derivative floor and sidedness, we ask how much of the input range of an activation produces nearly zero gradients. We define Dead- Band Width (DBW) ) as the fraction of a typical pre-activation range (e.g.. [-100, 100]) where the magnitude the
Leaky/PReLU/RRLLL We call effective non-zero floor the derivative is non-zero on finite negative inputs near the decision boundary even though it decays toward as x + -oo (c.g., Swish/GeLU),
'B kink we continuous but not differentiable z= (C" but not C² e.g. ReLU, Rekky, By smooth least once differentiable ==0 (C' +; e.g., ELU, Swish).
5

---

Published as a conference paper at ICLR 2026
activation's is first derivative, [x)(())] falls below certain threshold, e < 10-3, Analytically computed DBW, (x)( < 10-3, strongly tracks desaturation outcomes across activations.
We hypothesize that this analytically derived DBW will positively correlate with experimentally observed adverse saturation dynamics. Intuitively, wider intrinsic dead-band means pre-activation scaling shocks are more likely to push numerous units into these unresponsive, This, in turn, diminishes the network's 's ability to desaturate, which we expect to manifest as greater overall saturation impact (higher AUSC) and increased non-recovery rates.
crrishnng-gradienn regions and keep them there.
Figure 4 strongly supports this hypothesis. revealing that the analytical Baad-Band Width Score is strongly and significantly correlated with adverse saturation outcomes. Activations with a higher DBW experience a greater overall saturation impact (Avg. AUSC, fail recovery entirely (Avg. SF Non-Recovery Rate, r = 0.84, p = 0.0013). DBW does not predict recovery speed. Successful desaturation, when it occurs, is consistently fast (around one epoch) and shows no correlation with the D DBW This suggests the DBW score predicts the likelihood and severity of saturation, but not the speed of recovery.
0.81, p 0.0016) and are much more likely to
Avg. AUSC
Avg. Rec. Time (Epochs)
Avg. Non-Rec, Rate (%)
relu
Zreell
50
0.10
rm0.83
clu (p-0.0016)
sigmal
tanh
Aug. (Epochs)
2.0
mm-0.25
(p-0.45)
Avg. Non Rec. Rate 1%6)
40
r-O 84
(p=0.0013)
sigmold ranh
Avg. AUSC
0.00
1.8
30
0.06
swish
1.6
20
0.04
prolu
0.8Rcl
1.2
10
1reb
beaky_ relu
ftel relu.reeu
gek.
zlamaid
Janh
eeky.relu
0.0
0.8
1.0
Dead-Band Width Score
Dead-Band Width Score
Dead-Band Width Score
Figure 4: Correlation of Dead-Band Width Score with Saturation Recovery Metries (All Gammas Aggregated). (Left):: Average Area Under Saturation Curve (Avg. AUSC) vs. Dend-Band Width Score. A strong positive correlation (Pearson r 0.81, p = 0.0016) is observed. (Middle): Average Saturation Fraction (SF) Recovery Time (for successful recoveries, measured by epochs) vs. Dead-Band Width Score. No significant correlation is found (Pearson r -0.25, p = 0.45). (Right): Average SF Non-Recovery Rate (%) vs. Dead- Band Width Score. A strong positive correlation (Pearson r = 0.84, p = 0.0013) is observed, indicating functions more prone to saturation are more likely to fail SF recovery.
5 IMPLICATIONS FOR ACTIVATION- DESIGN
Sections 3 suggest three rules for plasticity- friendly nonlincarities: (i) maintain non-zero derivative floor, (ii) keep negative-side responsiveness in moderate, activation-specific 'Goldilocks zone'. and (iii) prefer a C (smooth) transition at the origin when (i)- are held fixed. Conversely, avoid rwo-sided saturation and wide analytic dead bands (1y2'l < 10-3), which track larger AUSC and non-recovery.
Our measurements reveal two competing objectives: (A) recovery success (low non-recovery rate after shocks) and (B) recovery speed/extent (low AUSC, short time-to-recoven) conditional on recovery. Kinked C rectifiers with a strict floor minimize non-recovery across 7 (Fig. 2 middle/right), whereas one-sided C² middle), yet fail more frequently at the largest shocks (Fig. 2 right). Thus "smooth beats kink" is not universal: we prioritize (A) recovery success irreversible non-recovery dominates downstream performance and use (B) a tie-breaker among activations that recover. Therefore, we keep the strict floor and negative linear leak of the Leaky-ReLL family aiming for an empirical 'Goldilocks zone', and introduce smoothness only insofar as it preserves those two properties.
shapes often recover faster when they do recover (Fig- 3
6

---

Published as conference paper ICLR 2026
5.1 SMOKTH-LEAKY AND RANDOMIZED SMOOTH-LEAKY
Guided by Sec. 5 (i) strict non-zero floor, (ii) moderate leak, (iii) prefer C' over C when (i)- are held fixed-we introduce two drop-in rectifiers that keep capacity unchanged.
The Smooth-Leaky activation function (Fig- 5) is designed as a direct, C², drop-in substitute for Leaky ReLU that preserves the negative-side floor and the positive-side identity while moving the kink with a smooth, curved transition region. It for < O, f(z) x for
Smooth-Leaky
10
8
is asymptotically linear (f(x)
r >> 0) and controlled by a leak a plus smooth transition set by (p.c) c): f(x) = or + xx
cz
where is the sigmoid. Here, fixes the negative-side floor, and (p, c) set the width/steepness of the transition.
0-
10.0-7.5
550
-2.5
0.0
7.5
10.0
To add lightweight exploration around moderate leak we introduce Randomized Smooth-Leaky by replacing the fixed with random slope drawn uniformly from [1, each forward pass; at inference we fix r to its mean (7+u)/2:
Figure 5: Smooth-Leaky with 0=0.1,
p-3.0, c=5.0. Randomized Smooth-Leaky draws from bounds.
f(x) r (1 x
T~ r~ u), Ties 1+w
This randomized variant preserves the strict floor and C² transition while encouraging robustness small variations in negative-side responsiveness. Limitations of multi-parameter design and computational budget- fairness are explained in App. B.2
6 CONTINUAL SUPERVISED LEARNING
Following Kumar et al. 2023 we evaluate five supervised continual image-classfficttinn benchmarks spanning two shift types: input distribution shift (Permuted MNIST, 5+1 CIFAR, Continual ImageNet) and concept shift (Random Label MNIST, Random Label CIFAR). Training proceeds sequence of tasks without task task-identity signals where the agent receives bati-batches for a fixed duration per task. Permuted MNIST Goodfellow et al. (2013) applies a fixed random pixel permutation
shared subset for each task. Random Label MNIST Lyle et al. (2023) and Random Label CIFAR assign random labels to a fixed subset encourage memorization. CIFAR 5+1 draws and alternates hard (5 classes) and easy (single class) tasks from CIFAR-100. Evaluation focuses on hard tasks to stress plasticity loss mitigation. Finally, Continual ImageNet Dohare al. 2024);; Russakovsky al. (2015) performs task-binary classification over two ImageNet classes which do not repeat across tasks, ensuring non-overlapping class exposure and clearer measurement of plasticity over time. An extended explanation of each benchmark problem is found in App.
Across the five continual benchmarks, we observe a clear pattern, reported in Tab, 2 First, rectifiers with a learnable or randomized negative branch dominate: Leaky-ReLU, RReLU, PReLU, Smooth-Leaky, and Rand. Smooth-Leak, consistently outperform ReLU- especialyy on the harder smooth rectifiers such as Swi Swish/S SiLU are competitive but typically trail the best leaky-family, members. Second, we again observe the first reported 'Goldilocks zone' for the negative branch (cf. Sec. 3 the strongest performers cluster around an initial/effective negative slope the range (0.6, 0.9) (including the mean of RReLU bounds and neuron- wise in PR)LU). We also evaluated CReLU and Rat ional activations Abbas et al. (2023); Delfosse et al.] (2021b), Deep Fourier Features Lewandowski et al.] (2024), which satisfies criteria for smooth, non-zero gradient floor, maintaining responsiveness via periodic oscillation rather than a fixed linear slope, and Swi SWiGLU [Shazeer (2020), a gated nonlinearity that that gained recent popularity due to improve Transformer feed-forward layers and widely adopted in PaLM and LLaMA Touvron Toovrooouvronn
while
7

---

Published as a conference paper ICLR 2026
Permuted MNIST
Random Label
Random Label
CIFAR
Continual
Activation
MNIST
CIFAR
5+1
ImageNet
ReLU
78.85 + 0.06
20.03 + 2,46
25.79 + 6.18
4.76 1.01
73.71 0.43
Leaky-ReLU
84.14 0.01
91.53 #0.18
98.34 #.001
48.86
0.77
85.28 0.20
Sigmoid
76.96 + 0.07
79.59 0.75
52.24 2.99
1.79 +.019
63.89 + 7.38
Tanh
70.32 0.544
63,40 10122
58.56 1.05
28.59 1:334
70.97 1.4.
RReLU
83.95 1.0.2
93.10 1.002
98.02 1.003
53.60 1.06
84.97 1.117
PReLU
82.62 #0.5
92.67 +0.23
96.86 #.332
43.30 #0661
82.37 #0.11
Swish (8iLU)
83,41 #.03
67.73 +0.66
87.40
35.31 1.87
82.64 0.99
GeLU
78.97 0.09
38.79 #095
12.12
17.60 1.71
75.49 10111
CeLU
82.93 0.04
17.16009
29.64 10.44
54.23 1.44
81.15 1.66
eDU
80.50 0.09
84.23 +0.70
20.166
47.64 1.44
80.10 0.34
SeLU
80.43 1.16
79.95 1091
84.61 22.07
49.07 1.25
80,98 0.49
CReLU
82.66 0.04
89.47 +0.28
92.90 10.13
20.56 2.28
84.85 +
Rational
80.08 0005
92.35 1.97
94.82
40.41 #4221
80.65 #.38
SwiGLU
77.69 #0.26
31.20 2.10
83.06
9.57 + 1.81
63.57 2204
Deep Fourier
83.69 1.04
92.61 0.04
96.24 10.51
72.29 2.11
76.03 0.75
Smooth-Leaky
84.03 0.02
91.69 0.12
98.36 - 0.00
49.87 + 1.67
85.38 0.25
Rand. Smooth-leaky
84.26 + 0.02
93.33 0.05
98.42 + 0.01
57.01 + 1.59
86.23 0.13
Table 2: Total Average Online Task Accuracy (%) on Continual Supervised Benchmarks, averaged over 5 independent runs. Values are reported mean + standard deviation (SD). Statistical significance between the top two performers in each column was determined using an independent wampl Welch's 's t-test (p 0.05). Rand. Smooth-Leaky statistically significant with respect the next pertt-rrformnggggg activation (Smooth-Leeky)) Smooth-Leekky is also significant compared to the next best in Rand. Label CIFAR, CIFAR 5+1 and Continual ImageNet.
et al. (2023) Chowdhery al. 2023). Constrained rationals were excluded, previous work shows that they improve RL stability but reduce plasticity (uudej et al. (2025), which is our main focus. In our experiments, CReLU, i loaal and Deep Fourier outperform ReLU, but remain below other standard activations and our proposed variants (see Tab. E2 and Fig. ETI
7 CONTINUAL REINFORCEMENT LEARNING
Continual learning particularly critical reinforcement learning (RL), where non-stationarity arises not only from changes in the environment but also from the agent's evolving policy, which affects the data distribution even in fixed environments. This tight feedback loop between learning and data collection makes RI. especially vulnerable loss of plasticity, where neural networks become progressively less responsive to new experiences. Recent work has shown that deep RL agents suffer from a gradual decline in representational diversity and gradient signal quality as training progresses, leading suboptimal adaptation in later stages of learning Dohare et al. (2024); Abbas et al. (003)). Diagnosing and understanding this phenomenon RL can be more challenging than supervised learning settings due high variability in algorithmic design (e.g., model--asee vs. model-free, use of replay buffers, off-policy dynamics) and the inherent stochasticity of aggnttennireen environmentt interactions. As result, systematic demonstrations of plasticity loss in RL require carefully controlled protocols and extensive experimentation, challenge that approach from the perspective activation functions and their influence on network adaptability. Activation functions play pivotal role in deep reinforcement learning. Although ReLU and Tanh remain the most widely used options (c.g., |Mii et al. (2013) 2015); Hessel et al. (2018)), both exhibit limitations that affect learning dynamics Nauman et al. (0024). ReL.U, previously mentioned, is susceptible to issues such the dormant neuron phenomenon Sokar et al. (2023), loss of plasticity Lyle et al. (2023 2022), and overestimation when encountering
inputs Ball al. 2023) Tanh, commonly employed to constrain outputs
8

---

Published as a conference paper ICLR 2026
within fixed range, suffers from saturation at its extremes, leading to vanishing gradients that hinder efficient learning in deep networks Paacaanu et al. 2013).
Therefore, investigate plasticity loss in a continual RL setting, we train a single PPO agent Schulman et (0117) on fixed and repeating sequence of four MuJoCo locomotion tasks using Gymnasium Towers et al. (2024) : Hal fCheetah-v- + Hopper-v4 - Wa ker2dvv4 + Ant-v4 + Ha 1 FCheetah-v/ -> Hopper-v4 - ..... The agent cycles through this sequence three times, training for 1M timesteps per environment per cycle (total 12M). Episodes terminate early on invalid configurations following Dohare et al. (2024) (e.g., falls for Hopper/Walkere d, unstable heights for Ant); Hal aaafehheta runs full-length. The policy and value networks share a two-layer MLP backbone (256 units each) that is updated across all tasks. For each environment
attach lightweight input adapter (to map its observation space) and task-specific output head (for its action space); both persist across cycles and continue learning when the environment reappears. We trained using the Adam optimizer. Hyperparameters are in App. B3
Rand.
Metric
Swish
PReLU
Sigmoid
Smooth- Leaky Smooth-Leak,
IQM 95% CT 0.3149 + 0.071 0.2716 + 0.038 0.3329 + 0.059
0.3875 + 0.038
0.3305 + 0.037
Table 3: Average Plasticity Score across 5 seeds (higher is better). activations. See Table Fl for full comparison of all activations.
We only report the pop-performing 1 Values are reported as Min-Mo Min-
Normalized IQM mean 95% CI wilthwidth.
PRolU
Rand: Smaoth-Leaky
Sigmold
Smooth Leaky
Swish
Humanoldes
Antwvs
0:20
Hopper-ut
00M1
0.28
330
628
0.4
3.4
6:2
3.2
0:3
Staps (M030010
S0eps (40
Scepe (4880141
Stege (MBllon))
Figure 6: Plasticity Score across 5 seeds (95% bootstrap Cls) showing a complete sequence of 3 cycles across all 4 environments. The reports the Min-Max Normalized IQM of this score across seeds, only showing the top-performing activations to avoid clutter.
Plasticity Score (definition). To rigorously aggregate performance across environments with distinct reward scales, we employ normalized scoring protocol following Agarwal al. (202T). For each we compute the mean episodic return of the last cycle's 8 steady-state (final 15% of steps). This value Ri.e for B...-Rm
environment e is Min-Max normalized: Sr.e Rmias Rmmm Global bounds [Railn R.maax are determined via a robust percentile sweep across all activations. To prevent physics simulation instabilities in failure cases (e.g., Ffumanoidv33 vv from skewing the mean with unbounded negative values, we apply a stability floor: rewards below functional failure threshold are clipped to the lower bound (Sr.e 0). Finally, we report the Interquartile Mean (IQM) to provide a robust measure of expected plasticity. By filtering outliers while accounting for performance magnitude, the resulting Plasticity Score E [0, 1] reflects the agent's efficiency relative to the environment's 's theoretical maximum.
7.1 TRAINABILITY VS. GENERALIZABBILITY
Anchoring plasticity loss to single mechanism risks conflating cause and effect. We distinguish trainability
(learning capacity on current data) from generalizability (transfer to unseen variations). Prior work (Berariu et 2021) distinguishes trainabliiy-rredueedd reduced ability to lower loss on new data (Dohare al.) 2021 Elsayed & Mahmood 2024 Lyle et al.) (022) from generalizabliiyy-rede rreduce performance on unseen data (Ash &
Adams 2020 Zilly al. 2021). Mitigating plasticity loss should improve both, not merely memorize the
9

---

Published as a conference paper at ICLR 2026
latest data Lee al. (2024) nevertheless, their relationship remains unsettled, and terminology is often mixed Klein et al. (2024). In brief: loss of trainability (no effective parameter updates) can cause loss of functional plasticity, but the reverse need not hold models may still update yet fail to transfer gains.
This failure of plasticity has two faces in RL: (i) train-side adaptation can the agent still improve on the data it now collects?-and (ii) transfer to perturbed test conditions (e.g., randomized MuJoCo friction Dohare et al.] 2024)). We report two complementary metrics. The Plasticity Score (IQM) summarizes late-cycle trainability. The Generalization Gap (AGAP) measures how much the train-test gap widens over time (App. F For each cycle and environment e, GAP c.e R Ri where R is the expected return (measured at the end of cycle c). Thus A(GAP,) 3AP3, GAP r,ei larger A means the train-test gap widened over time (worse transfer). We aggregate á per activation using the Interquartile Mean (IQM) across environments. Crucially, these metrics must be analyzed together.
A high Plasticity Score is not preferable if it comes with a large, widening gap, just as a small gap is meaningless if absolute rewards are near zero (as seen in physics failures). Our two-metric system captures this tension. While Rand. and thus zero gap). in stable environments like Ant and Cheeta ah where it maximizes the Plasticity Score, it also attains lower IQM A (GGAP.) compared to baselines (Tab. 4 This indicates that its trainability does not come at the cost of overfitting; rather, it encourages solutions that transfer well, provided the dynamics remain stable. In contrast, S Sigmoid achieves moderate Plasticity Score via safety- its bounded nature prevents divergence in Humanoid-yer it exhibits larger widening of the gap in other tasks.
Smooth-Leake
suffers from stability issues in Humano idd (zero rewards
These metrics are complementary, not redundant. Plasticity Score asks "did the agent remain adaptable on the data it collected?", while (GAAP.) asks "did that adaptation carry to perturbed tests?". Our activation designs primarily target sustained train-side plasticity (strict floor, moderate leak, C' transition)- prerequisite for learning under shift. We report both metrics to surface, not mask, this open question.
Activation
HalfCheetah-vä v5
Humancid-v5
Ant-v5
Hopper-v5
A AQQ
Swish (SiLU)
533.55 + 1314.41
0.00 + 0.00
780.79 + 730.11
13.39 37.27
273.47
PReLU
839.60 + 596.03
-316.28 + 2211.72
94.17 + 660.28
-22.35 + 74.09
35,91
Sigmoid
-288.53 1 2576.85
18.92 = 111.38
276.48 + 1018.48
152.14 + 129.02
85.53
Smooth-Leaky
847.49 + 1611.57
0.00 + 0.00
44.09 1089.34
27.01 72.58
35.55
Rand. Smooth-Leaky
-9.388 + 863.55
0.00 + 0.00
-336.13 971.75
-68.68 + 96.31
59.03
Table 4: Change in Generalization Gap (AGAP). We report the difference in generalization gap between the final and first cycles (A(GAP.)) = GAP3. GAP).). Positive values indicate a widening gap (worse transfer) over time. Values are mean + 95% CI. The rightmost column reports the Interquartile Mean (IQM) of these deltas across environments. Note: Runs exhibiting physics simulation failures (reward -500) are treated as having gap of 0.0 (indicating no meaningful performance difference to overfit).
8 CONCLUSION AND FUTURE WORK
The choice of the activation function greatly impacts performance and plasticity in continual learning: it must be designed, not assumed. We do not claim universal wins, but our results show that first- tist-principles activations mitigate plasticity loss and improve trainability. Building on these findings, future work will test interactions with other standard continual learning approaches such as experience replay and regularization, narrow the RL generalization gap by tracking train-side plasticity and test-time robustness, which will help clarify when improved trainability helps versus when it simply leads to overfitting move from fixed "Goldilocks zone' slopes to adaptive, per-neuron self-tuning. We plan to derive a stronger theoretical understanding (curvaturedeesaturation bounds for smooth-leaky families) and scale to larger and more challenging CL. domains while studying the interplay of activation design with optimizer and normalization effects, culminating in principle-guide automated search for new robust activations.
recent data, and
10

---

Published as a conference paper ICLR 2026
ACKNOWLEDGMENTS
This material is based on work supported by the National Science Foundation under Grant No. 2218063 and 2239691. The authors acknowledge the Vermont Advanced Computing Center (VACC) at the University of Vermont for providing computational resources that have contributed to the research results reported in this paper.
REFERENCES
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on lifelong learning agents, pp. 620-636. PMLR, 2023.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances neural information processing systems, 34:29304-29933 2021.
Jordan Ash and Ryan Adams. On warm-starting neural network training. Advances in neural information processing systems, 33:3884-3894, 2020.
Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 Proceedings of Machine Learning Research, pp. 1577-1594. PMLR, 23-29 Jul 2023. URL htttps://proceedinggs. mlr www.pprew2//e////. / 123a. html
Jonathan Barron. Continuously differentiable exponential linear units. arXiv preprint arXiv:1004.07483, 2017.
Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks. arXiv preprint arXiv:2106.00 00042, 2021.
Garrett Bingham and Risto Miikkulainen. Efficient activation function optimization through surrogate modeling, Advances in Neural Information Processing Systems, 36:6634-6661, 2023.
Garrett Bingham, William Macke, and Risto Miikkulainen. Evolutionary optimization of deep learning activation functions. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp- 289-296, 2020.
Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribution shifts: An empirical study with visual data. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8281-8290, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter: Fast and accurate deep network learning by exponential linear units (clus), arXiv preprint arXiv:1511.07200 4(5):11, 2015.
Quentin Delfosse, Patrick Schramowski, Alejandro Molina, Nils Beck, Ting- Yu Hsu, Yasien Kashef, Salva and Julius Zimmermann. Rational activation functions. https://github.com com/ m1 Ai-rseearchrzationne act aetpllllcctttttttaeeee iveations 2020.
11

---

Published as a conference paper ICLR 2026
Quentin Delfosse, Patrick Schramowski, Alejandro Molina, and Kristian Kersting. Recurrent rational networks. arXiv preprint arXiv:2102.09407, 2021a.
Quentin Delfosse, Patrick Schramowski, Martin Mundr, Alejandro Molina, and Kristian Kersting. Adaptive rational activations to boost deep reinforcement learning. arXiv preprint arXiv:2102.09407; 2021b.
Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprini arXiv:2 100.20000000 06320,,55 2021.
Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Ruapm Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 032:768-778 2024.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighter linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Mohamed Elsayed and A Rupam Mahmood, Addressing loss of plasticity and catastrophic forgetting in continual learning. arXiv preprint ar 0rX1v:2404000781, 2024.
Vivek F Farias and Adam D Jozefiak. Self-normalized resets for plasticity in continual learning. arXiv preprint arXiv:2410.2000 2024.
Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed AI Kader Hammoud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem Real-time evaluation in online continual learning: new hope. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11888-11897, 2023.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirleenth international conference artificial intelligence and statisties, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
lan J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 13/.::3322.//1 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-10344 2015.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. 2016.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence batch-normalized models.
Advances in neural information processing systems, 30, 2017.
Arthur Juliani and Jordan Ash. study of plasticity loss in on-policy deep reinforcement learning. Advances in Neural Information Processing Systems, 37:1138844 13884-1139111 2024.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998) 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3511-352 2017.
12

---

Published as a conference paper ICLR 2026
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
Timo Klein, Lukas Miklautz, Kevin Sidak, Claudia Plant, and Sebastian Tschiatschek. Plasticity loss deep reinforcement learning: A survey. arXiv preprint (.X.x:2241..48/280 2024.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under patieeeeeaatttniaitionn inhibits data-efficient deep reinforcement learning. arXiv preprint M:Xiv:2010.14499, 2020.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization. arXiv preprint arXiv:2308. 11988, 2023.
Hojoon Lee, Hanseul Cho, Hyunseung Kim, Daehoon Gwak, Joonkee Kim, Jaegul Choo, Se- Young Yun, and Chulhee Yun. Plastic: Improving input and label plasticity for sample efficient reinforcement learning. Advances in Neural Information Processing Systems, 36:62270-62295, 2023.
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: maintaining plasticity with hare and tortoise networks. In Proceedings of the 41st International Conference on Machine Learning, pp. 06416-26438, 2024.
Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C Machado. Directions of curvature an explanation for loss of plasticity. arXiv preprint arXiv:2312.00246, 2023.
Alex Lewandowski, Dale Schuurmans, and Marlos C Machado. Plastic learning with deep fourier features. In The Thirteenth International Conference on Learning Representations, 2024.
Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Ling Pan. Measure gradients, not activations! enhancing neuronal activity in deep reinforcement learning. arXiv preprint arXiv:2505.24061, 2025.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning, arXiv preprint arXiv:2204.0955 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney, Under- standing plasticity in neural networks. In International Conference Machine Learning, pp. 23190-23211. PMLR. 2023.
Clare Lyle. Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762, 2024.
Andrew I. Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. Proc. ieml, volume 30, pp. 3. Atlanta, GA, 2013.
Franco Manessi and Alessandro Rozza. Learning combinations of activation functions. In 2018 24th international conference on pattern recognition (ICPR), pp. 61-66. IEEE, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5660 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
13

---

Published as a conference paper ICLR 2026
Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Padé activation units: End-to-end learning of flexible activation functions in deep networks. In International Conference on Learning Representations, 2019.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICMI-1O), pp. 807-814, 2010.
Michal Nauman, Michat Bortkiewicz, Piotr Mitoš, Tomasz Trzciński, Mateusz Ostaszewski. and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. arXiv preprini arXiv:2403.00554 2024.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. International conference on machine learning, pp. 16828-16847.
PMLR, 2022.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural net- works. In Sanjoy Dasgupta and David McAllester (eds.). Proceedings of the 30th International Conference on Machine Learning. volume 28 of Proceedings Machine Learning Research, pp. 1310-1318, At- mlr
lanta, Georgia, USA. 17-19 Jun 2013. PMLR. URL http::://pppceeeeetooo /ress//ppoccccccnnn...eeesssssas
pascanul3 html
Quang Pham, Chenghao Liu, and Steven Hoi. Continual normalization: Rethinking batch normalization for online continual learning, arXiv preprint arXiv:2203.16100 2022.
Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. Online continual learning without the storage constraint. arXiv preprint arXiv:2305.099 09253, 2023.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint 0,X1::1770005994.2001
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-22)) 1-222, 2015. doi: 10. 1007/s1 1263-015-0816.y.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv:/707.000 776447 2017.
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference on machine learning, pp. 2217-2225, PMLR, 2016.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXi::2002.052002 2020.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pp. 32145-32168. PMLR, 2023.
Rafal Surdej. Michal Bortkiewicz, Alex Lewandowski, Mateusz Ostaszewski, and Clare Lyle. Balancing expressivity and robustness: Constrained rational activations for reinforcement learning. arXiv preprint arXtv:2507.14736. 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 201iv:2302.1397/ 2023.
14

---

Published as a conference paper ICLR 2026
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024.
Gido M Van de Ven, Tinne Tuytelaars, and Andreas Tolias. Three types incremental learning. Nature
Machine Intelligence, 4(12):1185-1197, 2022.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:/505.000 2015.
Julian Zilly, Alessandro Achille, Andrea Censi, and Emilio Frazzoli. On plasticity, invariance, and mutually frozen weights in sequential task learning. Advances neural information processing systems, 34: 12386-1239,, 2021.
15

---

Published as conference paper ICLR 2026
A
CHARACTERIZATION OF ACTIVATION FUNCTION PROPERTIES
Activation
HDZ
NZG
Sat-:
Sat
C
NonM
SelfN
L/
8
ReLU Nair & Hinton
2010
LeakyReLU Maas ct al.
2013
PReLU He et al 2015
RReLU et al. 2015
Sigmoid
Tanh
Swish (SiLU) Ramachandran et al.
2017
GeLU Hendrycks & Gimpel 2016
ELU
Clevert et 2015
CELU Barron (001
SELU
Klambauer
2017
CRel.U Shang et 2016
Rational
Delfosse al.
2021b
SwiGLU Shazeer
(2020
Deep Fourier Lowandowow et al. 2024
Smooth-Leaky
R-Smooth-Leakk4
Table A1: Binary property grid (V= present, absent). Abbreviations. HDZ: hard dead zone; NZG: non-zero gradient for x < 0; Sat two-sided saturation; Sat- negative-side saturation; Ct first derivative continuous; NonM: non-monotonic segment; SelfN: self-normalizing output; l/Raip: learnable or randomized slope; f non-zero second derivative.
inputs (risk of effective inactivity via saturation). ELU C' only SELU has small derivative jump at x because ل 1. SELU's self-normalizing behavior holds under the preseribed a)) Unconstrained activations smooth, non-monotonie, and non-saturating; outputs grow large under training. Rational do not have "slopes" like Leaky-Re ReLU//ReLU;; they learn polynomial coefficients. Each branch retains ReLU dead zone, concatenation ensures that for any imput least one branch active, avoiding global inactivity. SwiGLU is gating mechanism (acting on a vector split) rather than scalar activation ((=). While it lacks static hard dead zone based on input sign, the multiplicative gating creates a dynamic dead zone: if the gate branch saturates zero, gradients for the value branch vanish. Proposed in this work. Potentially monotonic depending on choice slope parameter (and on the randomization bounds Smooth-Leaky.
*Gradients small but non-zero except for
EXPERIMENTAL AND ARCHITECTURAL DESIGN
Models for both Case Studies 3 and share the same backbone (4 x conv 256-unit MLP) and are trained on the 20-task Malit-CIFARIII 100 benchmark; Kaiming initialization uses the correct gain for each starting slope (ao=0.25 for every PReLU). Metrics recorded every epoch include final average accuracy (ACC), online forgetting, dead unit fraction, gradient signal-to-noise, Amax for the fcl block, and- for FReLU-the full trajectory (t).).
For continual supervised learning deliberately compact networks accentuate popacity-limitee plasticity loss: model may attain high average online accuracy on a few tasks, yet its ability to adapt degrades over long sequences. We employ (ii) CNN: two 5 x 5 convolutional layers with 16 output channels each, every conv followed by 2 x max pooling, then two fully connected layers of width 100. All architectures end with linear classifier whose
backbones. (i) MLP: two hidden layers, each of width 100.
16

---

Published as a conference paper ICLR 2026
oulput size is 10 for Permuted MNIST, Random Label MNIST, and Random Label CIFAR; 100 for 5+1 CIFAR; and 2 for Continual ImageNet.
For continual RL (Sec. 7. policy and value functions share a multi head MLP designed for sequential adaptation: a shared backbone with two hidden layers of 256 units is trained across all tasks, while for each environment we instantiate a dedicated imput adapter that maps observation space to the backbone and dedicated output hsad(s) for its action (and value) space. This modular design reuses core features while accommodating heterogeneous sate-action spaces.
B.1 LIMITATIONS AND MPLICATIOOS OF CHOOSING AN OPTIMIZER.
valid question how our results might interact with different optimizers (e.g.. SGD, RMSProp) beyond Adam. Our study's 's primary goal was to isolate the effect of activation functions on plasticity, which required fixing other key variables to create a controlled experiment. We intentionally selected the Adam optimizer for two main reasons:
Consistency with Prior Work: Our continual supervised learning benchmarks adapted from recent literatur(Kumar et al. (203)) which used Adam for their experiments. Anchoring our design to these established baselines allows for more direct comparisons.
A "Stress Test" for Plasticity: More importantly, adaptive optimizers like Adam have been specifically implicated in the literature as a contributing factor to plasticity loss. Prior work has suggested that Adam can struggle to update large-man magg-maae weights Dohare al, (2021) and has been shown to lose its ability to learn on non-stationary tasks Lyle et al. (2023).
We reasoned that if we wanted to study how activation functions mitigate plasticity loss, we should run experiments in a setting known to induce it. By demonstrating that our activation-design principles can sustain plasticity even when paired with optimizer known exacerbate the problem, provide more robust finding. We do not claim this is the optimal cilmiize--actiattion pair for SOTA results, but rather that activation design a fundamental mechanism for preserving plasticity, even under these challenging and fixed conditions.
B.2 LIMITATIONS AND IMPLICATIONS OF HULTI-PARAMER ACTIVATION DESIGN.
Smooth-Leaky and Randomized Smooth-Leaky expose several shape-controlling hyperparameters. This enlarges the tuning space-and thus the cost but also grants useful control, as reflected in our benchmarks. A common response is to replace sweeps with adaptive or learnable parameters. However, as shown in Section 3 adaptive, granular learnable slopes (e.g.. PReLU) underperformed in all our settings, while bounded, stochastic constraints (e.g., RReLU) worked better. In continual learning, where short-term objectives dominate, unconstrained learned parameters can drift to task-local optima that hurt performance across tasks, Attempts automate shape learning (Bingham & Mikkkulainen 2023 Bingham et al. 2020
Manessi & Rozza 2018) with highly expressive forms (c.g., Rational activations Delfosse et al. (2020) also lagged behind our first-principles designs here. This suggests that we have not yet reconeiled expressivity with robust automation. A promising direction is to rethink where and on what timescale hyperparameters are adapted-ppoeentalle decoupling their updates from the main training loop to better handle ntasstationarity.
matcht robustness and computational fairness. Our proposed activation family admits larger hyprraaraeeter grid than some baselines (because it has more tunable parameters), potentially inflating performance by enabling broader search. To address this, evaluate budget-matched robustness rather than enforcing an arbitrary equality in the raw number of tested settings across methods. For each activation i and dataset, define tuning budget N, as the number of evaluated hyperparameter configurations for i
17

---

Published as a conference paper at ICLR 2026
(Table B2). We then simulate practitioner with budget N; who can try N; randomly chosen configurations and keep the best result: we bootstrap a best-of-N, distribution by repeatedly sampling N; configurations (with replacement) and taking the maximum accuracy. We perform one-sided Jann-Whitney U test with alternative
H1 Bf (Rand Smooth-Leaky) Of
and report the resulting p- p-values in Table B1 Under this formulation, p-values close to 0 (reported as < 10-4 if values are smaller than such for simplicity) indicate that Rand. Smooth-Leate yields consistently higher best-of-N, outcomes than activation when both are given the same number of random hyperparameter trials, while p-values closer indicate the opposite direction.
The results show that Rand Smooth-Leeaky remains competitive under matched budgets across a wide range of baselines and datasets. In particular, for PERMUTED- -MNIST, RANDOM LABEL MNIST, RANDOM LABEL CIFAR, and CONTINUAL IMAGENET, Rand Smooth-Leaky significantly dominates most baselines under their own budgets (e.g., ReLU, Sigmoid, Tanh, PReLU, Swish/SiLU, GeLU, eLU, SeLU, CR Rational, and SwiGLU yield p < 10-4 in nearly all cases). This indicates that the strong performance of Rand Smooth-Lead Leaky not contingent on conducting an unusually large, constrained hyperparameter search: even when restricted to the same tuning budget as each baseline (typically N; E {2, 36, 40, 62, 64) depending on the activation), random budgeted sampling is sufficient to recover high-performing configurations with high probability.
Table [8 also makes the failure modes explicit via the directionality of the one-sided test. DeepFourier yields p-values of 1.00 or close to 1.00 on several datasets (e.g.. PERMUTED-MNIST. and CIFAR 5+1), indicating that under its small budget (N; = 2) its best-of-N; can exceed Rand Smooth-Leaky's best-of-N; in those settings; conversely, Rand Smooth-Leaky significantly exceeds DeepFourier on RANDOM LABEL MNIST, RANDOM LABEL CIFAR, and CONTINULL IMAGENET (p < 10 4) Similarly, Smooth-Leakk exhibits mixed pattern: Rand Smooth-Leeaky is not higher PERMUTED- DERMUTTD-MNIST, RANDOM LABEL. CIFAR, and CONTINUAL. IMAGENET (p-values 1.00), but it is significantly higher on RANDOM LABEL MNIST and CIFAR 1+1 (p < 10-4). However, this suggests the strength of the other activation function introduced: Smooth-Leaky. Thus, making such comparison between them independently good to whichever is better. Finally, CeLU shows a single notable exception on CIFAR 5+1 1 (p-value 1.00) while being dominated elsewhere, illustrating that cataset-speciffe interactions can invert the relative best-of-busd budget ordering. While other activation functions are close but not statistically significant in some datasets.
Overall, these do not rely on matching the raw number of hyperparameter combinations across methods (which is not meaningful once baselines plateau within their own hyperparameter ranges). Instead, by evaluating best-of-N; under matched budgets derived from each baseline's 's sweep size, we show that Rand Smooth- Leaky is typically robust to hyperparameter choice and achieves high performance even under limited random search budgets, while transparently identifying the few dataset/activation pairs where this dominance does not hold.
matched tests for computational fairness directly answer the concern: our conclusions
B.3 CRELU INTEGRATION.
To insert CReLU without changing the hidden width the rest of the network, we follow a napacity-neutral design: for any hidden layer with target width H, the preceding linear layer produces //2 pre-activations = E H/2 and we apply CReLU(z) concatenation. This mirrors the "CReLU (+half)" configuration: the parameter count of the producer linear is halved (inputx H /2 instead of input x Hd) the consumer linear still receives H features, and the forward shape everywhere else is identical to the ReLU baseline. Thus CReLU's representational benefit (explicit positive/negative phase features) is preserved while keeping model size and interface unchanged, enabling plug-in replacement of the activation without branching logie in the forward pass.
mmxx(2,00 0), max(- 0)] E R. which restores the width to H by
18

---

Published as conference paper ICLR 2026
Permuted MNIST
Random Label
Random Label
CIEAR 5+1
Continual ImageNet
Activations
MNIST
CIFAR
ReLU
10
< 10
<
10
10
Leaky-ReLL ReLU
1.00
0,0005
0,5820
0.0004
10
Sigmoid
10
10
10
10
10
Tanh
10
< 10
< 10
< 10
10-
RReLU
0.0310
0.471
0.0118
1.00
10-4
PReLU
10
0.071
< 10
< 10
< 10
Swish (SiLU)
10
10
< 10
10
10
GeLU
10
< 10
< 10
10
< 10
CeLU
10
< 10
< 10-
1.00
10
eLU
10
10
< 10
0.62
10
SeLU
10
< 10
< <10
10
0.37
10
CRcLU
0.0008
0.035
<0
<00
10
Rational
0.28
< 10
< <10
< <10-
10
SwiGLU 10
10
< 10
10
10
DeepFourier 1.00
< 10
0.77
< 10
10
Smooth-Leaky
1.00
10
1.00
10
1.00
Table Bl: Budget-fairness comparison significance test. For each baseline activation i, combined learn- ing rates, and dataset, let N, be the number of evaluated hyperparameter configurations for (Table (B2) We bootstrap best-of-N, performance by repeatedly sampling N; configurations (with replacement) and taking the maximum accuracy. We then perform one-sided Mann-Vy Khan-WWitney U test with alternative H1 BestOf (Rand Smooth-Leaky) > BestOf (i). Reported values are p-values. We perform 50 bootstrap repetitions.
For convolutional blocks with target channel width C' and fully connected blocks with target width H, we apply CReLU in a capacity-neutral manner: the producer layer outputs C/2 (or H/2) pre-activations z, and we apply CReLU(2) = [max(=, 0), max(- 0)] along the channel/feature dimension, restoring the width to C' (or H) by concatenation. This mirrors the "CReLU (+half)" configuration, preserving parameter count relative to ReLU baseline while exposing explicit positive/negative phase features. Subsequent layers therefore see the same interface shapes as the baseline, allowing CReL.U U to be swapped in without modifying the forward pass.
B.4 HYPER-PARAMETRL SWEEPS FOR ACTIVATION FUNCTIONS
Activation
Symbol(s)
Values explored
/ CReLU Deep Fourier
0.0
Leaky-ReLUUI
Swish/ GeLU
a 1
0.3, 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7,088, 0.9, 1.0, 1.05, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8,
0.6,0,06
2,1, 25
ELU CELU
0.1, 1.0,, 1.5, 2.0. 2.2, 2,, 2.6, 3..
4.0, 4,2, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9, 5,0,
SELU
1.0, 1.3, 1,673, 2.0, 1,6,,, 2.3, 2.4, 2.6, 2.8, 3.0, 3.1, 35, 39,
4.1, 4.3, 4.5, 4.7, 4.8, 4.9, 5.0 5.2, 5,,,
RReLU
U = 11,
[0.01,0,05], (0.05.0.10), (0.1,0.3), 125.0.333333, [0.3.1.0].
[0.4,1.0], [0.5,1.0], (0.6, [0.,1,0000 [0.7,100, (0.8,1.0],
10.9,1.01, (1.0,15)), 11772,21)1 (1443,,,,, (1.16822.188)
[0.9232,2.4222,, [1.548,1.798,, (0.673,2663), [0.423,2.923]
19

---

Published as a conference paper ICLR 2026
Activation
Symbol(s)
Values explored
PReLU Sigmoid / Tanh SwiGLU DFF Rational
scope
global, layer, neuron
P(O), Q6), V. A,
Grid over: (P(z), Q(x)) & (5,)),
Ve {A, B, C, D
A, E (ReLU, Leaky ReLU, Swish, Tanh, Sigmoid}
Smooth-Leaky"
(c,p.a)
Grid over:
4p E (1,2,3.4.5)
a (0.1,0.3. 0.5, 0.65, 0.700.80..)
(total 175 combos).
Rand-Smooth-Lealke
(Gp.L,u)
Grid
cpp (1,4,3,4,5) lower bound E {0.01, 0.3,
upper bound u E (0.05, 0.8, 1.0)
(total 450 combos); U(1,, u) sampled
per element during training.
Table B3: A summary of the syper-parameter sweeps performed for each activation function. The table details the specific values, ranges, and distributions tested for the corresponding symbols during our experiments. Best-performing hyperparameters are used to compute the results in Sections 3 and
Proposed in this work. See Section and App. G
The hyperparameter sweep of activation function continual supervised problems continual RL sim- ilar those presented Table [B3 with the exception of Smooth-L.cake Leaky and Ran. Smooth-Leakky where we will add new values to the shape-controlling variables and as well as new upper (u) and lower bounds. There- fore, the full list of variables will be: C,P {0.1, 0.3, 0.5, 0.8, 2, 3, 4,5), and the new bounds are [t, u] [0.673, 2.673). [0.55, 1.0]. [0.05, 0.70]. [0.01, 0.02], [0.01, 0.10]. which were used test wider range of possibili- ties outside intuitive bounds. We also introduce two activations functions presented previous works as mentioned earlier: CReLU and Rational Activations. CReLU, as a concatenated ReLU, only has the original negative value of O. On the other hand, Rational Activations are defined by the polynomial of the numerator P(x), denominator Q(x), the rational version (V) used (c.g., A, B, C, D), and the function (Ay) that is trying approximate (c.g., RcLU, Swish, etc).
B.5 I.I.D. VS CLASS- CLASS-INCREMENTAL CONTINUAL LEARNING COMPARISON HYPERPARAMETERS
Following the grid search over the hyper-parametee parameters listed in Table B3 we chose the best- test-performing configuration for
each activation function presented in Table B4 Fach continual- lleamingg run comprised 100 epochs across 20 tasks (100 classes in total, five classes per task). We adopted standard Experience Replay (ER) framework with random-sampling huffer size JM] 10,000, capped at 500 examples per task. The larger-than-ustag buffer was intended to mitigate catastrophic forgetting and ensure consistent performance across activation functions, allowing us to isolate the effects of reduced plasticity (following the intuition presented in Dohare et al. (0244).
B.6 UNDERSTANDING MAASS-INCREENNAL CONTINUAL LEARNING METRICS
properly understand choice metrics throughout this paper, we first need define and explain some of the classical metrics used in continual learning- Final Average Accuracy (ACC), Backward Transfer (BWT), and Forward Transfer (FWT) and relate them to loss of plasticity. Consider learning process unfolding over V+1 sequential phases/tasks t t=0, ...., N. Between consecutive phases t-1 and t (t > 1), the data distribution, environment dynamies, reward signal, task objective may change, inducing non-stationarity; denote the distribution phase by E., The
20

---

Published as conference paper ICLR 2026
Permuted MNIST
Random Label
Random Label CIEAR
CIEAR
Continual ImageNet
Activations
MNIST
5+1
CeLUU
64
64
64
64
CReLU
2
2
DeepFourier
2
2
2
eLU
64
64
64
64
64
GeLU
64
64
64
64
64
Leaky-ReLU
64
64
64
PReLU
36
36
36
36
Rational
20
26
12
41
14
ReLU
2
2
2
RReLU
40
40
40
40
40
SeLU
62
62
Sigmoid
2
2
2
Smooth-Leaky
175
175
175
175
175
SwiGLU
2
2
2
Swish (SiLUU
64
64
64
64
Tanh
2
2
Table B2: Configuration counts for budget fairness comparison. N₂ denotes the number of successfully evaluated hyperparameter configurations for activation i on each dataset. These values define the tuning budget used in Table BT
Activation
i..d.
CL
HP
LR
HP
LR
ReLU
0.001
0.0001
Leaky-ReLU
0.01
0.001
0.7
0.0003
RReLU
(0. 125, 0.333)
0.001
(0.673,
2.673)
0.0003
PReLU
layer
0.001
neuron
0.001
Swish
0.001
0.05
0.0001
GeLU
1.0
0.001
0.05
0.001
CeLU
1.0
0.001
2.4
0.001
eLU
1.5
0.001
3.9
0.0001
SeLU
1.673
0.001
3.7
0.0001
Tanh
0.001
0.0001
Sigmoid
0.001
0.001
Table B4: Optimal Hyperparameters (HP) and Learning Rate (LR) on Split-CIFAR100. A dash (-) indicates that such activation function uses the unique or baseline parameter (e.g., ReLU does not have any shape- controlling parameter since is linear on the positive right side and 0 on the negative left side).
model enters phase with parameters 0.-1 and is updated for Timee adaptation steps producing intermediate iterates 0r. (k= 1,..., Tphsee and terminating at 0, OrT
Then: (i) ACC+= Σ Ar, summarizes performance after learning T tasks but mixes current-task performance, retention of past tasks, and interference: (ii) BWT, 7-m Σ (Ar.: - A..s) quantifies forgetting/retention: and (iii) FWTT compares pre-training performance on future tasks to suitable haseline, capturing positive/negative transfer. Although BWT, and FWTT are widely reported in continual learning, in our setting we focus on performance metrics related to ACC, following recent work (Kumar et al, (023). We therefore briefly outline their differences and use-cases.
""Steps" may be epachs, mini-batches, timesteps, depending the domain.
21

---

Published as a conference paper ICLR 2026
Because A averages over tasks, cannot isolate plasticity fast adaptation within current phase. In our class- incremental continual learning setup with experience replay (Sec. (Van de Ven et al. 2022). we report ACCr for overall performance. For the rest of our continual supervised learning benchmarks we shift to a more plasticity- focused evaluation and report lifetime aggregate of per-task Average Online Task Accuracy named Total Average Online Accuracy which track within-phase learning and, when task difficulty comparable, reveal trends of loss of plasticity.
B.6.1 DIFFERENT ACCURACY METRICS ACROSS CONTINUAL LEARNING
Average accuracy across seen tasks (ACCt). This the standard average accuracy used in (offline or online)
class-incremental CI. summarize performance after training up task T. Let ace!") be the test accuracy on task measured after finishing training on task T. Then
T Σ
(T) ACCT T acc
High ACCr implies some combination of plasticity (learning new tasks) and stability (retaining old ones). However, ACC₁ mixes current-tass performance. retention on past tasks, and the interference caused by learning the current task, does not isolate plasticity on most recent shift. In our Case Studies which use an experience-replay buffer. we report this metric as final accuracy.
Average online task accuracy (per task). To quantify adaptation on a single task independently of past-task test performance, we use the mean online accuracy over the hatches of task Tg. Let Mj he the number of mini-batches in task and let di, denote the online accuracy on batch of task i. Define
M, Σ
AOA(T.)
OLj
(4)
M,
This captures how quickly the agent learns the current task (plasticity). downward trend of AOA(T.) across tasks of equal difficulty indicates plaxticity loxs.
Total average online accuracy (lifetime). For optimal hyper-parameter selection we also aggregate online accuracy over all batches seen so far (common in online CL et al. (202I): Ghunaim et al] (2023); Prabhu el al.) (2023):(Knn))))) (umar et al. (2023)). Let Bgr Σ M, be the total number of processed batches up task T. and let de be the online accuracy at global batch index l. Then
Ber-1
M
TA
(5)
B&T
Σ
M
i=
If all tasks have equal length M, = M, this reduces to
MT-1
TAOAST
MT
Σ
az
(6)
We distinguish this lifetime aggregate from the per-task quantity AOA(T.). This metric is reported in Section 6
B.7 CURVATURE METRICS
In order study the properties of the curvature of neural network and how it affects the loss of plasticity, need to work with the Hessian matrix. For loss function L() (where 0 represents all the parameters the network), the Hessian matrix H is defined as:
H = A L())
This is symmetric matrix that captures the second-orde derivatives (curvature) of the loss function with respect to the parameters.
22

---

Published as a conference paper ICLR 2026
B.7.1 PRINCIPAL CURVATURE
The principal curvature is defined as the maximum eigenvalue Amax of the Hessian H. In other words, it's the largest value the set of eigenvalues (Ai, A2....,,,,, Aa) where is number of parameters (or the dimension of H). The largest eigenvalue indicates the steepest direction of curvature. If you move in the direction of the corresponding eigenvector (the principal direction), the loss increases most rapidly. We employ the Power iteration algorithm to find the dominant cigenvector, and thereby the dominant eigenvalue using the Rayleigh Quotient:
Ama
The principal eigenvector is the direction in parameter space coresponning to Amix- This is the steepest direction any movement along that eigenvector direction leads to the largest second- recond-order change in the loss.
B.7.2 EFFECTIVE RANK
To quantify the intrinsic dimensionality of the loss loss-landscape curvature in given layer, collect per- per-sample gradients (gr pm _ and stack them into a matrix G [g1,... ......... ..m] _ddxm We then form Gram matrix
M G E RTx..
compute its singular values σı 02 2 am. 2 0, and define the effective rank at threshold E (0. 1) by
Σイ Σ 2
erank, (G) = min
Effective Rank represents how many independent directions of principal directions that account for least a fraction T of total "energy" the gradient subspace Kumar et al.] (2020). Therefore, effective rank tells you the number eigenvectors (or directions) that are necessary to capture a specified fraction (for example. 99%) of the total curvature energy of the Hessian. In other words, it indicates how many independent directions contribute significantly to the curvature. If the effective rank is low, then most of the curvature is concentrated in just few directions; if it's high, then the curvature is spread out over many directions.
C EXPANDING ON CASE STUDY 1: NEGATIVE-SLOPE 'GOLDILOCKS ZONE'
C.I FAILURE MODES FOR SLOPE MAGNITUDE.
We continue investigating the primary drivers of functional plasticity loss. As provided in the main section results, the correlation between Final Average Accuracy (ACC) and dead unit fraction remains robust and significant (Pearson r = -051, p- = 8.2x 10-25), as seen in Fig. C1 right. In contrast, the correlation between ACC and final scaled gradient norms is negligible (Pearson r -0.1 11 -0.11,
= 0.029) (Fig. CT left). Scttings that maintain dead unit fraction below approximately 8% consistently achieve high ACC (27-33%) whereas dead unit rates above 25% lead to ACC collapsing 20% or lower. This indicates that while gradient magnitude is factor, the rise in dead units (gradient starvation) is a more direct and reliable indicator of plasticity loss than the raw magnitude of gradients, which can small in well-adapted, flat minima or large in failing networks with high curvature. The fact that dead units do not account for all performance variance aligns with our finding in Section that landscape curvature drives failure at high slopes, distinct from unit death. The results highlight that plasticity is not about maximizing any single statistic but about achieving a delicate balance, primarily tuned by the negative slope characteristics, to ensure unit survival while maintaining benign loss landscape.
C.2 ADAPTIVE, GRANULAR SLOPES ARE USEFUL- -UTUUUL--TTTT NEED GUIDANCE TO STAY IN-BAND
As seen in Sec. PReLU-N, with its per-neuron Iearnable a,, demonstrates this adaptability. Fig. (left) shows the dynamic evolution over time of how PRcLU-N learss heterogeneous a, values, idcally we will observe many neurons dynamically adjusting their slopes into beneficial range (c.g., 0.3-0.6, approaching the Goldilocks zonc), while others remain near zero required by their local input statistics. However, PReLU-N configuration (with default
23

---

Published as conference paper ICLR 2026
leakyyrelu
Jeaky
H5: Plasticity Units
Plasticity Loss Gradient
Pearson r -0.51 (p=B.2e-28))
Spearman p: -0.16 (p=0.0014)
Pearson -0.11 (p=0.029) Spearman p: 0.31 (p=2 (p=2.9e-11) 9.-e-10)
Final Average Accuracy 1%6
20
Avg Dead Unit Fraction (End Training)
Avg Scaled Gradient Norm (Last Task) (Smailer Decay)
Figure Cl: Person (r) correlation between Plasticity Loss and diverse metrics to evaluate the primary drivers of such phenomenon for some activation functions evaluated across Case Study 1 (see Section Units; Right: Average Gradient Norm.
Left: Dead
initialization 0.25) undershot the empirically optimal = 0.7 for fixed Leaky-ReLU but its performance surpassed PReI.U-G/L. which tended towards slope decay and lower ACC (o evolution for PReI.U-//L S-G. shown in Figs. [3). All
PRcLU scopes (Ncuron, Layer, Global) evolve towards values outside the Leaky-Rel Relky-RRl.U--amilyy 'Goldilocks Zone'.
While not optimal, this might suggest neuron- wise adaptability valuable trait, potentially enhanced further with optimized initialization or per-parameter learning rates if the learning can be guided towards the 'Goldilocks zone'. Which also opens the question of properly finding such zone for different experimental settings.
Evolution Mean Alpha PReLU-N
Evolution Fraction Small Alphas (<t) for PReLU-N Layers
80
0.6
sctivotion
Bard
Fraction of Apphss (%6)
60
Moan Alpha (0)
0.4
20
250
1000
1250
2000
Epoch
Global Epoch [Tasks 1-20, 100 epochs/task)
Figure C2: PReLU-N per-neuron learnable a. Left: Distribution of all individual neurons' per layer (activation.N) and the pre-defined 'Goldilocks zone' representing the best- pest-performing value in Leaky-Re Right: Fraction of a's < indicating values for which the post-activation will be 'saturated' (too small to produce significant changes).
C.3 SATURATION FRACTION AS METRIC FOR 'DEAD UNITS'.
Regarding Saturation Fraction used in Fig. T we acknowledge that recent works, such as Sokar et al. (2023) or Liu et (2025). highlighted the 'dormant neuron phenomenon', and 'GraMa' (Gradient Magnitude Neural Activity Metric) in deep reinforcement lcarning, proposing valuable metric based neuron's average absolute activation (or gradient) relative to its layer-neuron connections identify consistently underutilized units. While this provides generalized
24

---

Published as conference paper ICLR 2026
PReLU-L (per-layer) evolution
PReLU-G (global) evolution
Alpha (a) Value
3C3 VOTSOLO
vatton11
OCt vation.2
vatien_
Coldiacks
250
500
750
1000
1250
1500
1750
2000
250
500
750
1000
1250
1500
1750
2000
Global Epoch
Global Epoch
learnable a. Left: PReI.UGG. Distribution of all neurons' a for the whole network (1 learnable parameter shared across all layers); Right: PReLU-L. PReLU-G. Distribution of all layers' parameters for the whole network (1 learnable parameter per layer).
Figure C3: PRe..U-GG..er--eeee
approach 'dead units', investigation into diverse array of activation functions- each with unique output ranges and saturation characteristics (e-g., ReLU's unbounded positive output versus Tanh's strict [-1, 1] bounds, or ELU's negative saturation platcau) necessittee more concrete definition of what constitutes 'saturated' or "effectively dead' state based on post activation values. Therefore, while sharing the goal of identifying inactive units, our saturation fraction metrics employ criteria specific each activation function family (such proximity +/-1 for Tanh, near- output for ReLU, low output magnitude relative hatch statisties for functions like Swish Leaky ReLUU. This specific approach allows us to more precisely capture the distinct ways different activation architectures can of mnrespoonsvennss under duress, rather than applying a single relative threshold across functions with fundamentally different output scales and properties. See code for the full criteria used for each activation function.
or avoid states
D EXPANDING ON CASE STUDY 2: SATURATION- THRNIOOO STRESS TEST
D.1 STREES PROTOCOL
Let r {1.5, 0.5, 0.25, 2.0) be the set of shock amplitudes and let Ci user user- defined cycle length (we use C; = 10 for results in the main body Sec, 4.. During training we step through the epochs of every task as
Tx. mod G 0,
(L) "N
(c(1)
otherwise,
where the index & is advanced cyclically -+ (k- 1) mod [1'] each time shock epoch occurs. Thus every Ct epochs we devote exactly epoch to scale-shock whose value alternates 1 1 1.5 + 115.1--050000 0.25 5 1 2.00 The multiplicative factor applied after all layers and before its non-line con-linearity; other epochs run with y
D.2 DERIVATIVE- FLOOR RULE.
The data presented in Figure 2 provides strong support for the idea which posits that activation functions with non-zero derivative floors offer superior desaturation and recovery dynamics.
Fig. (left) reveals that, as expected, stronger "expanding" shocks (7=1.5,2.0) generally induce a higher overall saturation impact (AUSC) than "shrinking" shocks (y=0.25.0.5) across all floor types.
Critically, activations in the "Non-Zero Floor" category consistently demonstrate the most effective minimization of
AUSC, maintaining the lowest values particularly under expanding shocks. Conversely, "Zero Floor" activations exhibit
25

---

Published as a conference paper at ICLR 2026
the highest AUSC, indicating greater susceptibility to prolonged saturation. The "Effective Non- Floor" group shows a mixed response, with AUSC values sometimes higher than "Zero Floor" for certain shrinking shocks (e.g.. 7=0.5 but better than "Zero Floor" under strong expansion.
This pattern of robustness for "Non-Zero Floor" activations is further evidenced in their SF recovery capabilities, shown in Fig. (middlee) and Fig. _ (right). While successful recovery times tend to be quick (around epoch slightly more) lor stronger shocks (7= 1.5,0000 across most types that recover, the "Non-Zero Floor" category stands out by also maintaining the lowest non- non-recovery rates, especially for expanding shocks where it approaches 0-5%. In contrast, "Zero Floor" activations not only take noticcably longer to recover SF during shrinking shocks but also suffer from extremely high non-recovery rates across all shock conditions. The 'Effective Non-Zero Floor' group (typically smooth-tailes functions) demonstrates rapid SF recovery Fig. 2 high non con-recoerry rates, particularly akin to 'Zero Floor' types during shrinking shocks, and show less consistent AUSC improvements compared to "Non-Zero Floor' activations. This indicates their negative tail mechanisms, despite providing average non-zero gradient, may not consistently prevent saturation or ensure recovery across diverse shocks.
(middle) if recovery occurs. However, they suffer from
In conclusion, "Zero Floor" activations are clearly detrimental to saturation resilience. While "Effective Non-Zero Floor"' functions offer fast recoveries when successful, is the presence of consistent and sufficiently large "Non-Zero Floor"' that most effectively minimizes overall saturation impact and ensures reliable, rapid recovery from naturation-inducing shocks."
D.3 TWO-SIDED PENALTY.
We hypothesized that activations that saturate on both sides (Sigmoid. Tanh, Swish/GELU at very low 99 will accumulate 50% more saturation and take 2 50% longer recover than one-sided families. The data provides nuanced support for this. As seen Fig. B] "Two-Sided" functions indeed accumulate more saturation, evidenced by having the highest average Peak SF Fig. 3 (left) and high AUSC values 3 (right). While their successful SF recoveries are very fast around one epoch Fig. 3 (middle)-this is severely counterbalanced by a very high non-recovery rate of 49 49.83% (data not shown in figure). This high failure rate means that, in practice, they are far less reliable at desaturating than "One-Sided (Kink)" activations, which had non-recovery rate of 13.30%. If non-recovery is considered an infinite recovery time, then "Two-Sided" functions effectively "take longer to recover" on average due to frequent outright failure. Interestingly, "One-Sided (Smooth)" functions also exhibit a high non-recovery rate (48.93%) barely surpassing that of "Too-Sided" functions, despite their successful recoveries also being very fast. This suggests that while their smooth negative tail can allow for quick desaturation if conditions are right, they are also prone to getting stuck in saturated state.
Therefore, the penalty for two-sided saturation is evident in hoth the magnitude of saturation experienced (Peak SF, AUSC) and the overall reliability recovery (high non-recovery rates). The 'One-Sided (Kink)" group, while having a tail of longer successful recovery times, is actually the most reliable at achieving recovery.
D.4 PERFORMANCE RECOVERY TIME ON CASE STUDY
To assess the functional recovery of the network after pre-activation shocks, we measured the performance recovery time 795, defined as the number of epochs required to regain 95% of the pre-shock validation accuracy on the current task. This metric provides insight into the immediate resilience of the network's learning capability for the task hand when subjected to sudden internal perturbations.
Overall, the rate of complete performance non-recovery (failure to reach 95% of pre-shock current-task accuracy within the observation window) was very low across all experiments (1. 11.14)). This indicates that, in most instances, the model's 's ability perform the current task bounces back effectively after the applied shocks. Figure DT illustraass the mean Tes for these successful recoveries, grouped by activation 'Floor Type" (left) and 'Sidedness" (right), across different gamma shock intensities.
For mild to moderate expanding shocks (y=1.5) and all shrinking shocks (7=0.25,0.5), the mean Typ is consistently low (around 1.2-1.3 1.3 epochs) and remarkably similar across all floor types and sidedness categorics. This suggests that for less extreme shocks, most activation functions allow for rapid recovery of performance the current task. However, differences emerge under strong expanding shocks (7=2.0):
26

---

Published as conference paper ICLR 2026
Mean Epochts to Regain 95% Port iPerf Recovery, Tiiee)
Performance Recovery Time (T95) vs. Gamma
Performance Recovery Time (Tas) vs. Gamma
Floor Type
2.5
Zero Floor
One-Siden (Kink)
(Succdsfuul Recoveries Only)
Non-Zero Floor
One-Sided (Smooth)
Effective Non-Zero Floor
Two-Sided
2.0
1.5
1.0
0.5
0.0
0.25
0.5
1.5
2.0 0.25
0.5
1.5
2.0
Gamma
Gamma
Figure D1: Functional performance recovery time 795 versus gamma shock intensity. Grouped by Floor Type (Left) and Sidedness (Right), showing mean epochs for successful recoveries. Most activations demonstrate rapid recovery from mild moderate shocks; however, under strong 2=2.0 shocks, TWoo-Sided' functions are notably fastest and most reliable (0% non-recovery), while Non-Zero Floor' and 'One-Sided (Kink)' types show slower recovery. (Overall performance non-recovery rate: 1.14%)."
Impact of Floor Type (at 7=2.0): 'Non-Zero Floor' activations, despite their strong SF recovery characteristics, surprisingly exhibit the longest average (=3,4 epochs) for performance and also contribute most to the few performance non-recoveries observed (2.85% non-recovery rate for this group at 2=2.0). In contrast, 'Effective Non-Zero Floor' (s:1.55 epochs 7es, 0.36% non-recovery) and 'Zero Floor' (s11.9 epochs 795, 0.74% non-recovery) activations recover current-task performance more quickly under these strong shocks.
Impact of Sidedness (at 7=2.0): Most notably, 'Two-Sided' activations (e.g.. Sigmoid, Tanh) demonstrate exceptional current-task functional recovery. They consistently recovered performance (0% non-recovery rate in your stats) and did so fastest on average (x=1.1 epochs). 'One-Sided (Smooth)" activations also performed well. One-Sidde (Kink)" activations showed the slowest average Tyo and the highest performance non-recovery rate among sidedness categories for these strong shocks.
It is crucial to note that this Tuo metric reflects the immediate recovery of performance on the task currently being trained. While it indicates certain resilience transient internal shocks, it does directly measure the long-term impact catastrophic forgetting or the overall ability learn sequence tasks effectively.
The ultimate test an activation function's suitability for continual learning lies metrics like Final Average Accuracy (ACC) across all tasks and Average Forgetting (AF), evaluated at the end of the entire training sequence. It is in these end-to-end CL. metrics that more significant and often different disparities between activation functions emerge. For instance, while "Two-Sided activations show rapid current-task performance recovery here, their known issues with vanishing gradients and saturation might still to poorer overall ACC or higher forgetting in full scenario. The current res results offer valuable piece of the puzzle regarding short-term resilience, but the broader impact on plasticity and stability across the entire sequence of tasks remains best assessed by holistic CI. evaluation metrics.
EXPANDING ON CONTINUAL SUPERVISED LEARNING EXPERIMENTS
evaluate five supervised continual image elassification benchmarks spanning shift types: input distribution shift (Permuted MNIST. 5+1 CIFAR. Continual ImageNet) concept shift (Random Label MNIST, Random Label CIFAR). Across all settings, training proceeds as sequence tasks without task iise-ide signals: the model is never told when task switch occurs. Within each task, the learner receives mini-batches for a fixed duration (specified below) and is updated incrementally with cross cross-entropy the arriving batches. Summary of experimental settings shown Table ET
Following Kumar et al. 2023
27

---

Published as a conference paper at ICLR 2026
Permuted MNIST Goodfflllo et al. (013)) (input shift). We first sample fixed subset of 10,000 images from the MNIST training set. Each task is defined by drawing a new fixed random permutation over pixel indices and applying it to every image in the subset. The permutation is constant within task and independent across tasks, Each task presents exactly one pass (1 epoch) over its 10.000 permuted images mini-hatches of size 16; the next task then begins with a new permutation. We train for 500 tasks total. This setting induces strong input-space remapping while preserving label semantics within tasks, isolating rapid adaptation to input shift.
Random Label MN et al. (202)) (concept shift). We fix a subset of 1,200 MNIST images once. For each task, we generate a fresh random label for each image in this subset, leaving the inputs unchanged but altering the input to label mapping. To encourage memorization under an arbitrary target function, the model is trained for 400 epochs per task with batch size 16. After 400 epochs, a new task arrives with an independent random labeling; we run 50 tasks sequence. Inputs are identical across tasks; only concepts change, directly probing plasticity versus interference.
Random Label CIFAR (concept shift). Identical protocol to Random Label MNIST, but using images drawn from CIFAR-1O. 10. We again use a fixed subset of 1,200 images and re-assign random labels independently per task. Training uses 400 epochs per task, batch size 16, for 50 tasks. This mirrors the concept-shil regime higher-variability image domain than MNIST.
5+1 CIFAR (input shift with alternating difficulty).. Tasks are constructed from CIFAR-100 and alternate difficulty: even-indexed tasks are hard and odd-indexed tasks are easy. A hard task contains data from 5 distinct classes with 2,500 examples total (500 per class); an casy task contains data from a single class with 500 examples. Classes do not repeat across the sequence, ensuring
overlapping exposure. Each task lasts 780 timesteps, which corresponds approximately epochs on the hard task data set with batch size 32; easy tasks use the same 780 time step budget for consistency. We evaluated performance on the hard tasks only (single-class tasks are near the ceiling for all activation functions). Alternating input diversity stresses hoth rapid adaptation (when diversity spikes) and retention across shifts, serving targeted stress test for plasticity-loss mitigation.
Continual ImageNet Dohare et al. (2024) (input shift). Each task is binary classification problem between two distinct ImageNet classes. For every task we draw 1,200 images total (600 per class) and noop-s to 32x32, following Dohare et al. (2024), to reduce compute while maintaining semantic variability. Classes do not repeat across tasks, yielding clcar, non-overlapping episodes of input shift. We train for 10 epochs per task with batch size 100 and report task accuracy. Despite down-sampling, the setting retains ImageNet-level variability while enabling precise measurement of adaptation and retention under non-reused classes.
Tar-TTask Data Size
Benchmark
Batch
Epochs
Timesteps
Tasks
Permuted MNIST
10,000 images
16
1
625
500
Random Label MNIST
1,200 images
16
400
30,000
50
Random Label CIFAR
1,200 images
16
400
30,000
50
Hard: 2,500 images (5 classes, 500/class)
Hard: R 10
5+1 CIFAR
32
780
15
Easy: 500 images (1 class)
Easy: Rs 50
15
Continual ImageNet
1,200 images/task (600/class)
100
10
120
500
Table E1: Hyperparameters and schedule per benchmark. Timesteps denote parameter-update steps (i.e., mani-batches) within a task. For 5+1 CIFAR, fixed timestep budget per task implies approximate epochs depending on data size.
Notes. (i) In 5+1 CIFAR, classes do not repeat across tasks; tasks alternate easy/hard. 780 timesteps z 10 epochs on the hard set (since 2,500/32 z= 78.123 batches/epoch) and zs 50 epochs on the easy set (since 500/32 z 15.625). (ii) Continual ImageNet, images are downsampled to 32x to reduce compute; classes do not repeat across tasks, (iii) Timesteps are computed as the number of mini-hatches per task.
28

---

MNIST) favor Published as conference paper ICLR 2026
E. EXPERIMENTAL RESULTS EXPANSION
Here we expand the empirical comparison of activation functions across five continual supervised benchmarks. Table reports Total Average Online Task Accuracy. Two broad patterns emerge. First, rectifiers with learnable randomized negative branch dominate: leaky-RRe I.D, RReLU, PReI. Smooth-leaky. and randomized variant consistently outperform Rel.U-otten by large margins (c.g., CIFAR 5+1: Rel. 4.76 vs. Rand. Smooth-leekk 57.01). Second, smooth rectifiers (Swi (Swish/Se sh/ LU) also fare well, but tend to trail the best "leaky-family" members on the harder settings. On the -etooiiiaatioffienndl random-label tasks, several of these parametric/leaky activations saturate near 100%, whereas saturating si sigmoi and tanh struggle CIFAR with random labels consistent -o with their known optimization brittleness under distributional churn. Overall, the strongest single performer Rand. Smooth-Leaky, with Smooth- RReLU, and Leaky-ReIU close behind.
Table E2 clarifies why these familics succeed by showing the optimal shape parameters and learning rates. A striking regularity is 'Goldilocks zone' for the negative linear sides: When an activation exposes slope (or an effective initial slope) the negative branch, the best values typically land between their respective best-performing ranges. For it tends to be between 0.6 and 0.9, which was initially reported in Section 3 However, this is not a rule, and optimal Goldilocks zone' might vary between activations and settings. Nonetheless, this tends holds across Leaky-ReLD PROLU, Smooth-Leate bounds (important because that average initializes the effective leak). For these activations. that within that range tend to also have higher accuracy values than the ones that outside the preferred range. Intuitively, this regime prevents dead units and preserves gradient flow without collapsing the asymmetric gating helps continual adaptation. We also observe gentle I.R shift: simpler datasets PermuteedRandome 10 while CTFAR 5+1 prefers 10 -, matching the increased optimization stiffness there.
and Rand. Smooth-Leaky and even for RReLU when considering the average of its
also observe that those
Permuted MNIST
Random Label MNIST
Random Label CIFAR
Avg Online Task Accuracy (%)
90
80
50
50
40
20
100
200
300
400
500
10
20
40
50
10
30
50
CIFAR
Continual ImageNet
Aog Online Tesk Accuracy (%)
CeLU
ReLU
80
Deep Fourler
Selli
Sigmooi
GelUI
Smooth-Leaky
Leeky ReLU
Swiglu
60
PReLU
Swish
20
Rand. Smooth
Tanh
12
15
100
200
300
400
500
Figure E1: Comparison using Total Average Online Task Accuracy across all five Continual Supervised I Learning Benchmarks for all activation functions. See Tab. Optimal hyperparameters can seen Tab.
2 for the best total average online accuracies.
E2
29

---

Published as a conference paper at ICLR 2026
Random Label MNIST
Random Label
Permuted MNIST
CIFAR 5+1
Continull ImageNett
Activation
CIFAR
0.001
0.0000
[| 0,0001
[| |10,000
| 0.0001
Leaky ReID
0.6 |0901
0.001
0.6 0.001
0.4 [ 0.001
0.5 ] 0.001
sigmoid
0.001
[| 000001
0.001
[| 000000
|0001
Tanh
0.001
0.00001
0:0001
0.0001
000001
RReZZ
10.6, 0.81 0.001
[0.125, 0.533] 0.0011
10.6, 0.81 || 0.001
2.,673 0.000
[0.6. 0.8| 0.000
nzuron, 1.2 0.001
neuroo, - 0.1 1:001
global, - 0.65 0.0001
globbd, o -0--0 0.99
[| 0.0001
nesioon, - -0.55 000001
Swish (9120)
0,05 000001
0:01 | 0.001
0.01
0.1 0.001
0,05 0.001
Colu
0.5 0.001
0.8 0.000
0,05 | 0.001
1.0 0.0001
1.0 | 0.001
CelC
3.6 0.001
3.6 [| 000000
2.0 000000
3.3 0.001
3.3 0.001
GLU
1.0 0.001
3.6 | 0.0001
3.6 0.0001
3.6 [ 0.001
1.0 [ 0.001
S-1U
1.0 00001
3.0 | 0.0001
3.7 [| 0.0001
577 0.001
3.7 0.001
CRoLd
0.001
0.001
0.000
0.000
Rat ional
A, (5, 4), Leaky-Rel.11 0.001 0.001
D. 4), Tanh || 0.0001
A. (5, 4), Tanh 0.001
B. (5, 4). Swish || 0.001
C. 4), Rel.1 || 0:001
SwIGIO
0.0001
[.0001
000000
[000000
000001
Deep Pourier
0.001
[| 0.000
0.001
0.0001
|
Smooth Leaky
0.1. 0.3, 0.3 [| 0.001
0.3.
0.1 | 0.001
0.3, 0.5. 0.66 0.001
D.1. 5.0,0. 0.9 0.001
3.0. 2.0. || 0.001
Rand.
Sneath
0.3. 0000
0.3,06
3.0, 000011
0.5.
2.673 0.00
0.5. 0.5.0. 0.3 0.001
Table E2: Optimal Hyperparameters for each activation function in each Continual Supervised Benchmark Problem. Represented as activation function shape parameter on the left side of the symbol and the learning rate to the right. A dash (-) indicates that such activation function uses the unique or baseline parameter (e.g., ReLU does not have any shape-controlling parameter since is linear on the positive right side and 0 on the negative left side). The Total Average Online Accuracy reported for these optimal hyperparameters can be seen in Table 2 PReLU's a indicates initial parameter value. smoothh-Leake triplets indicate c, P, a, while Rand. Lmokth-Leaky indicates e, p, and bounds [1, u]. The tuple of values from Rational indicates Version, ((P), (Q), Function Approx. where (P) and (Q) are the numerator and denominator degrees respectively of the polynomial.
E.2 ANALYSIS OF GENERALIZATION GAPS
Table F33 reports the Generalization Gap (GAP), calculated as the difference hetween the final test accuracy on specific task (after training on it) and the average online training accuracy during that same task, averaged across all tasks in the sequence. In this online continual learning setting, a positive gap is the expected, hcalthy outcome. Because the "online training accuracy" averages performance from the very first batch (where the model is ignorant) to the last, it naturally underestimates the model's final capability. A positive gap (Test > Online Train) therefore quantifies the "learning gain": it confirms that the final, converged model performs better on held-out data than average performance while learning. On the other hand, a negative gap (Online Train > Test) is a critical diagnostic for overfitting or generalization failure. This occurs when the model fits the incoming stream of training batches but fails to retain that performance when evaluated on the static test immediately after. This distinction highlights key trade-off. For example, while Deep Fourier Features (DFF) demonstrate high plasticity (high online accuracy) in our main benchmarks, they exhibit large negative gaps on complex input-shift tasks (e.g.. 51.24% on CIFAR 5+1, 221.021.00 22.001 % on Continual ImageNet). This indicates that DFF's "adaptive linearity" comes at the cost of significant overfitting to the immediate training stream. In contrast, Rand. Smooth- Leaky achieves a far superior balance. On Continual ImageNet, it effectively closes the gap (-1.7%)). and CIFAR 5+1, it reduces the gap by nearly half compared to DFF (-29.00 29.667% vs. -51.24%). However. if we look directly at the performance values in Table 2 we see that DFF achieves 72.29% vs Rand. Smooth-Leaky, 57.0 57.01, which points to the same idea of conflation between trainability vs generalizability suggested in Section with loss of plasticity.
with regard to
We believe that in this case, this metric captures a different phenomenon, overfitting the test set. This is distinct concept from the specific challenge we are diagnosing in our RL analysis. That is the reasoning why for supervised benchmarks, we instead adopted the standard metrics for generalization in the continual learning community. As detailed in Appendix (rnrendixB.66.]] we use Total Average Online Accuracy (TAOA) (which follow the setup of Kumar ct al. (2023) for our main online benchmarks O which is common standard (Ghunaim et al. (2023) Prabhu et al. (2023);(Cr et al. [2021]. and Average Accuracy (ACCz) ) for our class- dlas-incrrmenntl case studies (Section and Scction are the accepted measures of test-set performance in these CL settings.
These metries
30

---

Published as conference paper ICLR 2026
Activation
Permuted MNIST
Rand. Label MNIST
Rand. Label CIEAR
CIFAR 5+1
Continual ImageNet
ReLU
9.37
3,98 1.52
3.52 1 2.51
23,42 1 3.85
5,49
0.17
Leaky-ReLU
5.33 = 0.03
8.20 0.29
167 1001
-0.81 1.42
-0.24
0.088
RReLU
5.19 = 0.02
6.83 0.11
2.69 10.01
-2.599 2.89
-0.34 0.117
PReLU
7.39 1 0.02
6,48 1: 0.90
3.66 1.331
23.11 1.0.88
0.87 1 0.33
Sigmoid
10,57 0.05
17.56 0.39
15.06
18.18 1.666
4,83 0.35
Tanh
12.49 0.10
15.25 0.21
11.57
25.97 0.55
4.77 +.332
Swish
6.00
0.39
10.11
3.38 1:2.33
1.72 1721:0018
GeLU
9.26 1.06
9.13 0.566
15.21 1..59
29.36 1..42
0.74 2.50
eLU
8.76 0.05
16.48 1.96
8,98 2.93
-2.199 #2.11
2.45 0.23
CeLU
7.41 + 0.05
22.69 + 0.88
3.45 0.40
28.31 +.00
1.67 + 0.24
SeLU
8.32 1 0.15
20.38 10664
16.47 1.7.7
-20.77 12.14
1,15 1 0.30
CReLU
7,42 = 0.02
10.33
6.74 1043
9.68 1.47
0.85 1010
Rational
6.22 + 0.06
8.09 0.95
8.32 + 0.23
20.80 6.144
2.01 0.32
SwiGLU
10.22 0.022
2.38 1.44
17.25 1.32
25.01 12.9
5.47 1.900
Deep Fourier
7.03 = 0.04
4,41 1.15
1.63 1 1,40
-51.24 2209
-2101 1.71
Smooth-Leaky
6.42 + 0.01
8.27 + 0.39
1.66 + 0.01
-16.97
+ 2.05
-0.44 0.31
Rand. Smooth-Leakk
5.98 1 0.02
6,47 0.09
1.59 1 0.00
-29.67 2.42
-1.47 1 0.20
Table F3: Generalization Gap (GAP) in percentage (%) across five datasets. Values are reported as Mean Std. The metric calculated per-task as GAP, TestAcc,,nn TrainAcc then averaged across all tasks. Positive values indicate healthy learning (the final model generalizes better than its average performance during training). Negative values indicate overfitting (the model performed better on the training stream than on the held-out test set). Higher is better. Best performance per dataset is bolded.
E.3 CONTINUAL LEARNING INTERVENTIONS TARGETING Loss OF PLASTICITY
We evaluate the activation functions used throughout this paper study the effects of being augmented with three standard continual-leaming (CI.) algorithms that explicitly target loss of plasticity: Iz-Innt regularization Kumar et al.) (2023 Self-Normalized Resets (SNR) Farias & Jozefiak (2024). and Elastic Weight Consolidation JEWC)]Kirpattriic (2017]. These three methods are applied orthogonally to the network architecture and activation function choices. They constrain refresh parameters, allowing us to quantify the extent to which activation design still contributes to plasticity when strong CL mechanisms are present.
al.
Le Regularization to the Initial Parameters. I2-Init modifies conventional weight decay by intro- ducing quadratic penalty that pulls the network parameters (0) back toward their initial values (0o). rather than toward zero. The total loss (L) is augmented by Ln-Init regularization term (Crami):
Init:
Lam
A
0o112
The rationale is that the randomly initialized network resides a relatively "plastic" region the parameter space, characterized by well-hehaved gradients and non-saturated units. As training progresses across tasks, 0 may drift into "stiff" regions where gradients less effective or highly anisotropic. In-Init counteracts this drift by softly anchoring 0 back toward the original, tigh-plattcctty configuration. In experiments, Arztoa is tunable hyperparameter, and the penalty applied throughout training.
Self-Normalizee Resets (SNR). SNR is reset-hased method specifically designed mitigate neuron-level loss plasticity. Instead continuous regularization, SNR monitors the firing statistics individual units over time. tracks an estimate of the expected activation frequency and computes an inter-activation time distribution for cach neuron. If neuron remains inactive for duration that is statistically unusual relative to its established baseline, SNR determines the unit effectively dormant and triggers reset. This reset re-initializes the neuron's 's incoming weights and corresponding optimizer state. core hyperparameter a percentile threshold (1)). A reset triggered when the probability, under the tracked firing-rate model, of observing an inactivity stretch at least as long as the current one falls below n,. Higher implics more aggressive resets (quicker refreshment of neurons). Lower corresponds to more conservative application of the reset mechanism. SNR intuitively injects fresh, plastic units into the network whenever existing ones become statistically improbable "dead" units.
31

---

Published as a conference paper at ICLR 2026
Elastic Weight Consolidation (EWC) EWC is a classical regularization based CL method that addresses catas- trophic forgetting by selectively restricting modifications to parameters deemed important for previously encountered tasks. Following the completion of each task, an approximate Fisher information matrix is computed to estimate the importance of each parameter. During subsequent task learning, the overall loss (C) is augmented with quadratic penalty term (Cawe):
LEWC = AEWC I -0,)
Where: F. is the i-th diagonal element of the Fisher information matrix, quantifying the importance of parameter 0..0; are the reference parameters (e.g.. those learned after the previous tasc).AEwc is the hyperparameter controlling the regularization strength. This penalty increases the cost of updates along directions crucial for past performance, thus preserving stability while permitting movement in less constrained directions essential for new task plasticity. EWC serves as a strong CL baseline that explicitly manages the staiiity-plaastciiy trade-off.
Ablations. By applying this complementary suite of CL mechanisms, can rigorously assess whether the effects arising from activation function geometry and design persist, and how strongly they interact with paramelerr-constrainnng or refreshing strategies. For all methods we report Total Average Online Task Accuracy (%) averaged over 5 independent runs. Values are reported as mean + standard deviation (SD). For each activation, we reuse its best hyperparameters from the main experiments (see Table and sweep small log- log-spaced grid over the CL-method hyperparameters, reporting the best value in each column. Vanilla indicates original results. For each CL intervention (L2-Init SNR, EWC) swept a small grid over its main hyperparameter and report best values per activation and dataset in Table E11 Table BI0 and Table
E.3.1 DISCUSSION OF RESULTS
Activation Choice Defines the Plasticity Ceiling, A consistent hierarchy emerges across all benchmarks: while CL interventions most notably L2-Init- can mitigate plasticity loss in fragile activations like ReLU, they cannot compensate for the geometric limitations of the activation function itself. For instance, in the challenging Random Label regimes (Tables B5 and Vanilla ReLU suffers total collapse (+2 20% accuracy), effectively requiring L2- Init to function at all. In contrast, our proposed Rand. SmrothhLeakky demonstrates inherent robustness, achieving high performance even without interventions, and reaching the highest overall performance ceiling when augmented with I.i (e.g., 95.20% on Continual ImageNet). This suggests that optimal plasticity, when coupled with CL interventions, requires a synergy: an activation function that maintains gradient flow (Smooth-Leaky) combined with a regularization method that maintains a favorable operating point (Lit-
Specific Activation Profiles. We observe distinct specialization among prior approaches. Deep Fourler exeels in tighfrreueency permutation and noise tasks (T6), effectively solving the task without intervention (96.21%), yet struggles to generalize to natural image distributions (Table ES} where it lags behind smooth non-monotonic functions. CReLU acts as a reliable "stabilized ReLU," consistently outperforming the baseline but often hitting a lower asymptotic limit than the smooth variants. Rand. Smooth LLaky effectively bridges this gap, matching Deep Fourier's robustness noise tasks while dominating on natural data, indicating it captures the necessary curvature for diverse task boundaries.
The "Constraint- DoonstranttPlasticttt""" Paradox. A critical anomaly is observed in the CIFAR 5+1 benchmark (Table H where CL. interventions actively degrade the performance of strong activations. While I.2 aids ReI.U, it catastrophic reduces the aecuracy of Deep Fourier (from 72.29% to 20.40%) and Rand. Smooth Leaky (from 57. to 34.56%). This suggests regime where the optimal solution lies far from the initialization (0o). By anchoring parameters to Oo. L2-Init prevents the significant semantic drift required for this specific task shift. Notably, Vanilla Deep Fourier achieves the highest performance in this setting, proving that in sconarios requiring extreme adaptation, the native plasticity of the activation function is superior to external constraints. Conversely, methods like EWC consistently underperform across most benchmarks, confirming that stability focused regularization is often antithetical to the requirements of plasticity-heayy regimes.
32

---

Published as conference paper ICLR 2026
Activation
Vanilla
+L2-Init
+SNR
+EWC
ReLU
78.85 = 0.06
93.36 + 0.38
1..10
53.85 2211
Leaky-ReLU
84.14 = 0.01
91.60
0.47
58.42
Sigmoid
76.96 = =007
89.88
#033
58.94 #3.33
Tanh
70.32 = =034
86.12
#032
59.35
RReLU
83.95 0.02
90.31
0.44
88.91
+0333
57.95 +5.19
PReLU
82.62 = 0.05
92.49
0.59
88.72
+0338
58.87
1.47
Swish
83,41 = 0.03
90.40
0.63
89.24
#0.27
60.50
2257
GeLU
78.97 = 0.09
93.24
0.36
85.72
+0.40
56.43
3318
CeLU
82.93 = 0.04
93.06
0.24
89.51
#0.35
62.10 #5.64
eLU
80.50 = 0.09
93.56
0.40
#0.42
57.94 1.65
SeLU
80.43 = 0.16
93.67
0.58
87.22
#034
54.88 =3.20
CReLU
82.66 = 0.04
93.22
+ 0.58
+.44 89.58
45.39
4.07
Rational
80.08 0.05
87.96
0.56
10.17+0.00
50.10
2.30
SwiGLU
77.69 0.26
88.93
= 0.23
#0.30
74.95 1.72
Deep Fourier
83.69 = 0.04
93.76
0.18
90.68
#0.50
38.98 #322
Smooth-Leaky
84.03 0.02
92.19
0.43
89.91
60.16 3.30
Rand. Smooth-Leaky
84.26 = 0.02
93.92 0.37
90.10 + 0.48
61.88 2.20
Table E4: Effect of activation function with series of Continual Learning algorithms Permuted MNIST.
Activation
Vanilla
+I.2-Init
+SNR
+EWC
ReLU
20.03 2.46
100.00 0.00
14.65
1.94
12.13 0.74
LeaKy-ReLU
91.53 0018
99.90 0.18
100.00 0.00
22.77 1.80
Sigmoid
79.59 #0.75
99.42
0.48
91.43 #1553
82.10 12.65
Tanh
63.40 0.12
98.67
+ 0.33
92.63 1.45
13.07 11.11
RReLU
93.10 0.002
100.00
0.00
99.95 0.07
77.85
25.94
PReLU
92.67 0.23
100.00
0.00
100.00
0.00
85.10 #33.32
Swish
67.73 #0.46
89.00
5.14
100.00
0.00
11.49
GeLU
38.79 +095
99.90 + 0.14
15.40
1.55
36.57
#3.11
CeLU
37.16 0.90
100.00
0.00
20.98
+ 1.96
27.27
1.03
eLU
84.23 0.70
100.00
0.00
99.98
0.04
54.37
44166
SeLU
79.95 0.91
100.00
+ 0.00
98.45
1.09
56.03
37.14
CReLU
89.47 +028
100.00
0.00
100.00
+ 0.00
16.75 12.2
Rational
92.35 1.97
100.00
+ 0.00
100.00
0.00
36.27
- 19.21
SwiGLU
31.20 #2110
99.85
0.34
12.75
0.65
12.13 1.19
Deep Fourier
92.61 0.04
100.00
+ 0.00
100.00
+ 0.00
47.45
+ 47.97
Smooth-Leaky
91.69 0.12
99.40
1.34
100.00
0.00
82.75
38.57
Rand. Smooth-Leaky
93.33 +
100.00 0.00
99.93 +0.55
33.52 2.83
Table E5: Effect of activation function with a series of Continual Learning algorithms on Random Label MNIST.
33

---

Published as conference paper ICLR 2026
Activation
Vanilla
+L2-Init
+SNR
+EWC
ReLU
25.79 1.18
93.68 + 2.81
18.23
8.29
12.77 + 1,47
Leaky-ReLU
98.34 0.01
100.00
0.00
100.00 + 0.00
100.00 1.00
Sigmoid
52.24 52.24
46.83 48.50
31.53 #499
12.77
Tanh
58.56 1.05
100.00
= 0.00
87.55
12.24
11.88 1.72
RReLU
98.02 +.003
100.00 + 0.00
100.00
0.00
87.25
+20.40
PReLU
96.86 0.32
100.00 0.00
100.00
0.00
64.73
+48.29
Swish
87.40 2.44
100.00
0.00
100.00
0.00
70.25
400.7
GeLU
42.85 #2112
11.45 1.45
29.10 19.10=11.11
11.14
13.83 #3.82
CeLU
29.64 10.44
99.57 0.57
17.72
8.77
11.20 #0.25
eLU
57.45 20.16
100.00
0.00
44.92
46..84
29.50
#39.43
SeLU
84.61 #2.07
100.00
0.00
99.97
#0007
43.95 #46.0
CReLU
92.90 +0.13
100.00
0.00
100.00
+ 0.00
83.53 +36.82
Rational
94.82 +.775
96.9 1.24
95.56
0.56
30.92 38.66
SwiGLU
83.06 #3.51
84.57
34.5
98.63
= 1.87
19.88 = 12.59
Deep Fourier
96.24 0.51
100.00
0.00
100.00
0.00
100.00 + 0.00
Smooth-Leaky
98.36 0.00
100.00
+ 0.00
100.00
0.00
100.00
= 0.00
Rand. Smooth-Leaky
98.42 0.01
100.00
0.00
100.00
0.00
100.00 - 0.00
Table E6: Effect of activation function with series of Continual Learning algorithms Random Label CIFAR.
Activation
Vanilla
+L,2Init
+SNR
+EWC
ReLU
4.76 1.01
47.52 + 11.87
31.76 #7.13
21.20 + 14.37
Leaky-ReLU
48.86 0.70
33.60
12.91
23.44 1.61
Sigmoid
1.79 #.19
20.00
1.000
2..00
18.40
Tanh
28.59 22.34
45.36
42.56
RReLU
53.60 1.06
34.00
29.60 10.48
PReLU
43.30 #0.61
56.94-7.99
57.64
56.92
Swish
35.31 1.87
19.44+8..
41.68 +
29.36
GeLU
17.60 1.71
54.48
55.56
49.36 4.996
53.28
16.69
CeLU
54.23 1.44
27.04
5.92
31.44
11.17
35.84-8.55
eLU
47.64 1.44
36.16.7.66
34.08 #2.69
18.15
SeLU
49.07 1125
32.56 44881
29.60 17.14
CReLU
20.56 2222
20.00 +0000
4.00
16.72 + 10.79
Rational
40.41 #421
0..00
20.00
25.20 16.01
SwiGLU
9.57 + 1.81
38.56
4.48
35.28 +8337
37.60 11.30
Deep Fourier
72.29 2.11
20.40
1.52
20.08 +0188
27.12 +510
Smooth-Leaky
49.87 + 1.67
35.20
19.55
32.96 10.46
31.12 + 11.40
Rand. Smooth-
57.01
1.59
34.56 1601
29.44 + 7.38
34.72 + 10.91
Table E7: Effect of activation function with series of Continual Learning algorithms CIFAR 5+1.
34

---

Published as conference paper ICLR 2026
Activation
Vanilla
+L2-Init
+SNR
+EWC
ReLU
73.71 0.43
85.60 + 8.29
81.20 + 10.55
64.80
+ 11,45
Leaky-ReLU
85.28 0.20
80.00
8.25
80.00 1.66
Sigmoid
63.89 #7.38
71.29 0.56
69.60
14.10
72.00
13.78
Tanh
70.97 0044
71.11 1.45
19.53
66.00
10.86
RReLU
84.97 +0117
84.40
0.65
85.20
8.566
80.40
+ 10.14
PReLU
82.37 0.11
74.80
+ 14.39
80.80
11.28
71.60
15.58
Swish
82.64 0.99
82.01
0.56
83.60
11.32
78.40
20.85
GeLU
1..990..11
80.40 5.90
+.805
72.40 10.53
CeLU
81.15 #0.68
84.00 #3.12
80.40 #4.56
eLU
80.10 #0334
86.80 2
#4.34
53.20 = 10.83
SeLU
80.98 #0.49
84.36 1.23
75.60 #
CReLU
84.85 0.25
88.40
+ 4.56
86.40
10.24
79.60 +7.40
Rational
80.65 0.38
86.16 1.13
81.60 +
52.40 +8.99
SwiGLU
63.57 2.04
72.00 20.59
79.60
10.71
82.80 = 10.26
Deep Fourier
76.03 0.75
78.40 8.17
51.60
74.80 = 12.38
Smooth-Leaky
85.38 +
91.20 + 2.28
88.80 =5.40
90.40 #
Rand. Smooth-Leaky
86.23 0.13
95.20 = 1.20
86.40 + 10.81
81.60 = 12.44
Table E8: Effect of activation function with series of Continual Learning algorithms Continual ImageNet.
Permuted MNIST
Rand-label MNIST
Rund-label CIFAR
5+1 CIEAR
Cont. ImageNet
Activation
LR
LR
LR
LR
A
LR
le:3
le44
1e44
Leaky--Ret
le:3
le:3
le:3
le.4
le.3
le.4
le.3
3e.4
le.3
le-4
3igmood
3e-4
1c-3
Tanh
10-4
le-3
lc-4
1c-3
lc-4
lc-4
1c-4
1c:3
1c-4
16-3
PReLU
le:3
le-4
le:3
3e4
le-4
le-3
1e-4
3e-4
le-4
3e-4
1o-3
30-4
lc-3
lo-3
1o-3
3c-4
Swish
le-3
le-3
le-3
3e-
le:3
1e-3
le-3
3e-4
1e-4
le-3
le:3
eT.Il
le.3
le-3
le.4
3-4
le-4
16.4
le-3
le-4
1e.3
le-3
GeIU
163
304
163
3-4
1c44
1e4
16-4
SeLU
1e-3
le-3
le4
1e4
le-4
3e-4
le-3
1c-4
1e4
le-3
SwiGLu
le-4
le44
le3
le3
le4
le4
le4
le44
le.3
le4
Rational
lo-3
3-4
le4
1e-3
CReLJ
le-3
le:3
le3
le4
3e4
le3
le:3
le3
le.3
le4
neep Fourier
le.3
Smooth-Teaky
le-3
le-3
le3
le4
le:3
1633
le-3
lc4
16-3
364
Rand. Smooth Jeaky
163
163
163
164
16-3
163
163
163
Table E9: Best IniInit hyperparameters per activation and dataset. We report the learning rate and the L2- L2-Init
coefficient A. We sweep over [le - 4, 3e 4, le 3] values.
35

---

Published as conference paper ICLR 2026
Permuted MNIST
Rand-label MNIST
Rand-label CLEAR
5+1 CIEAR
Cont. ImageNet
LR
7
LR
LR
"
BeLU
lo-3
30-3
lo-4
30-3
lc-4
1e-2
1c4
3e-3
1c-4
le-3
Leaky--RLL
3e-3
le:3
S1 gmadid
le-3
3e-3
le-3
3e.3
le.3
le:2
le-1
3e.3
le.3
3e-3
zann
le-3
le4
PReL3
le.3
3e.3
le.3
le.3
le.4
3e.3
le.4
3e.3
le.4
le.3
BRoTO
le-3
le-3
le-3
3e-3
le-3
le-2
le-3
le-2
le-3
3e-3
lo-2
lo-3
lo-4
36-3
1o-3
1c-2
lo-3
lo-2
ELU
le-3
3e-3
le-4
le-3
1e-4
36-3
le-3
3e-3
1e-3
36-3
CELU
le4
SFMJ
le-3
le-3
le-4
3e.3
le-4
le.3
le-1
3e-3
le.3
le-3
Swiah
1o-3
lo-3
l63
16-2
lo-4
le.3
le.3
CReLD
le-3
3e-3
le-3
le-2
Ie.3
le-2
le-3
3e:3
1e-3
36-3
Ratiooaal
lo-3
le-2
lo4
le-3
le-
36-3
l6-3
3e-3
le-3
le-3
SwiGLU
1e4
3c3
keep Fourier
le:3
3e-3
le3
le-2
le-4
3e-3
le3
3e3
le.3
le-2
Smooth-Leaky
1e.
Rand. Smooth-Seaky
le-3
le-2
le33
16-2
le.3
16.2
16-
16-
1643
le-3
Table E10: Best SNR hyperparameters per activation and dataset. We report the learning rate and the SNR 7) controlling how rare an inactivity event must be (under the estimated firing-rate model) before resetting neuron (tail probability P(A >= <= 11). For all runs used an activation magnitude threshold to count as "fired" of с = 1x10 and a max effective window size for mean estimate of 1000. We sweep over
r) E [le3, 3e 3, le 2] values.
Permuted MNIST
Rand-label MNIST
Rand-label CIFAR
S+I CIFAR
Cont. ImageNet
Activation
Ret.
le-3
lel
le-4
Sel
le44
lel
le4
Sel
lc-4
lel
Leaky
1o-3
1c3
1o-3
16-3
2d2
1c-3
si gmaid
le-3
lel
le-3
lel
1e:3
lel
le-3
lel
1e-3
lel
le.4
PReTO
le-3
lel
le-3
lel
le-4
Sel
le-4
lel
le-4
Tel
RReLU
1o-3
lel
16-3
1o-3
1o-3
le-3
lel
le-4
1e.3
1e2
le-3
ELU
lo-3
lel
1o-4
lel
le-4
Sel
1o-3
262
1e-3
2c2
1e4
SELU
le:3
lel
le44
lel
le-4
2+2
le:3
262
le:3
lel
Swiish
le.3
CReLJ
le-3
lel
le-3
lel
le-3
Sel
le-3
lel
16-3
lel
Rational
1o-3
lel
164
1e:3
163
lel
164
SwIGLU
lc-4
lel
le-3
1e4
lel
le4
Deep Fourler
lo-3
lel
1o3
2c2
lc4
2c2
163
362
16-3
lel
Smooth-Lesky
1e:
Band. Smooth-Leaky
le:3
lel
le.3
lel
1e.3
Sel
1e.3
262
1e.
lel
Table E11: Best EWC hyperparameters per activation and dataset. We report the learning rate and the EWC strength of quadratic penalty A. For all runs we used a decay for older tasks of 7 We sweep over A [10, 50, 200] values.
F EXPANDING ON CONTINUAL REINFORCEMENT LEARNING EXPERIMENTS
Robust Metrics for Non-Stationary RL. Previously, we defined generalization gap as GAP Datrin Ress For each activation and environment, we summarize the change across cycles as (GGAP. = GAP GAP value of A < 0 implies the gap shrinks (improved transfer), while A > 0 implies it widens (worse transfer).
To provide rigorous summary across environments with vastly different reward seales (c.g., Humanoid Hopper), depart from simple means medians, which be sensitive outliers mask the magnitude of failures. Instead,
36

---

Published as a conference paper ICLR 2026
IQM + 95% CI
Activation
Plasticity Score
LR
Optimal HP
ReLU
0.1593 0.043
0.0001
Leaky-ReLU
0.1580 +01115
le-05 0.8
Sigooid
0.3301 0.055
0.0001
Tanh
0,2121 #00028
lc-05
RReLU
0.2255 #0.059
0.0001
Bounds: [0.125, 0.333]
PReLU
0.2695 0.0388
0.0001
Layer. a=0.65
Swish (SiLU)
0.3130 0.077
0.0001
0.01
GeLU
0.1658 0.035
0.0001 1.0
eLU
0,1476 001107
le-05 1.0
CeLU
0.1431 #0.070
le-05
1.0
SeLU
0.2182 #0.032
lc-05 1.673
CReLU
0.1200 #0.010
lc-05
Rational
0.2217 +00018
le-05
C, Leaky-ReI.U
SWIGLU
0,0781 +00008
0.0001
Deep Fourier
0.1766 0.008
le-05
Smooth-Leaks
0.3283 0.037
0.0001
C:0.5, P:2.0, a=0.1
Rand. Smooth-Leaky
0.3853 + 0.038
0.0001
C:0.1, P:1.0. Bounds: [0.01, 0.02]
Table F1: IQM plasticity score across seeds with 95% bootstrap confidence intervals (higher is better). Best IQM per column is bolded. We also report optimal learning rate (LR) and optimal hyperparameters (HP) per activation fuunction. In the case of bounded activations we provide the lower and upper bounds. PReLU provides granularity level the number of parameters per activation layer. A dash (-), in the Optimal HP column, indicates that such activation function uses the unique or baseline parameter (e.g., ReLU only has slope a = 0).
we adopt Interquartile Mean (IQM) (Agarwal al. 2021 for all coosseenviromment aggregations. Crucially, we address the issue of physics simulation instabilities (where plasticity loss leads to rewards < -00).. We apply a stability filter: any run resulting in physics explosion treated as a functional failure. For the Generalization Gap, this means assigning gap of 0.0 (as no meaningful performance difference exists hetween two failures), preventing numerical Table
artifacts from skewing the aggregate A. We report the average reward per environment for all activation functions
Plasticity Score Analysis. Separately, report Plasticity Score that captures late-cycle functional performance on training environments. This distinct from generalization gap; it answers: "can the agent still perform well after repeated shifts on the data it now collects?" To make this metric comparable across tasks, we normalize returns to [0, 1 using robust global bounds derived from the entire experimental suite. We apply stability floor to clipped rewards (treating physics failures 0.0) and report the IQM across seeds and environments.
Under this rigorous metric, the highest Plasticity Scores obtained by Rand. Smooth-Leaak (0.388) and Sigmoid (0.340), followed by Smooth-Leake (0.330) and Swish (0.315) (sec Tab. [T for full details). This ranking highlights a critical trade-off between peak plasticity and safety:
Rand. Smooth-L.ce Leaky achieves the highest aggregate score by dominating in solvable locomotion tasks (Ant. Hal FChestah), demonstrating that smooth, randomized non-linearities facilitate superior gradient flow and rapid adaptation. However, lacks an upper bound, which leads divergence the volatile Humanoid environment.
Sigmoid achieves the second-best score via stability. Its bounded nature (0, 1) prevents physics explosions in Humanoid d, securing baseline of performance where others fail. However, this saturation limits its peak learning capacity simpler environments, resulting in lower total score than the randomized variant.
37

---

Published as conference paper ICLR 2026
Activation
HallCheetab-v5
Humanoid-v5
Ant-v5
Hopper-v5
ReLU
730.7 = 787.7
-1.0 1.0% 1.7 10
1041.7 313.6
-6004.2 10
Losky-ReLü
2.5 10 = 4.3 105
x 10 + 1.3 x 10
x 10* 9.1 10
107.0 62.8
Si.gmoie
2113.2 + 811.5
342.2 46.4
1769.3 + 243.6
336.2 + 97.9
Tanh
1896.2 1: 578.8
1.4.9-1366.1
234.9 126.7
109.7 4
RReIJ
1260.8 = 1259.2
9.3 1.5 10a
1151.3 200.7
159.9 125.1
PReli
2123.3 +
-5.5
10
1525.3
13.8
Swish (SiLU)
2161.9 806.8
-7.6 1..20 10
2135.2 614.1
1
GeLU
850.8 457.7
-1.5 22.00=2265 10"
766.1 = 144.8
31.9 41.3
e
274.9 2997.3
x + 10
4,1 x 10 7.4 10*
116.5 + 102.7
CeLU
521.3 +: 897.6
194.6 1 96.4
42.2 1: 248.5
49.1 1: 75.4
SetU
1437.2 618.1
270.0
3
174.0 38.4
CReLJ
-47.66 681.
1.0 100 = 10*
823.7 165.7
12.1 4.6
Kational
1526.4 4:
6.1 10 3.2 104
1083.3 +: 368.6
174.9 102.3
SwiGLU
4.6 10 6.0 10
2.1 x 10 1 3.3 10
105.8 +: 106.8
118.0 + 27.4
Deep Fourler
592.5 = 125.7
176.3 + 41.5
327.7 15.2
185.9 = 47.0
Smooth-Leaky
2622.4 536.5
2.2 x 10 2.2 x 10
2202. +: 533.6
173.8 +: 39.6
Rand. Smooth-Leaky
3221.5 922.6
6.0 x 2.6 10
2791.2 320.2
187.2 19.4
Table F2: Average reward per environment (Mean Std Dev) over seeds. Higher is better. Best performance per environment is bolded. Large negative values (e.g., for SwiGLU or Humanoid) indicate potential issues.
Regarding transfer, high plasticity often correlates with widening gap. Sigmoid and Swish show positive IQM A (gaps widen). suggesting that their adaptation somewhat specific to the current stationary distribution. Rand. Smooth-Leaky while failing in Humanoid, actually exhibits the lowest IQM A among high- performers in the environments where it remains stable, suggesting its randomized landscape encourages more generalizable solutions. Conversely, traditional activations like ReI.U and Leaky-Rel.U) produce low Plasticity Scores, consistent with their known instability under repeated shifts.
Table FI provides the raw absolute rewards activations. This essential for contextualizing Generalization Gap (Tb 4) a low gap should only be considered a "success" if the corresponding absolute reward in Tab, Fl indicates the agent has actually solved the task.
We therefore present both viewpoints: (i) Plasticity Score (IQM) for functional performance under non-stationarity, and (ii) Generalization Gap (IQM A) for evaluation of the adaptation carried perturbed tests. Full per-activation,
per-environment results are Tab. F3 and cycle-by-cycle evolution is visualized Fig. FI
F.1 INTERPRETING NEGATIVE GENERALIZATION GAPS IN CONTINUAL RL
A strongly negative generalization gap the end of training (GAPP... c.e. is paradoxical under non-stationary streams (e.g.. randomized MuloCo friction Dohare al. 2024)). the training environment keeps shifting. the agent can lose plasticity- it struggles to re-fit the current regime- so R depressed. Yet the policy may retain robust, regime-invariant skills that carry to perturbed test conditions, where evaluation is noise-free and does not suffer on-policy update instability. Consequently R. exceed Ri yielding negative GAPce. Our summary A (GGAP.) becomes highly negative when transferability improves over the training cycle even within-cycle adaptation degrades an acceptable and informative outcome in this setting.
G NOVEL ACTIVATION FUNCTION FORMULATIONS
Our characterization study enables the principled design of many novel activation functions; representative examples include:
38

---

Published as conference paper ICLR 2026
Activation
HalfCheetaab-55
Humanoid-v5
Ant-v5
Hopper-v55
IQM A
Mean A
Std A
ReLU
124.81 511.3
203.06 158.47
183.00 + 473.82
-287.15 + 547.78
153.91
55.93
231.12
Rea
546.91 + 1425.72
0.00 + +0000
166.86
711.59
-1.91 30.97
.0.9
94.04
312.12
Sigmoid
+: 2576.85
18.92 111.38
276.48
1018,48
152.14
129.02
85.53
39.75
Tanh
467.58 + 1506.19
-2.87 52.27
21.61 213.56
-99.31
117.34
-26.09
-124.54
230.58
RReLU
$27.09 +: 1341.00
38.56 + 1398.73
-26.83 + 530.91
21.89 + 69.33
30.22
140.18
259.43
PRcLU
839.60 596.03
-31.228
2211.72
-22.35 74.09
148.78
Swish (S11U)
533.55 = 1314.41
0.00 0.00
780.79
130.11
13.39 =
273.47
331,93
388.92
GeLU
317.71 + 730.89
-9,92 245.15
258.21
+ 395.62
-32.29 40.17
124.15
133.43
180.32
237.41 740.39
617.50
989.90
-81.21 + 162.09
-10.55
113.43
190.79
315.58
CeLU
-341.13 459.03
-99311 74.86
-28.551 + 277.68
3.56 44.991
-165.41
-167.10
169.68
Scld
837.97 +: 1628.38
15.91 80.82
-33988
+ 185.25
$1.90 52.95
33.90
141.47
496.86
CReLU
-238.91 1039,91
640,56
1255.50
329.34 299.75
161 #3.40
165.48
183.15
383.71
Rational
550.82 + 999.991
862.09
+ 2004,43
270.39 526.94
54.72
107.53
410.60
434,51
350.01
Swig'0
0.000 0.00
0.00 + 0.00
336..5 + 186.38
-16.59
44.16
-8.30
-85.80
160.72
Deep
568.95 551.22
0.34
4477.68 208.27
41.53
-238.67
-251.19
316.77
Smcoth-Leaky
847.49 + 1611.57
0,00 0.00
44.09 + 1089.34
27.01
72.58
35.55
229.65
412.30
Rand. Smooth-Leaky
49.38 +: 883.36
0.00 +: 0.00
-336.13 13 + 971.75
-68.68 96.31
-59.03
-113.55
151.18
Table F3: First-to-last cycle A of the Generalization Gap per environment and activation function (rounded to 2 decimals). The + values indicate the 95% confidence interval. The IQM, mean, and std A are calculated across the environments. Lower is better (negative values indicate the generalization gap decreased, improving plasticity). Best (lowest/most negative) per column is bolded.
HallCheetah-v5
Humanold.vS
Hopperu5
Swish-
109
-50
-168
PReUU
316
-98
Sigmoio
Lmakt
Leaky
B02
-149
108
2
Cycle
Cyde
Oyde
Oyde
-1000
-500
500
1000
Generalization Gap (train test)
Figure Fl: The heatmap reports end-of-cycle GAPce c.e per activation (rows) and cycle (columns). Colors centered at (green- negative values, test > train; red = positive values, train >> test). Values are computed on held-out friction variant of the training environment. See Tab. F3 for the across-cycle summary (GAPP.) Together, these views reveal when apparent trainability gains translate (or fail to translate) into generalization.
G.1 BOUNDED PRELU (Bo-PRELU)
The Bounded Parametric Rectified Linear Unit (Bo-PReLU) is designed to combine the adaptability of PReLU with enhanced stability by constraining its learnable negative slope. Our case studies found that extreme slope values can be detrimental to performance. Bo-PReLU addresses this by forcing the slope to remain within a predefined "Goldilocks" range, preventing it from becoming excessively large or small.
The function follows the standard PReLU formulation:
f(x)
(7)
aE
<
The key innovation lies in how a is learned. It is constrained to the range [amin amax]. To ensure this constraint is without interfering with mraseent-based optimization, employ the reparameterization trick. An unconstrained parameter, Okaw learned, and the final slope is derived during the forward pass as:
Amin (Amax amin) a (Orow)
(8)
where the sigmaid function. This makes Bo-PReLU robust and stable learnable rectifier.
39

---

Published as conference paper ICLR 2026
Activation
HDZ
NZG
Sat=: Sat-
C
NonM
SelfN
L L/Retp S"
RSeLU4 Bo-PReLU
Table G1: Binary property grid (V= present, absent). Abbreviations. HDZ: hard dead zone; NZG: non-zero gradient for x < 0; Sat two-sided saturation; Sat- negative-side saturation; C²; first derivative continuous; NonM: non-monotonic segment; SelfN: normalonmn output; L/Rap' learnable randomized slope; f non-zero second derivative. Proposed this work.
Permuted MNIST
Random Label MNIST
Random Label CTFAR
CIFAR 5+1
Continual ImageNet
Activation
PReLU
82.62 + 0.05
92.67 0.23
96.86 + 0.32
43.30 + 0.61
82.37 0.11
Bo-PReLU
84.23 + 0.02
91.57 0.11
98.41 + 0.01
48.15 + 1.20
85.72 + 0.11
Rand. Smooth-Leaky
84.26 + 0.02
93.33 0.05
98.42 + 0.01
57.01 + 1.59
86.23 + 0.13
Table G2: Total Average Online Task Accuracy (%) Continual Supervised Benchmarks, averaged over 5 independent runs. Values are reported as mean + standard deviation (SD). Statistical significance was determined using independent two-sample Welch's t-test (p 0.05).
Rand. Smooth and Continual ImageNet making Bo- Ro-PReLU highly competitive and highlighting strengths of creating activation functions following the principles studied on Section and Section 4 Optimal Hyperparameters for Bo-PReLU are indicated in Table G4] while the rest are in Table F2
Smooth-Leaky Leaky is only statistically significant with respect Bo- Bo-PReLU Random Label MNIST, CIFAR 5+1
G.2 RANDOMIZED -LOPE SELU (RSELU)
The Randomized-Sloo Scaled Exponential Linear Unit (RSELU) is a hybrid activation function designed to merge the stochastic regularization benefits of RReLU with the training stability of SELU's self-normalization property.
The function has modes of operation. During training, introduces randomization the negative slope:
Ax Ar(exp(z)-11
if 2
f(x)
where 21l1
if
During inference, the randomization is removed ensure deterministic output, and the random variable is fixed the mean of its distribution, (I + 22/2. 2. crucial feature this design is that the bounds and u are chosen to be symmetric
10.0
Riel
2.5
-2.5
-5.0
-10.0
-7.5
-3.0
-2.
5.0
10.0
Figure G1: Bo-PReLU where ayéén 0.6, Qmaa 0.8 and Qjnit 0.65.
40

---

Published as conference paper ICLR 2026
custom RGeLU Activation
15
10
101
110.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
16.0
Figure G2: R-SeLU with bounds (1,, u) where r 0,.923,,.44233
around the original SELU alpha parameter (Rs 1.6732). This ensures that the self- self-normalizing property of SELU preserved "in expectation" throughout training, providing a stable foundation while the slope randomization encourages robust learning and plasticity.
Permuted MNIST
Random Label MNIST
Random Label CIFAR
CIFAR 5+1
Continual ImageNet
Activation
Seluu
80.43 0.16
79.95 0.91
84.61 + 2.07
49.07 + 1.25
80.98 + 0.49
R-SeLU
81.72 = 0.04
65.60 2.07
47.23 = 0.65
39.90 + 2.91
79.87 = 0.13
Rand. Smooth-Leaky
84.26 = 0.02
93.33 0.05
98.42 + 0.01
57.01 + 1.59
86.23 + 0.13
Table G3: Total Average Online Task Accuracy (%) Continual Supervised Benchmarks, averaged over
5 independent runs. Values are reported as mean + standard deviation (SD). Statistical significance was determined using independent two-sample Welch's 's t-test (p 0.05).
Permoted MNEST
Label
Label
CIFAR S+1
Continall
Activation
MINIST
CIFAR
ImaggeNet
R-9e107
[0.423,
[0,425,
0.001 LO]
[1,0732,
(16752.16752)
0.001 LO]
layes 0.75,
0.65,
(0.5.
Table G4: Optimal Hyperparameters for extra custom activation function in each Continual Supervised Benchmark Problem. Represented activation function shape parameter the left side of the symbol and the learning rate to the right. Bo-PReLU's indicates initial parameter value and [t. u] bounds. R-SeLU only indicates bounds [4, u).
H THE USE OF LARGE LANGUAGE MODELS (LLMs)
We used an LLM as tool for early brainstorming, code debugging, and writing/editing of the earlier drafts of this paper. During ideation, used it to enumerate experiment variants and sanity sanity-check design choices: for code, requested bug-finding hints and refactoring suggestions that we implemented only after manual review and testing; text, we used it to improve clarity, organization, and grammar. The LLM did not generate novel research ideas, experiments, or results on our behalf; all methodological innovations, analyses, and conclusions are our own. We verified any technical claims, equations, and citations suggested during assisted drafting and did not include uncited, model geder-ggeee text verbatim. No proprictary or personally identifiable data was provided the LLM. The authors retain full responsibility for the content of this paper, and we aflirm the novelty and originality the work.
41


Reviews:
[{"review_id": "nfgbIAl7YD", "reviewer": "Reviewer_mY97", "summary": "This paper looks at the effect of various characteristics of activation functions on the plasticity of neural networks on supervised continual learning and RL benchmarks. These characteristics include things like the slope of the activation function for negative inputs, where they identify a “Goldilocks zone,” and dead-band width which is the portion of the activation function domain that has a small gradient. They propose two different activation functions, Smooth-Leaky and Randomized Smooth-Leaky which are modifications of the Leaky-ReLU activation function with a smooth transition region rather than a kinked one.", "questions": "- Could you further clarify the reasoning behind the criteria for dead units defined for non-ReLU activation functions? For ReLU it makes sense because there is a region where the value of the activation function is 0 with slope 0. That doesn’t seem to be the case with most of the other activation functions studied, and it seems a bit odd to talk about dead-unit fractions.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The initial part of the paper with the case studies on the negative slope and dead band width are valuable experiments for the community.\n- The diversity of metrics and analyses presented for the experiments is quite high.", "weaknesses": "- The presentation of the paper can use some work. Most of the results for the experiments in Sections 3 and 4 are not presented in the main paper, and are instead presented in an Appendix. This is fine for one or two auxiliary results, but here, most of the main results for a significant portion of the paper are presented elsewhere. I’d ask that either the authors figure out a way to present a coherent story with results in the main paper, either by moving some parts of the paper to the appendix (not just the graphs), or if that is not possible in the page limit, consider another venue.\n- The paper introduces two activation functions Smooth-Leaky and Randomized Smooth-Leaky. These activation functions introduce 3 and 4 new hyperparameters respectively which are searched for in a grid, with 175 and 450 combinations respectively. Compare this to the baseline of Leaky-ReLU which had 12 configurations evaluated. Furthermore, these hyperparameters were individually tuned on each setting. The amount of extra tuning given to the new methods is an unfair advantage, and could definitely be the cause of any improvement in performance rather than something inherent about the methods themselves.\n- The metrics used to measure plasticity in the RL section seem to be flawed. If we look at just the rewards, Sigmoid seems to significantly outperform the other activation functions on 2/4 environments. Yet the plasticity score seems designed in a way to remove that effect, taking the median across environments to result in a number where the proposed method does better. A better approach could be to do some kind of normalization of the return and presenting the mean. Taking the median across 4 environments specifically excludes the environment where none of the activation functions except for Sigmoid are able to learn.\n- For generalizability, there’s another metric presented which is a non-standard metric. It looks at the gap between train performance and a test environment which is some perturbation of the train environments. The metric reported compares the difference between the gap after cycle 3 and cycle1. First, it’s unclear what the perturbations are. Second, this metric has no notion of what the actual reward obtained in each of the settings was. A small gap with low absolute rewards is less preferable to a large gap with high absolute rewards, but this metric doesn’t capture that. Finally, measuring generalizability is much easier in the supervised setting, and has been done in several prior works. The results on those settings should be reported before creating such a metric.\n- Tables 1, 2, and 4 are missing standard deviations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hTKpqaB0xw", "reviewer": "Reviewer_kvHD", "summary": "The authors explore the role of activation function design on plasticity in continual learning. The authors introduce the Smooth-Leaky and the Randomized Smooth-Leaky activation functions.The authors evaluate their proposed activation functions in the supervised class incremental continual learning benchmarks and in RL environments and show that the randomized smooth leaky activation function consistently outperforms existing activation functions.", "questions": "- Given the recent popularity of the SwiGLU, why isn't this activation function evaluated?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The empirical results clearly show that the proposed activation function randomized smooth-leaky consistently attains superior average online task accuracy. \n- The experiments consider a comprehensive set of baseline or existing activation functions.\n- The discussion and experiments regarding the generalization gap, specifically, that plasticity loss can manifest in reduced training loss and test loss in different ways, is a positive touch to the paper which is not usually evaluated in this space. As the authors point out, maximizing plasticity via maximizing training loss may result in over-fitting. While earlier papers have argued that solely analyzing test loss in continual settings can be confounded by issues of regularization and generalization. The focus that the authors put here is a strength of this paper.", "weaknesses": "- There is a paper by Lewandowski et al. \"Plastic Learning with Deep Fourier Features\" which also examines the role of activation function design on plasticity. It would be useful if the authors could comment on this line of work, and ideally, incorporate this paper's contribution as a baseline competitor. \n- It would be worthwhile to apply a few existing SOTA methods for plasticity loss such as reset-based: SNR and regularization-based: L2 Init, and to generate a table with results for a reasonable set of METHOD X + ACTIVATION FUNCTION Y. When given an additional continual learning intervention how important is the choice of activation function?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qC5h9Zh4ii", "reviewer": "Reviewer_hFaa", "summary": "The authors provide a thorough investigation on the role of activation functions in addressing plasticity loss in continual learning. They propose two activation functions, Smooth-Leaky and Randomized Smooth-Leaky, which involve maintaining a non-zero derivative floor, optimizing negative-side responsiveness and ensuring smooth transitions at the origin. They evaluate these activations across supervised continual learning benchmarks and RL environments, showing their effectiveness in maintaining plasticity.", "questions": "Please refer to the comments in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The main strength of the paper lies in its rigorous evaluation of activation functions across continual learning benchmarks (supervised and RL).\n- Proposed activation functions, and the concept of a \"Goldilocks zone\" for negative-side responsiveness, offer a interesting perspective on mitigating plasticity loss in continual learning. ​The findings highlight the role of activation functions in maintaining plasticity.\n- Metrics like saturation fraction and dead-band width provide valuable insights into activation function behavior under non-stationary settings.\n- The paper is well-structured and clearly written, with visualizations, explanations.", "weaknesses": "- While the paper discussed impact unconstrained adaptive slopes (e.g., PReLU), it does not explore alternative methods to guide parameters into the \"Goldilocks zone\" which itself is primarily identified empirically, but lacks theoretical grounding or formal analysis. Therefore, explaining why this range is optimal would help the paper overall.\n- The paper does not explore how activation functions interact with different optimizers or learning rate schedules. For example, adaptive optimizers like Adam or RMSProp may behave differently with the proposed activations. Are the proposed activations universally effective across all continual learning settings? Since the dynamics of plasticity loss may vary significantly between tasks, a more nuanced analysis of task-specific performance could provide deeper insights.\n- The reinforcement learning experiments focus solely on MuJoCo tasks, Have authors tried running experiments on domains like sparse reward or high-dimensional tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VVdoRsHXNk", "reviewer": "Reviewer_5MGv", "summary": "The paper argues that activation choice is a primary lever for sustaining plasticity in continual learning. It presents a property-level analysis (negative-branch responsiveness, saturation, “dead-band” width) of activation functions and proposes two drop-in $C^1$ activations, Smooth-Leaky and Randomized Smooth-Leaky, that maintain a non-zero derivative floor. The paper evaluates the activation functions on standard Continual Learning tasks.", "questions": "What distribution over pre-activations is used for the effective negative slope in Figure C-1(per-layer? per-epoch? across tasks?), and how sensitive are the conclusions to that choice? Please report variability across layers/epochs and any normalization applied before computing.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The paper provides a substantial empirical study on the properties of many activation functions under the lens of continual learning. The case studies are well thought out and well conducted, they isolate negative-branch slope, saturation, and “dead-band width” as key predictors under shift. The study of desaturation dynamics under shocks is an important diagnostic of activation functions and ties well activation shape to recovery behavior after controlled shifts.\nThe suggested drop-in replacements are intuitive and easy to implement.", "weaknesses": "In general, the paper is verbose and somewhat difficult to parse. Table 2 reports statistical significance but does not provide confidence intervals or which statistical tests were used. There also appears to be an imbalance in the hyper-parameter sweeps: the two proposed activation functions are explored across 175 and 450 combinations, respectively, whereas other hyper-parameters are examined over only a handful of settings. This imbalance could bias the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eUc1stMV08", "reviewer": "Reviewer_fK2M", "summary": "This work presents two case studies to investigate the activation functions in the context of continual learning and identifies three principles for designing plasticity-friendly nonlinearities. Building upon these insights, the authors propose two new activation functions: **Smooth-Leaky** and **Randomized Smooth-Leaky**. Empirical studies on continual supervised learning and continual reinforcement learning demonstrate that the proposed activation functions achieve superior performance compared to existing alternatives.", "questions": "- Do the proposed activation functions result in greater wall-clock time compared to existing activation functions?\n- Could the authors provide the results of the **Smooth-Leaky** activation function in the continual reinforcement learning experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The case studies are interesting and thoughtfully executed, providing solid and well-supported conclusions. \n- The large-scale empirical studies on continual supervised learning and continual reinforcement learning clearly demonstrate the superiority of the proposed activation function over prior alternatives.\n- The proposed activation functions show promise for broader application in domains that rely on network plasticity.\n- The authors also provide code to improve reproducibility.", "weaknesses": "- As the experiments already include multiple runs, I recommend that the authors report not only the mean results but also measures of statistical significance—such as standard deviation or 95% confidence intervals—for example, in Table 1 and Table 2.\n- The continual reinforcement learning evaluation appears limited to a single experimental configuration. Expanding this analysis to include additional environments or settings would provide stronger support for the generality of the proposed activation functions.\n- *Case Study 1* and *Case Study 2* are important parts of the paper, but too many related results and figures are placed in the appendix. To enhance readability, the authors might consider reorganizing the content despite the page limit.\n- The term $C^1$ transition first appears in the introduction (line 60), but without explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)