Activation Function Design Sustains Plasticity in Continual Learning

## Abstract

Abstract In independent, identically distributed (i.i.d.) training regimes, activation functions have been benchmarked extensively, and their differences often shrink once model size and optimization are tuned. In continual learning, however, the picture is different: beyond catastrophic forgetting, models can progressively lose the ability to adapt (referred to as loss of plasticity ) and the role of the non-linearity in this failure mode remains underexplored. We show that activation choice is a primary, architecture-agnostic lever for mitigating plasticity loss. Building on a property-level analysis of negative-branch shape and saturation behavior, we introduce two drop-in nonlinearities ( Smooth-Leaky and Randomized Smooth-Leaky ) and evaluate them in two complementary settings: (i) supervised class-incremental benchmarks and (ii) reinforcement learning with non-stationary MuJoCo environments designed to induce controlled distribution and dynamics shifts. We also provide a simple stress protocol and diagnostics that link the shape of the activation to the adaptation under change. The takeaway is straightforward: thoughtful activation design offers a lightweight, domain-general way to sustain plasticity in continual learning without extra capacity or task-specific tuning.

## 1 Introduction

Continual learning requires neural networks to acquire new knowledge over time without erasing previously learned information. This poses a fundamental challenge: maintaining a balance between plasticity—the ability to adapt to new data—and stability—the ability to retain prior knowledge. While *catastrophic forgetting* refers to poor performance on previously learned tasks when they are no longer explicitly trained on, loss of plasticity is a distinct phenomenon: networks might retain past capabilities but become increasingly incapable of learning new ones. Despite growing interest, *loss of plasticity* remains less understood and underexplored, particularly in reinforcement learning (RL) settings where the agent’s evolving policy changes the distribution of data it encounters, making it difficult to disentangle learning ability from environmental exposure.

Recent work documents symptoms associated with plasticity loss in deep RL, including reduced gradient magnitudes Abbas et al. (2023), increasing parameter norms Nikishin et al. (2022), rank-deficient curvature Lyle et al. (2022); Lewandowski et al. (2023), and declining representation diversity Kumar et al. (2020; 2023); Dohare et al. (2024). Yet no single factor explains its onset across settings. Lyle et al. (2024) propose a “Swiss cheese” view: multiple, partly independent mechanisms—e.g., pre-activation distribution shift, uncontrolled parameter growth, and the scale of bootstrapped value targets in temporal-difference learning—can each contribute. Mitigation strategies range from architectural refresh (Continual Backprop’s generate-and-test replacement of low-utility units Dohare et al. (2021)) to regularization that targets plasticity retention Lyle et al. (2022); Kumar et al. (2023); see Klein et al. (2024) for a survey.

We argue that a more fundamental knob is hiding in plain sight: the *activation function*. Differences among activations often shrink in i.i.d. training once model size and optimization are tuned, but under continual, non-stationary data they can matter substantially. This motivates a property-level study of how activation shape, especially negative-side responsiveness and saturation, affects plasticity(^0^00Code available at: [https://anonymous.4open.science/r/activations_plasticity-E431/](https://anonymous.4open.science/r/activations_plasticity-E431/)). Previous work has shown that alternative activations such as CReLU Shang et al. (2016); Abbas et al. (2023); Lee et al. (2023) or rational functions Molina et al. (2019); Delfosse et al. (2021b; a) can improve performance under non-stationary conditions. Section [2](https://arxiv.org/html/2509.22562v1#S2) closes with a simple IID vs. class-incremental (C-IL) comparison using a shared setup to illustrate this contrast and motivate the case studies that follow.

Our contributions are as follows:

- •
Comparison of activations function performance across supervised continual learning and non-stationary RL benchmarks (Sec. [2](https://arxiv.org/html/2509.22562v1#S2), Tab. [A1](https://arxiv.org/html/2509.22562v1#A1.T1)).
- •
Analysis of activation function properties identifying a moderate, non-zero negative-side responsiveness (‘Goldilocks zone’) and small dead-band width as key predictors of sustained plasticity (Secs. [3](https://arxiv.org/html/2509.22562v1#S3), [4](https://arxiv.org/html/2509.22562v1#S4)).
- •
Two drop-in activation functions—*Smooth-Leaky* and *Randomized Smooth-Leaky*—that preserve a non-zero derivative floor and target the moderate-leak regime with a $C^{1}$ transition, improving continual adaptation without adding parameters (Secs. [6](https://arxiv.org/html/2509.22562v1#S6), [7](https://arxiv.org/html/2509.22562v1#S7)).

## 2 Activation Functions and Plasticity in Continual Learning

Activation functions are the first gatekeepers of gradient information. Their slope near zero, negative input behavior, and degree of saturation jointly determine how much learning signal survives backpropagation, a critical factor once data distributions change. We provide a comparative overview of commonly used activation functions, highlighting their potential to either exacerbate or mitigate plasticity loss.

Rectifiers. Classic ReLU is efficient but prone to the *dead‑unit* problem (Maas et al., 2013): neurons that output 0 receive no gradient and thus become inactive permanently. Continual‑learning studies confirm a rising fraction of dormant units over time (dormant neurons phenomenon Sokar et al. (2023)), shrinking gradient norms and eventually freezing learning (Abbas et al., 2023; Dohare et al., 2024). ReLU networks often show an increase in parameter norms during training (as they drive outputs with ever-larger weights) and a drop in the number of effective directions in weight space that can reduce error Lewandowski et al. (2023). Leaky‑ReLU alleviates this with a constant negative slope, while PReLU (He et al., 2015) and RReLU (Xu et al., 2015) postpone dormancy by making the negative slope learnable or by randomly sampling it during training.

Saturating sigmoids. Sigmoid and Tanh map inputs to bounded ranges; when units saturate, derivatives shrink toward zero and gradients can vanish, slowing learning Glorot & Bengio (2010). This is relevant for continual learning, which requires sustained adaptation under shift and where loss of plasticity has been repeatedly observed Dohare et al. (2024); Abbas et al. (2023); Juliani & Ash (2024). (See Sec. [4](https://arxiv.org/html/2509.22562v1#S4) for empirical evidence.)

Smooth non-monotonic. Swish (Ramachandran et al., 2017) and GeLU (Hendrycks & Gimpel, 2016) are smooth, weakly non-monotonic rectifiers that preserve small—but non-zero—gradients for inputs near and below zero. This mitigates “dying ReLU” behavior (Maas et al., 2013), so units remain trainable when pre-activation distributions drift under shift. In continual settings, this responsiveness supports ongoing adaptation; empirically, non-monotonic/smooth rectifiers have shown advantages in both supervised and RL domains (Ramachandran et al., 2017; Hendrycks & Gimpel, 2016; Elfwing et al., 2018), and in our experiments (Secs. [3](https://arxiv.org/html/2509.22562v1#S3)–[4](https://arxiv.org/html/2509.22562v1#S4)) they exhibit lower dead-unit fractions and stronger late-cycle adaptation than zero-floor rectifiers.

Exponential variants. ELU/CELU (Clevert et al., 2015; Barron, 2017) reduce bias shift via a negative branch (CELU is $C^{1}$), while SELU (Klambauer et al., 2017) self-normalizes activations toward zero mean/unit variance. Under continual, non-i.i.d. data, where batch-statistics can drift and hurt retention, such built-in stabilization helps maintain trainable scales across tasks (Ioffe, 2017; Pham et al., 2022).

##### Why focus on continual learning?

Under identical models and training budgets on Split-CIFAR-100, activation function rankings compress in i.i.d. joint training but separate sharply in class-incremental (C-IL) settings (Van de Ven et al., 2022) (see Table [1](https://arxiv.org/html/2509.22562v1#S2.T1)). This motivates probing how negative-branch behavior affects plasticity under shift. Unless noted, all case studies use the same 4-layer CNN backbone, optimizer, and training budget (full details in App. [B](https://arxiv.org/html/2509.22562v1#A2)), isolating activation effects from architectural or optimization confusion.

**Table 1: Split-CIFAR-100 average accuracy (5 runs) with identical architecture, optimizer, and budget. Performance differences across activation functions are modest under i.i.d. joint training but widen under class-incremental learning (C-IL). RReLU attains the top mean in both settings; the C-IL improvement is statistically significant (all $p<0.05$), whereas i.i.d. differences are not.**
|  | ReLU | LReLU | RReLU | PReLU | Swish | GeLU | CeLU | eLU | SeLU | Tanh | Sigmoid |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| i.i.d. | 72.11 | 72.00 | 73.71 | 71.43 | 73.16 | 72.51 | 72.66 | 72.64 | 72.12 | 66.49 | 58.78 |
| C-IL | 24.41 | 28.57 | 32.95 | 22.71 | 24.43 | 20.91 | 22.79 | 27.59 | 27.49 | 26.44 | 25.47 |

## 3 Case Study 1: Negative-Slope ‘Goldilocks Zone’

We now test whether *negative-side responsiveness* drives plasticity under shift. Using the shared setup from Sec. 2 (Table [1](https://arxiv.org/html/2509.22562v1#S2.T1)), we sweep the negative branch for three families: piece-wise linear (Leaky-ReLU, RReLU), smooth-tailed activations (Swish, GeLU, ELU/CELU/SELU), and adaptive (PReLU at global/layer/neuron scopes). Our goals are: (i) test if there exists a consistently good value of the negative-side slope across activation functions; (ii) test whether smooth tails approximate the effective slope ($\bar{s}$) of optimal linear regimes; and (iii) assess whether adaptively learned slopes discover optimal values without extra guidance.

‘Goldilocks zone’ for negative slopes (with shape-matched comparison). For activations with a *constant* negative-branch slope (Leaky-ReLU, RReLU), performance reliably peaks for a moderate leak $0.6\!\lesssim\!\bar{s}\!\lesssim\!0.9$ and degrades once $\bar{s}\!\gtrsim\!0.9$ (Fig. [C1](https://arxiv.org/html/2509.22562v1#A3.F1), Tab. [1](https://arxiv.org/html/2509.22562v1#S2.T1)), suggesting the existence of a ‘Goldilocks zone’ for negative slopes. To compare smooth-tailed functions (ELU/CELU/SELU, Swish, GELU) on a common scale, we project their negative-branch behavior onto an effective slope axis, $\bar{s}=\mathbb{E}_{x<0}[\varphi^{\prime}(x)]$, representing the average derivative for negative inputs. When matched by $\bar{s}$, these smooth tails still underperform linear leaks within the ‘Goldilocks zone’ and exhibit higher dead-unit fractions, approaching (but not surpassing) the linear-leak peak only for $\bar{s}>1$ (App. [C.1](https://arxiv.org/html/2509.22562v1#A3.SS1)).

Failure Modes for Slope Magnitude. Building on the ‘Goldilocks zone’ phenomenon, we examined why performance degrades outside it. Across activations we consistently observe a *dead-unit regime* as $\bar{s}\!\to\!0$ ($\approx\!45\%$), and accuracy tracks this effect closely: Final Average Accuracy (ACC) (Sec. [B.5.1](https://arxiv.org/html/2509.22562v1#A2.SS5.SSS1)) is moderately and significantly correlated with dead units (Pearson $(r){=}\,-0.59$, $p{=}5.7{\times}10^{-8}$), whereas its correlation with final scaled gradient norms is weaker but also significant (Pearson $(r){=}\,-0.28$, $p{=}0.019$) (App. [C.2](https://arxiv.org/html/2509.22562v1#A3.SS2)). At very large $\bar{s}$ (near/above $1$), we sometimes see *conditioning spikes*—higher dominant Hessian eigenvalue and lower gradient-Gram effective rank ($\lambda_{\max}\!\uparrow$, Effective rank $\!\downarrow$)— but these appear mainly for smooth-tailed shapes and are not universal for constant-slope leaks (Fig. [C2](https://arxiv.org/html/2509.22562v1#A3.F2); App. [B.6](https://arxiv.org/html/2509.22562v1#A2.SS6), [C.2](https://arxiv.org/html/2509.22562v1#A3.SS2)). Therefore, dead units are the primary driver of failure, and we treat curvature as a useful *diagnostic* rather than a general mechanism, and focus our designs on maintaining negative-side responsiveness.

Current adaptive, learnable slopes fail and need constraints to stay in-band. Under non-stationarity, the “right” negative-side responsiveness is not uniform across units or tasks. We therefore asked whether learnable or randomized slopes can find and *maintain* their specific ‘Goldilocks zone’. Per-neuron PReLU (PReLU-N) drifts below the band ($\approx 0.3-0.6$) over training and attains 30.1% ACC (see Figs.. [C.3](https://arxiv.org/html/2509.22562v1#A3.SS3)); layer/global scopes drift even further (Figs. [C5](https://arxiv.org/html/2509.22562v1#A3.F5)). Thus, adaptivity is *relevant*—it offers robustness when a single fixed leak cannot serve all units—but unconstrained adaptation does not reliably remain in the pre-defined ‘Goldilocks zone’, which explains why learned slopes may sit outside it despite good (but suboptimal) ACC.

## 4 Case Study 2: Desaturation Dynamics Under Shocks

Case Study [3](https://arxiv.org/html/2509.22562v1#S3) showed that negative leak (a non-zero derivative floor(^1^11We call an activation *zero-floor* if $\inf_{x}|\varphi^{\prime}(x)|=0$ (e.g., ReLU, and Sigmoid/Tanh whose derivatives approach 0 in the tails). We call it *non-zero-floor* if there exists $\alpha>0$ such that $\varphi^{\prime}(x)\geq\alpha$ on the negative branch (e.g., Leaky/PReLU/RReLU). We call it *effective non-zero floor* if the derivative is non-zero on finite negative inputs near the decision boundary even though it decays toward 0 as $x\to-\infty$ (e.g., Swish/GeLU).)) is necessary but not sufficient: after a distributional shift, many pre-activations can be pushed deep into an activation’s tail, where gradients are effectively zero. We hypothesize that the time it takes a network to *desaturate* (how quickly gradients reopen after a shock) is a key determinant of adaptation delay in lifelong settings. Therefore, we subject the network to a protocol which isolates how quickly different activation families reopen gradients and regain performance after controlled shocks, complementing the steady-state results from Case Study [3](https://arxiv.org/html/2509.22562v1#S3).

Every $C_{\ell}{=}10$ epochs we apply a one-epoch *scaling shock* by multiplying all pre-activations $z$ by $\gamma\in\{1.5,\,0.5,\,0.25,\,2.0\}$, then revert to $\gamma{=}1$ (App. [D.1](https://arxiv.org/html/2509.22562v1#A4.SS1)). Large $\gamma$ (e.g., $2.0$) pushes $z$ into negative tails or positive plateaus (saturation); small $\gamma$ (e.g., $0.5$) produces the mirrored event. Each activation uses its best negative-slope setting from Case Study [3](https://arxiv.org/html/2509.22562v1#S3) (Table [B2](https://arxiv.org/html/2509.22562v1#A2.T2)). A unit is *saturated* at a step if $|\varphi^{\prime}(x)|<10^{-3}$. We report:
(i) Peak SF: the maximum saturated fraction immediately after each shock;
(ii) AUSC: the *area under the saturation curve* over the recovery window (lower is better);
(iii) Recovery time $\bm{\tau_{95}}$: steps needed to regain $95\%$ of pre-shock performance (App. [D.5](https://arxiv.org/html/2509.22562v1#A4.SS5)).

Derivative-floor rule. Activations with a *strict* non-zero derivative floor (Leaky-ReLU, RReLU, PReLU) achieve the lowest AUSC and near-zero non-recovery rates (<5%) even under the strongest shocks (Fig. [1](https://arxiv.org/html/2509.22562v1#S4.F1), middle/right). In contrast, zero-floor types (ReLU, Sigmoid, Tanh) show the largest AUSC and very high non-recovery across all $\gamma$ (Fig. [1](https://arxiv.org/html/2509.22562v1#S4.F1), left).

Figure: Figure 1: Desaturation under scaling shocks $\gamma$. Left: mean AUSC (lower is better). Middle: SF recovery time (epochs to halve the saturated fraction after the shock; successful recoveries only). Right: SF non-recovery rate (%). Groups: Zero-floor = ReLU, Tanh, Sigmoid; Non-zero-floor = Leaky-ReLU, RReLU, PReLU; Effective non-zero-floor = ELU, CELU, SELU, GELU, Swish. See App. [D.2](https://arxiv.org/html/2509.22562v1#A4.SS2) for details.
Refer to caption: h2_1_gamma_vs_AUSC_SF_rec_time.png

Two-sided penalty. Activations that saturate on *both* sides (Sigmoid, Tanh) suffer the worst shocks: they show the largest peak saturated fraction and AUSC (Fig. [2](https://arxiv.org/html/2509.22562v1#S4.F2), left/right), and fail to desaturate in roughly half of runs (49.8%). One-sided (Kink(^2^22By *kink* we mean continuous but not differentiable at $x{=}0$ ($C^{0}$ but not $C^{1}$; e.g., ReLU, Leaky-ReLU). By *smooth* we mean at least once differentiable at $x{=}0$ ($C^{1}{+}$; e.g., ELU, Swish).)) *non-zero-floor* rectifiers (Leaky-ReLU, PReLU, RReLU) recover far more reliably (non-recovery $\approx$ 13.3%). One-sided (Smooth) (ELU, CELU, SELU) sit between these extremes: when they recover, they do so quickly (Fig. [2](https://arxiv.org/html/2509.22562v1#S4.F2), middle), but failures still occur frequently at strong shocks. Overall, a single hard saturation boundary is less harmful than two; maintaining a non-zero derivative floor on the negative side remains the most protective.

Figure: Figure 2: Sidedness effects under shocks. Left: Peak saturated fraction during the shock (higher = more units saturated). Middle: Saturation Fraction (SF) time-to-half-recover (epochs; successful recoveries only; lower is better). Right: AUSC (lower is better). Groups: One-sided (kink) = Leaky-ReLU, PReLU, RReLU; One-sided (smooth) = ELU, CELU, SELU; Two-sided (saturating) = Sigmoid, Tanh. See App. [D.3](https://arxiv.org/html/2509.22562v1#A4.SS3) for details.
Refer to caption: h2_2_two_sided_penalty.png

The width of the dead band predicts shock sensitivity. Beyond the findings of derivative floor and sidedness, we ask how *much* of the input range of an activation produces nearly zero gradients. We define a *Dead-Band Width (DBW)* as the fraction of a typical pre-activation range (e.g., [-100, 100]) where the magnitude of the activation’s first derivative, $\lvert\varphi^{\prime}(x)\rvert$ falls below certain threshold, $\epsilon<10^{-3}$. Analytically computed DBW, $|\varphi^{\prime}(x)|<10^{-3}$, strongly tracks desaturation outcomes across activations: it correlates with AUSC ($r=0.81,\;p=0.0016$) and with non-recovery ($r=0.84,\;p=0.0013$). Intuitively, wider dead bands expose more inputs to vanishing gradients during shocks, prolonging saturation and increasing failure rates (see App. [D.4](https://arxiv.org/html/2509.22562v1#A4.SS4)).

## 5 Implications for Activation-Function Design

Sections [3](https://arxiv.org/html/2509.22562v1#S3)–[4](https://arxiv.org/html/2509.22562v1#S4) suggest three rules for plasticity-friendly nonlinearities: (i) maintain a *non-zero derivative floor*, (ii) keep negative-side responsiveness in a moderate, activation-specific ‘Goldilocks zone’, and (iii) prefer a *$C^{1}$ (smooth) transition at the origin when (i)–(ii) are held fixed*. Conversely, avoid *two-sided saturation* and wide analytic *dead bands* ($|\varphi^{\prime}|<10^{-3}$), which track larger AUSC and non-recovery.

Our measurements reveal two competing objectives:
*(A) recovery success* (low non-recovery rate after shocks) and
*(B) recovery speed/extent* (low AUSC, short time-to-recover) *conditional on recovery*.
Kinked $C^{0}$ rectifiers with a strict floor minimize non-recovery across $\gamma$ (Fig. [1](https://arxiv.org/html/2509.22562v1#S4.F1), middle/right), whereas one-sided $C^{1}$ shapes often recover faster when they do recover (Fig. [2](https://arxiv.org/html/2509.22562v1#S4.F2), middle), yet fail more frequently at the largest shocks (Fig. [1](https://arxiv.org/html/2509.22562v1#S4.F1), right).
Thus “smooth beats kink” is not universal: we prioritize (A) recovery success—irreversible non-recovery dominates downstream performance—and use (B) as a tie-breaker among activations that recover. Therefore, we keep the strict floor and negative linear leak of the Leaky-ReLU family aiming for an empirical ‘Goldilocks zone’, and introduce smoothness only insofar as it *preserves* those two properties.

### 5.1 Smooth-Leaky and Randomized Smooth-Leaky

Figure: Figure 3: Smooth-Leaky with $\alpha{=}0.1$, $p{=}3.0$, $c{=}5.0$. Randomized Smooth-Leaky draws $\alpha$ from bounds; visually it matches Smooth-Leaky for the sampled $\alpha$.
Refer to caption: smooth_leaky_sample.png

Guided by Sec. [5](https://arxiv.org/html/2509.22562v1#S5)—(i) strict non-zero floor, (ii) moderate leak, (iii) prefer $C^{1}$ over $C^{0}$ when (i)–(ii) are held fixed—we introduce two drop-in rectifiers that keep capacity unchanged.

The Smooth-Leaky activation function (Fig. [3](https://arxiv.org/html/2509.22562v1#S5.F3)) is designed as a direct, $C^{1}$, drop-in substitute for Leaky ReLU that preserves the negative-side floor and the positive-side identity while removing the kink with a smooth, curved transition region. It is asymptotically linear ($f(x)\!\approx\!\alpha x$ for $x\!\ll\!0$, $f(x)\!\approx\!x$ for $x\!\gg\!0$) and controlled by a leak $\alpha$ plus a smooth transition set by $(p,c)$:

$$ $f(x)=\alpha x+(1-\alpha)\,x\,\cdot\sigma\!\left(\tfrac{cx}{p}\right)$ (1) $$

where $\sigma$ is the sigmoid. Here, $\alpha$ fixes the negative-side floor, and $(p,c)$ set the width/steepness of the transition.

To add lightweight exploration around a moderate leak we introduce Randomized Smooth-Leaky by replacing the fixed $\alpha$ with a random slope $r$ drawn uniformly from $[l,u]$ on each forward pass; at inference we fix $r$ to its mean $(l{+}u)/2$:

$$ $f(x)=r\,x+(1-r)\,x\,\sigma\!\left(\tfrac{cx}{p}\right),\qquad r\sim\mathcal{U}(l,u),\quad r_{\text{test}}=\tfrac{l+u}{2}.$ (2) $$

This randomized variant preserves the strict floor and $C^{1}$ transition while encouraging robustness to small variations in negative-side responsiveness. Limitations of multi-parameter design are explained in App. [B.1](https://arxiv.org/html/2509.22562v1#A2.SS1).

## 6 Continual Supervised Learning

Following Kumar et al. (2023), we evaluate five supervised continual image-classification benchmarks spanning two shift types: input distribution shift (Permuted MNIST, 5+1 CIFAR, Continual ImageNet) and concept shift (Random Label MNIST, Random Label CIFAR). Training proceeds as a sequence of tasks without task-identity signals where the agent receives mini-batches for a fixed duration per task. Permuted MNIST Goodfellow et al. (2013) applies a fixed random pixel permutation to a shared subset for each task. Random Label MNIST Lyle et al. (2023) and Random Label CIFAR assign random labels to a fixed subset to encourage memorization. CIFAR 5+1 draws and alternates hard (5 classes) and easy (single class) tasks from CIFAR-100. Evaluation focuses on hard tasks to stress plasticity loss mitigation. Finally, Continual ImageNet Dohare et al. (2024); Russakovsky et al. (2015) performs a task-binary classification over two ImageNet classes which do not repeat across tasks, ensuring non-overlapping class exposure and clearer measurement of plasticity over time. An extended explanation of each benchmark problem is found in App. [E](https://arxiv.org/html/2509.22562v1#A5).

**Table 2: Total Average Online Task Accuracy (%) across all Continual Supervised Benchmark Problem by activation function. Averaged over 5 independent runs. Rand. Smooth-Leaky is statistically significant (p $<$ 0.05) with respect to the next best-performing activation, Smooth-Leaky is also significant compared to the next in Rand. Label CIFAR, CIFAR 5+1 and Continual ImageNet.**
| Activation | Permuted<br>MNIST | Random Label<br>MNIST | Random Label<br>CIFAR | CIFAR<br>5+1 | Continual<br>ImageNet |
| --- | --- | --- | --- | --- | --- |
| ReLU | 78.85 | 20.03 | 25.79 | 4.76 | 73.71 |
| Leaky-ReLU | 84.14 | 91.53 | 98.34 | 48.86 | 85.28 |
| Sigmoid | 76.96 | 79.59 | 52.24 | 1.79 | 63.89 |
| Tanh | 70.32 | 63.40 | 58.56 | 28.59 | 70.97 |
| RReLU | 83.95 | 93.10 | 98.02 | 53.60 | 85.07 |
| PReLU | 82.62 | 92.67 | 96.86 | 43.30 | 82.37 |
| Swish (SiLU) | 83.41 | 67.73 | 87.40 | 35.31 | 82.64 |
| GeLU | 78.97 | 38.79 | 42.85 | 17.60 | 75.49 |
| CeLU | 82.93 | 37.16 | 29.64 | 54.23 | 81.14 |
| eLU | 80.50 | 84.23 | 57.45 | 47.64 | 80.10 |
| SeLU | 80.43 | 79.95 | 84.61 | 49.07 | 80.98 |
| CReLU | 82.66 | 89.47 | 92.90 | 20.56 | 84.85 |
| Rational | 80.18 | 91.27 | 94.82 | 40.41 | 80.65 |
| Smooth-Leaky | 84.03 | 91.69 | 98.36 | 49.87 | 85.38 |
| Rand. Smooth-Leaky | 84.26 | 93.33 | 98.42 | 57.01 | 86.23 |

Across the five continual benchmarks, we observe a clear pattern, reported in Tab. [2](https://arxiv.org/html/2509.22562v1#S6.T2). First, *rectifiers with a learnable or randomized negative branch* dominate: Leaky-ReLU, RReLU, PReLU, Smooth-Leaky, and Rand. Smooth-Leaky consistently outperform ReLU—especially on the harder settings—while smooth rectifiers such as Swish/SiLU are competitive but typically trail the best leaky-family members. Second, we again observe the first reported *‘Goldilocks zone’* for the negative branch (cf. Sec. [3](https://arxiv.org/html/2509.22562v1#S3)): the strongest performers cluster around an initial/effective negative slope in the range $[0.6,\,0.9]$ (including the mean of RReLU bounds and neuron-wise $\alpha$ in PReLU). We also evaluated CReLU and Rational activations, recently proposed to mitigate loss of plasticity Abbas et al. (2023); Kumar et al. (2023); Delfosse et al. (2021b). Constrained rationals were excluded, as previous work shows that they improve RL stability but reduce plasticity Surdej et al. (2025), which is our main focus. In our experiments, both CReLU and Rational outperform ReLU, but remain below other standard activations and our proposed variants (see Tab. [E2](https://arxiv.org/html/2509.22562v1#A5.T2) and Fig. [E1](https://arxiv.org/html/2509.22562v1#A5.F1)).

## 7 Continual Reinforcement Learning

Continual learning is particularly critical in reinforcement learning (RL), where non-stationarity arises not only from changes in the environment but also from the agent’s evolving policy, which affects the data distribution even in fixed environments. This tight feedback loop between learning and data collection makes RL especially vulnerable to loss of plasticity, where neural networks become progressively less responsive to new experiences. Recent work has shown that deep RL agents suffer from a gradual decline in representational diversity and gradient signal quality as training progresses, leading to suboptimal adaptation in later stages of learning Dohare et al. (2024); Abbas et al. (2023). Diagnosing and understanding this phenomenon in RL can be more challenging than supervised learning settings due to high variability in algorithmic design (e.g., model-based vs. model-free, use of replay buffers, off-policy dynamics) and the inherent stochasticity of agent-environment interactions. As a result, systematic demonstrations of plasticity loss in RL require carefully controlled protocols and extensive experimentation, a challenge that we approach from the perspective of activation functions and their influence on network adaptability. Activation functions play a pivotal role in deep reinforcement learning. Although ReLU and Tanh remain the most widely used options (e.g., Mnih et al. (2013; 2015); Hessel et al. (2018)), both exhibit limitations that affect learning dynamics Nauman et al. (2024). ReLU, as previously mentioned, is susceptible to issues such as the dormant neuron phenomenon Sokar et al. (2023), loss of plasticity Lyle et al. (2023; 2022), and overestimation when encountering out-of-distribution inputs Ball et al. (2023). Tanh, commonly employed to constrain outputs within a fixed range, suffers from saturation at its extremes, leading to vanishing gradients that hinder efficient learning in deep networks Pascanu et al. (2013).

Therefore, to investigate plasticity loss in a continual RL setting, we train a *single* PPO agent Schulman et al. (2017) on a fixed and repeating sequence of four MuJoCo locomotion tasks using Gymnasium Towers et al. (2024): HalfCheetah-v4 $\rightarrow$ Hopper-v4 $\rightarrow$ Walker2d-v4 $\rightarrow$ Ant-v4 $\rightarrow$ HalfCheetah-v4 $\rightarrow$ Hopper-v4 $\rightarrow$ $\cdots$. The agent cycles through this sequence three times, training for 1M timesteps per environment per cycle (total 12M). Episodes terminate early on invalid configurations following Dohare et al. (2024) (e.g., falls for Hopper/Walker2d, unstable heights for Ant); HalfCheetah runs full-length. The policy and value networks share a two-layer MLP backbone (256 units each) that is updated across all tasks. For each environment we attach a lightweight input adapter (to map its observation space) and a task-specific output head (for its action space); both persist across cycles and continue learning when the environment reappears. Hyperparameters are in App. LABEL:tab:hyper_sweep.

**Table 3: Mean Plasticity Score across 5 seeds (higher is better). We only report the top-performing activations. See Table [F1](https://arxiv.org/html/2509.22562v1#A6.T1) for a full comparison of all activations. ^† Values are reported as mean $\pm$ 95% CI half-width. For bootstrap CIs, which can be asymmetric, we report the larger one-sided margin so the printed $\pm$ interval conservatively covers the equal-tailed 95% CI [L,U].**
| Metric | Swish | PReLU | Sigmoid | Rand. Smooth-Leaky |
| --- | --- | --- | --- | --- |
| Mean $\pm$ 95% CI^† | 1120.28  $\pm$ 229.22 | 684.88  $\pm$ 115.90 | 993.61  $\pm$ 190.38 | 1310.94  $\pm$ 98.37 |

Figure: Figure 4: Plasticity Score across 5 seeds (95% bootstrap CIs) showing a complete sequence of 3 cycles across all 4 environments. The table [3](https://arxiv.org/html/2509.22562v1#S7.T3) reports the mean of this score across seeds, only showing the top-performing activations to avoid clutter.
Refer to caption: rewards_best.png

##### Plasticity Score (definition).

For each run $r$: (i) in the *last cycle* of training, compute the mean episodic return for each environment over the *final* $p{=}15\%$ of that environment’s cycle (steady-state); (ii) take the *median across environments* to avoid any single task dominating the summary. This per-run statistic is the *Plasticity Score* reported in Table [3](https://arxiv.org/html/2509.22562v1#S7.T3). This aggregates performance *across multiple environments* while emphasizing late-cycle adaptability.

Across seeds, Rand. Smooth-Leaky achieves the highest plasticity score, outperforming Swish, Sigmoid, and PReLU. These results highlight that smooth and randomized variants provide more robust adaptation and retention across sequential environments compared to classical activations. Figure [4](https://arxiv.org/html/2509.22562v1#S7.F4) complements this analysis by showing reward trajectories across cycles for the top-performing activations.

### 7.1 Trainability vs. Generalizability

Anchoring *plasticity loss* to a single mechanism risks conflating cause and effect. Prior work (Berariu et al., 2021) distinguishes *trainability*—reduced ability to lower loss on new data (Dohare et al., 2021; Elsayed & Mahmood, 2024; Lyle et al., 2022)—from *generalizability*—reduced performance on unseen data (Ash & Adams, 2020; Zilly et al., 2021). Mitigating plasticity loss should improve both, not merely memorize the latest data Lee et al. (2024); nevertheless, their relationship remains unsettled, and terminology is often mixed Klein et al. (2024). In brief: loss of trainability (no effective parameter updates) can *cause* loss of functional plasticity, but the reverse need not hold—models may still update yet fail to *transfer* gains.

Thus, we can argue that loss of *plasticity loss* has two faces in RL: (i) *train-side adaptation*—can the agent still improve on the data it now collects?—and (ii) *transfer to perturbed test conditions*. We report one metric for each. Plasticity Score (Sec. [7](https://arxiv.org/html/2509.22562v1#S7), Tab. [3](https://arxiv.org/html/2509.22562v1#S7.T3)) summarizes late-cycle train-side performance.
Generalization Gap compares the same policy on matched test variants (perturbed dynamics/seeds; App. [F](https://arxiv.org/html/2509.22562v1#A6)): for each cycle $c$ and environment $e$,
$\mathrm{GAP}_{c,e}=R^{\text{train}}_{c,e}-R^{\text{test}}_{c,e}$,
where $R$ is the expected return (measured at the end of cycle $c$).
Thus $\Delta(\mathrm{GAP}_{e})=\mathrm{GAP}_{3,e}-\mathrm{GAP}_{1,e}$; larger $\Delta$ means the train–test gap *widened* over time (worse transfer). We aggregate $\Delta$ per activation by the median across environments.

Among the high-plasticity activations, Rand. Smooth-Leaky stands out, while not uniformly best per environment: it attains the lowest median $\Delta(\mathrm{GAP}_{\text{e}})$ across MuJoCo (Tab. [4](https://arxiv.org/html/2509.22562v1#S7.T4)), indicating the least widening of the train–test gap on average—and on Humanoid-v5 it even narrows the gap ($\Delta<0$). In contrast, Swish and Sigmoid score highly on late-cycle Plasticity Score (Fig. [4](https://arxiv.org/html/2509.22562v1#S7.F4), Tab. [3](https://arxiv.org/html/2509.22562v1#S7.T3)) yet exhibit positive $\Delta(\mathrm{GAP}_{\text{e}})$, revealing a trainability–transfer tension. PReLU is competitive on some environments (best $\Delta$ on Ant-v5 and Hopper-v5) but has a higher median overall.

These metrics are complementary, not redundant. Plasticity Score asks “did the agent remain adaptable on the data it collected?”, while $\Delta(\mathrm{GAP}_{\text{e}})$ asks “did that adaptation carry to perturbed tests?”. Our activation designs primarily target sustained train-side plasticity (strict floor, moderate leak, $C^{1}$ transition)—a prerequisite for learning under shift. Improving transfer likely requires coupling activations with policy/regularization choices (e.g., band-regularizing the learned leak toward target bounds, entropy/KL tuning, dynamics randomization). We report both metrics to surface, not mask, this open question.

**Table 4: Change in Generalization Gap from cycle 1 to cycle 3, per environment and activation (rounded to 2 decimals). $\Delta(\mathrm{GAP}_{\text{e}})=\mathrm{GAP}_{\text{3,e}}-\mathrm{GAP}_{\text{1,e}}$; $\Delta(\mathrm{GAP}_{\text{e}})>0$ means the train–test gap *widened* (worse transfer). The rightmost column is the *median across environments* to reduce sensitivity to scale differences. Full results in Table [F2](https://arxiv.org/html/2509.22562v1#A6.T2). We interpret large negative generalization values in App.[F.1](https://arxiv.org/html/2509.22562v1#A6.SS1).**
| Activation | HalfCheetah-v5 | Humanoid-v5 | Ant-v5 | Hopper-v5 | median $\Delta$ |
| --- | --- | --- | --- | --- | --- |
| Sigmoid | 1521.45 | 18.92 | 276.48 | 152.14 | 214.31 |
| PReLU | 839.60 | 128132.00 | 94.17 | -22.35 | 466.88 |
| Swish (SiLU) | 627.24 | -127118.70 | 1035.22 | 20.88 | 324.06 |
| Rand. Smooth-Leaky | 103.80 | -1236645.00 | 563.09 | 19.22 | 61.51 |

## 8 Conclusion and Future Work

The choice of the activation function greatly impacts performance and plasticity in continual learning; it must be designed, not assumed. We do not claim universal wins, but our results show that first-principles activations mitigate plasticity loss and improve trainability. Building on these findings, future work will test interactions with other standard continual learning approaches such as experience replay and regularization, narrow the RL generalization gap by tracking train-side plasticity and test-time robustness, which will help clarify when improved trainability helps versus when it simply leads to overfitting on recent data, and move from fixed ‘Goldilocks zone’ slopes to adaptive, per-neuron self-tuning. We plan to derive a stronger theoretical understanding (curvature/desaturation bounds for smooth-leaky families) and scale to larger and more challenging CL domains while studying the interplay of activation design with optimizer and normalization effects, culminating in a principle-guided automated search for new robust activations.

## References

- Abbas et al. (2023)
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado.
Loss of plasticity in continual deep reinforcement learning.
In *Conference on lifelong learning agents*, pp. 620–636. PMLR, 2023.
- Ash & Adams (2020)
Jordan Ash and Ryan P Adams.
On warm-starting neural network training.
*Advances in neural information processing systems*, 33:3884–3894, 2020.
- Ball et al. (2023)
Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine.
Efficient online reinforcement learning with offline data.
In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), *Proceedings of the 40th International Conference on Machine Learning*, volume 202 of *Proceedings of Machine Learning Research*, pp. 1577–1594. PMLR, 23–29 Jul 2023.
URL [https://proceedings.mlr.press/v202/ball23a.html](https://proceedings.mlr.press/v202/ball23a.html).
- Barron (2017)
Jonathan T Barron.
Continuously differentiable exponential linear units.
*arXiv preprint arXiv:1704.07483*, 2017.
- Berariu et al. (2021)
Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath.
A study on the plasticity of neural networks.
*arXiv preprint arXiv:2106.00042*, 2021.
- Bingham & Miikkulainen (2023)
Garrett Bingham and Risto Miikkulainen.
Efficient activation function optimization through surrogate modeling.
*Advances in Neural Information Processing Systems*, 36:6634–6661, 2023.
- Bingham et al. (2020)
Garrett Bingham, William Macke, and Risto Miikkulainen.
Evolutionary optimization of deep learning activation functions.
In *Proceedings of the 2020 Genetic and Evolutionary Computation Conference*, pp. 289–296, 2020.
- Cai et al. (2021)
Zhipeng Cai, Ozan Sener, and Vladlen Koltun.
Online continual learning with natural distribution shifts: An empirical study with visual data.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 8281–8290, 2021.
- Clevert et al. (2015)
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.
Fast and accurate deep network learning by exponential linear units (elus).
*arXiv preprint arXiv:1511.07289*, 4(5):11, 2015.
- Delfosse et al. (2020)
Quentin Delfosse, Patrick Schramowski, Alejandro Molina, Nils Beck, Ting-Yu Hsu, Yasien Kashef, Salva Rüling-Cachay, and Julius Zimmermann.
Rational activation functions.
[https://github.com/ml-research/rational_activations](https://github.com/ml-research/rational_activations), 2020.
- Delfosse et al. (2021a)
Quentin Delfosse, Patrick Schramowski, Alejandro Molina, and Kristian Kersting.
Recurrent rational networks.
*arXiv preprint arXiv:2102.09407*, 2021a.
- Delfosse et al. (2021b)
Quentin Delfosse, Patrick Schramowski, Martin Mundt, Alejandro Molina, and Kristian Kersting.
Adaptive rational activations to boost deep reinforcement learning.
*arXiv preprint arXiv:2102.09407*, 2021b.
- Dohare et al. (2021)
Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood.
Continual backprop: Stochastic gradient descent with persistent randomness.
*arXiv preprint arXiv:2108.06325*, 2021.
- Dohare et al. (2024)
Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Ruapm Mahmood, and Richard S. Sutton.
Loss of plasticity in deep continual learning.
*Nature*, 632:768—774, 2024.
- Elfwing et al. (2018)
Stefan Elfwing, Eiji Uchibe, and Kenji Doya.
Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.
*Neural networks*, 107:3–11, 2018.
- Elsayed & Mahmood (2024)
Mohamed Elsayed and A Rupam Mahmood.
Addressing loss of plasticity and catastrophic forgetting in continual learning.
*arXiv preprint arXiv:2404.00781*, 2024.
- Ghunaim et al. (2023)
Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem.
Real-time evaluation in online continual learning: A new hope.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 11888–11897, 2023.
- Glorot & Bengio (2010)
Xavier Glorot and Yoshua Bengio.
Understanding the difficulty of training deep feedforward neural networks.
In *Proceedings of the thirteenth international conference on artificial intelligence and statistics*, pp. 249–256. JMLR Workshop and Conference Proceedings, 2010.
- Goodfellow et al. (2013)
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio.
An empirical investigation of catastrophic forgetting in gradient-based neural networks.
*arXiv preprint arXiv:1312.6211*, 2013.
- He et al. (2015)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.
In *Proceedings of the IEEE international conference on computer vision*, pp. 1026–1034, 2015.
- Hendrycks & Gimpel (2016)
Dan Hendrycks and Kevin Gimpel.
Gaussian error linear units (gelus).
*arXiv preprint arXiv:1606.08415*, 2016.
- Hessel et al. (2018)
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.
Rainbow: Combining improvements in deep reinforcement learning.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 32, 2018.
- Ioffe (2017)
Sergey Ioffe.
Batch renormalization: Towards reducing minibatch dependence in batch-normalized models.
*Advances in neural information processing systems*, 30, 2017.
- Juliani & Ash (2024)
Arthur Juliani and Jordan Ash.
A study of plasticity loss in on-policy deep reinforcement learning.
*Advances in Neural Information Processing Systems*, 37:113884–113910, 2024.
- Klambauer et al. (2017)
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter.
Self-normalizing neural networks.
*Advances in neural information processing systems*, 30, 2017.
- Klein et al. (2024)
Timo Klein, Lukas Miklautz, Kevin Sidak, Claudia Plant, and Sebastian Tschiatschek.
Plasticity loss in deep reinforcement learning: A survey.
*arXiv preprint arXiv:2411.04832*, 2024.
- Kumar et al. (2020)
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine.
Implicit under-parameterization inhibits data-efficient deep reinforcement learning.
*arXiv preprint arXiv:2010.14498*, 2020.
- Kumar et al. (2023)
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy.
Maintaining plasticity in continual learning via regenerative regularization.
*arXiv preprint arXiv:2308.11958*, 2023.
- Lee et al. (2023)
Hojoon Lee, Hanseul Cho, Hyunseung Kim, Daehoon Gwak, Joonkee Kim, Jaegul Choo, Se-Young Yun, and Chulhee Yun.
Plastic: Improving input and label plasticity for sample efficient reinforcement learning.
*Advances in Neural Information Processing Systems*, 36:62270–62295, 2023.
- Lee et al. (2024)
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle.
Slow and steady wins the race: maintaining plasticity with hare and tortoise networks.
In *Proceedings of the 41st International Conference on Machine Learning*, pp. 26416–26438, 2024.
- Lewandowski et al. (2023)
Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C Machado.
Directions of curvature as an explanation for loss of plasticity.
*arXiv preprint arXiv:2312.00246*, 2023.
- Liu et al. (2025)
Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Ling Pan.
Measure gradients, not activations! enhancing neuronal activity in deep reinforcement learning.
*arXiv preprint arXiv:2505.24061*, 2025.
- Lyle et al. (2022)
Clare Lyle, Mark Rowland, and Will Dabney.
Understanding and preventing capacity loss in reinforcement learning.
*arXiv preprint arXiv:2204.09560*, 2022.
- Lyle et al. (2023)
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney.
Understanding plasticity in neural networks.
In *International Conference on Machine Learning*, pp. 23190–23211. PMLR, 2023.
- Lyle et al. (2024)
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney.
Disentangling the causes of plasticity loss in neural networks.
*arXiv preprint arXiv:2402.18762*, 2024.
- Maas et al. (2013)
Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al.
Rectifier nonlinearities improve neural network acoustic models.
In *Proc. icml*, volume 30, pp.  3. Atlanta, GA, 2013.
- Manessi & Rozza (2018)
Franco Manessi and Alessandro Rozza.
Learning combinations of activation functions.
In *2018 24th international conference on pattern recognition (ICPR)*, pp. 61–66. IEEE, 2018.
- Mnih et al. (2013)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.
Playing atari with deep reinforcement learning.
*arXiv preprint arXiv:1312.5602*, 2013.
- Mnih et al. (2015)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.
Human-level control through deep reinforcement learning.
*nature*, 518(7540):529–533, 2015.
- Molina et al. (2019)
Alejandro Molina, Patrick Schramowski, and Kristian Kersting.
Padé activation units: End-to-end learning of flexible activation functions in deep networks.
In *International Conference on Learning Representations*, 2019.
- Nair & Hinton (2010)
Vinod Nair and Geoffrey E Hinton.
Rectified linear units improve restricted boltzmann machines.
In *Proceedings of the 27th international conference on machine learning (ICML-10)*, pp. 807–814, 2010.
- Nauman et al. (2024)
Michal Nauman, Michał Bortkiewicz, Piotr Miłoś, Tomasz Trzciński, Mateusz Ostaszewski, and Marek Cygan.
Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning.
*arXiv preprint arXiv:2403.00514*, 2024.
- Nikishin et al. (2022)
Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville.
The primacy bias in deep reinforcement learning.
In *International conference on machine learning*, pp. 16828–16847. PMLR, 2022.
- Pascanu et al. (2013)
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.
On the difficulty of training recurrent neural networks.
In Sanjoy Dasgupta and David McAllester (eds.), *Proceedings of the 30th International Conference on Machine Learning*, volume 28 of *Proceedings of Machine Learning Research*, pp. 1310–1318, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.
URL [https://proceedings.mlr.press/v28/pascanu13.html](https://proceedings.mlr.press/v28/pascanu13.html).
- Pham et al. (2022)
Quang Pham, Chenghao Liu, and Steven Hoi.
Continual normalization: Rethinking batch normalization for online continual learning.
*arXiv preprint arXiv:2203.16102*, 2022.
- Prabhu et al. (2023)
Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener.
Online continual learning without the storage constraint.
*arXiv preprint arXiv:2305.09253*, 2023.
- Ramachandran et al. (2017)
Prajit Ramachandran, Barret Zoph, and Quoc V Le.
Searching for activation functions.
*arXiv preprint arXiv:1710.05941*, 2017.
- Russakovsky et al. (2015)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.
ImageNet Large Scale Visual Recognition Challenge.
*International Journal of Computer Vision (IJCV)*, 115(3):211–252, 2015.
doi: 10.1007/s11263-015-0816-y.
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Shang et al. (2016)
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee.
Understanding and improving convolutional neural networks via concatenated rectified linear units.
In *international conference on machine learning*, pp. 2217–2225. PMLR, 2016.
- Sokar et al. (2023)
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci.
The dormant neuron phenomenon in deep reinforcement learning.
In *International Conference on Machine Learning*, pp. 32145–32168. PMLR, 2023.
- Surdej et al. (2025)
Rafał Surdej, Michał Bortkiewicz, Alex Lewandowski, Mateusz Ostaszewski, and Clare Lyle.
Balancing expressivity and robustness: Constrained rational activations for reinforcement learning.
*arXiv preprint arXiv:2507.14736*, 2025.
- Towers et al. (2024)
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al.
Gymnasium: A standard interface for reinforcement learning environments.
*arXiv preprint arXiv:2407.17032*, 2024.
- Van de Ven et al. (2022)
Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias.
Three types of incremental learning.
*Nature Machine Intelligence*, 4(12):1185–1197, 2022.
- Xu et al. (2015)
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li.
Empirical evaluation of rectified activations in convolutional network.
*arXiv preprint arXiv:1505.00853*, 2015.
- Zilly et al. (2021)
Julian Zilly, Alessandro Achille, Andrea Censi, and Emilio Frazzoli.
On plasticity, invariance, and mutually frozen weights in sequential task learning.
*Advances in neural information processing systems*, 34:12386–12399, 2021.

## Appendix A Characterization of Activation Function Properties

**Table A1: Binary property grid (✓= present, – = absent). Abbreviations. HDZ: hard dead zone; NZG: non‑zero gradient for $x<0$; Sat$\pm$: two‑sided saturation; Sat$-$: negative‑side saturation; $C^{1}$: first derivative continuous; NonM: non‑monotonic segment; SelfN: self‑normalizing output; L/R${}_{\text{slp}}$: learnable or randomized slope; $f^{\prime\prime}$: non‑zero second derivative. *Gradients are small but non‑zero except at extreme inputs (risk of effective inactivity via saturation). ^† ELU is $C^{1}$ only for $\alpha=1$. ^‡ SELU has a small derivative jump at $x=0$ because $\alpha\neq 1$. ^♣ SELU’s self-normalizing behavior holds under the prescribed $(\lambda,\alpha)$. ^□ Unconstrained activations are smooth, non-monotonic, and non-saturating; outputs can grow large under training. ^∘ Rational do not have “slopes” like Leaky-ReLU/PReLU; they learn polynomial coefficients. ^∗ Each branch retains the ReLU dead zone, but concatenation ensures that for any input at least one branch is active, avoiding global inactivity. ^△ Proposed in this work. ^◆ Potentially non-monotonic depending on the choice of slope parameter (and on the randomization bounds for R-Smooth-Leaky).**
| Activation | HDZ | NZG | Sat$\pm$ | Sat$-$ | $C^{1}$ | NonM | SelfN | $L/R_{slp}$ | $f^{\prime\prime}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ReLU Nair & Hinton (2010) | ✓ | – | – | ✓ | – | – | – | – | – |
| LeakyReLU Maas et al. (2013) | – | ✓ | – | – | – | – | – | – | – |
| PReLU He et al. (2015) | – | ✓ | – | – | – | – | – | ✓ | – |
| RReLU Xu et al. (2015) | – | ✓ | – | – | – | – | – | ✓ | – |
| Sigmoid | – | ✓* | ✓ | ✓ | ✓ | – | – | – | ✓ |
| Tanh | – | ✓* | ✓ | ✓ | ✓ | – | – | – | ✓ |
| Swish (SiLU) Ramachandran et al. (2017) | – | ✓ | – | – | ✓ | ✓ | – | – | ✓ |
| GeLU Hendrycks & Gimpel (2016) | – | ✓ | – | – | ✓ | ✓ | – | – | ✓ |
| ELU Clevert et al. (2015) | – | ✓ | – | ✓ | ✓^† | – | – | – | ✓ |
| CELU Barron (2017) | – | ✓ | – | ✓ | ✓ | – | – | – | ✓ |
| SELU Klambauer et al. (2017) | – | ✓ | – | ✓ | –^‡ | – | ✓^♣ | – | ✓ |
| CReLU Shang et al. (2016) | ✓^∗ | ✓ | – | – | – | – | – | – | – |
| Rational^□ Delfosse et al. (2021b) | – | ✓ | – | – | ✓ | ✓ | – | –^∘ | ✓ |
| Smooth-Leaky^△ | – | ✓ | – | – | ✓ | ✓^◆ | – | – | ✓ |
| R-Smooth-Leaky^△ | – | ✓ | – | – | ✓ | ✓^◆ | – | ✓ | ✓ |

## Appendix B Experimental and Architectural Design

Models for both Case Studies [3](https://arxiv.org/html/2509.22562v1#S3) and [4](https://arxiv.org/html/2509.22562v1#S4) share the same backbone (4 $\times$ 32 conv + 256-unit MLP) and are trained on the 20-task Split-CIFAR100 benchmark; Kaiming initialization uses the correct gain for each starting slope ($\alpha_{0}{=}0.25$ for every PReLU). Metrics recorded every epoch include final average accuracy (ACC), online forgetting, dead unit fraction, gradient signal-to-noise, $\lambda_{\max}$ for the fc1$\rightarrow$fc2 block, and— for PReLU—the full trajectory $\alpha_{i}(t)$.

For *continual supervised learning* we deliberately use compact networks to accentuate capacity–limited plasticity loss: a model may attain high average online accuracy on a few tasks, yet its ability to adapt degrades over long sequences. We employ two backbones. (i) MLP: two hidden layers, each of width $100$. (ii) CNN: two $5\times 5$ convolutional layers with $16$ output channels each, every conv followed by $2\times 2$ max pooling, then two fully connected layers of width $100$. All architectures end with a linear classifier whose output size is $10$ for Permuted MNIST, Random Label MNIST, and Random Label CIFAR; $100$ for 5+1 CIFAR; and $2$ for Continual ImageNet.

For *continual RL* (Sec. [7](https://arxiv.org/html/2509.22562v1#S7)), policy and value functions share a multi–head MLP designed for sequential adaptation: a shared backbone with two hidden layers of $256$ units is trained across all tasks, while for each environment we instantiate a dedicated input adapter that maps its observation space to the backbone and dedicated output head(s) for its action (and value) space. This modular design reuses core features while accommodating heterogeneous state–action spaces.

### B.1 Limitations and implications of multi-parameter activation design.

Smooth-Leaky and Randomized Smooth-Leaky expose several shape-controlling hyperparameters. This enlarges the tuning space—and thus the cost—but also grants useful control, as reflected in our benchmarks. A common response is to replace sweeps with adaptive or learnable parameters. However, as shown in Section [3](https://arxiv.org/html/2509.22562v1#S3), adaptive, granular learnable slopes (e.g., PReLU) underperformed in all our settings, while bounded, stochastic constraints (e.g., RReLU) worked better. In continual learning, where short-term objectives dominate, unconstrained learned parameters can drift to task-local optima that hurt performance across tasks. Attempts to automate shape learning (Bingham & Miikkulainen, 2023; Bingham et al., 2020; Manessi & Rozza, 2018) with highly expressive forms (e.g., Rational activations Delfosse et al. (2020)) also lagged behind our first-principles designs here. This suggests that we have not yet reconciled expressivity with robust automation. A promising direction is to rethink where and on what timescale hyperparameters are adapted—potentially decoupling their updates from the main training loop to better handle non-stationarity.

### B.2 CReLU integration.

To insert CReLU without changing the hidden width or the rest of the network, we follow a capacity–neutral design: for any hidden layer with target width $H$, the preceding linear layer produces $H/2$ pre–activations $z\in\mathbb{R}^{H/2}$, and we apply
$\mathrm{CReLU}(z)=\big[\max(z,0),\,\max(-z,0)\big]\in\mathbb{R}^{H}$,
which restores the width to $H$ by concatenation. This mirrors the “CReLU (+half)” configuration: the parameter count of the producer linear is halved (input$\times H/2$ instead of input$\times H$), the consumer linear still receives $H$ features, and the forward shape everywhere else is identical to the ReLU baseline. Thus CReLU’s representational benefit (explicit positive/negative phase features) is preserved while keeping model size and interface unchanged, enabling plug–in replacement of the activation without branching logic in the forward pass.

For convolutional blocks with target channel width $C$ and fully connected blocks with target width $H$, we apply CReLU in a capacity–neutral manner: the producer layer outputs $C/2$ (or $H/2$) pre–activations $z$, and we apply $\mathrm{CReLU}(z)=[\max(z,0),\,\max(-z,0)]$ along the channel/feature dimension, restoring the width to $C$ (or $H$) by concatenation. This mirrors the “CReLU (+half)” configuration, preserving parameter count relative to a ReLU baseline while exposing explicit positive/negative phase features. Subsequent layers therefore see the same interface shapes as in the baseline, allowing CReLU to be swapped in without modifying the forward pass.

### B.3 Hyper-parameter Sweeps for Activation Functions

**Table B1: A summary of the hyper-parameter sweeps performed for each activation function. The table details the specific values, ranges, and distributions tested for the corresponding symbols during our experiments. Best-performing hyperparameters are used to compute the results in Sections [3](https://arxiv.org/html/2509.22562v1#S3) and [4](https://arxiv.org/html/2509.22562v1#S4). ^△ Proposed in this work. See Section [5](https://arxiv.org/html/2509.22562v1#S5) and App. [G](https://arxiv.org/html/2509.22562v1#A7).**
| Activation | Symbol(s) | Values explored |
| --- | --- | --- |
| ReLU / CReLU | $\alpha$ | 0.0 |
| Leaky ReLU | $\alpha$ | 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 |
| ELU / CELU | $\alpha$ | 0.1, 0.5, 1.0, 1.5, 2.0, 2.2, 2.4, 2.6, 2.8, 3.0, 3.3, 3.5, 3.6, 3.9 |
| SELU | $\alpha$ | 1.0, 1.3, 1.673, 2.0, 2.3, 2.4, 2.6, 2.8, 3.0, 3.1, 3.3, 3.5, 3.7 |
| Swish / GeLU | $\beta$ | 0.01, 0.05, 0.1, 0.5, 0.8, 1.0 |
| RReLU | $U=[l,u]$ | [0.01,0.05], [0.05,0.10], [0.1,0.3], [0.125,0.333], [0.3,1.0],<br>[0.4,1.0], [0.5,1.0], [0.6,1.0], [0.6,0.8], [0.7,1.0], [0.8,1.0],<br>[0.9,1.0], [1.0,1.5], [1.6732,1.6732], [1.4232,1.9232], [1.168,2.178]<br>[0.9232,2.4232], [1.548,1.798], [0.673,2.673], [0.423,2.923] |
| PReLU | scope | global, layer, neuron |
| Sigmoid / Tanh | – | – |
| Rational | P(x), Q(x), V, $A_{f}$ | Grid over:<br>$(P(x),Q(x))\in\{(7,6),(5,4),(3,2)\}$<br>$V\in\{A,B,C,D\}$<br>$A_{f}\in\{ReLU,Leaky-ReLU,Swish,Tanh,Sigmoid\}$ |
| Smooth-Leaky^△ | $(c,p,\alpha)$ | Grid over:<br>$c,p\in\{1,2,3,4,5\}$<br>$\alpha\in\{0.1,0.3,0.5,0.65,0.7,0.8,0.9\}$<br>(total 175 combos). |
| Rand-Smooth-Leaky^△ | $(c,p,l,u)$ | Grid over:<br>$c,p\in\{1,2,3,4,5\}$<br>lower bound $l\in\{0.01,0.3,0.4,0.5,0.6,0.7\}$<br>upper bound $u\in\{0.05,0.8,1.0\}$<br>(total 450 combos); $r\sim\mathcal{U}(l,u)$ sampled<br>per element during training. |

The hyperparameter sweep of the activation function for continual supervised problems and continual RL is similar to those presented in Table LABEL:tab:hyper_sweep with the exception of Smooth-Leaky and Ran. Smooth-Leaky where we will add new values to the shape-controlling variables $c$ and $p$, as well as new upper ($u$) and lower $l$ bounds. Therefore, the full list of variables will be: $c,p\in\{0.1,0.3,0.5,0.8,1,2,3,4,5\}$, and the new bounds are $[l,u]=[0.673,2.673],[0.55,1.0],[0.05,0.70],[0.01,0.02],[0.01,0.10]$, which were used to test a wider range of possibilities outside of intuitive bounds. We also introduce two activations functions presented in previous works as mentioned earlier: CReLU and Rational Activations. CReLU, as a concatenated ReLU, only has the original negative value $\alpha$ of 0. On the other hand, Rational Activations are defined by the polynomial of the numerator P(x), denominator Q(x), the rational version (V) used (e.g., A, B, C, or D), and the function ($A_{f}$) that is trying to approximate (e.g., ReLU, Swish, etc).

### B.4 i.i.d. vs Class-Incremental Continual Learning comparison Hyperparameters

**Table B2: Optimal Hyperparameters ($HP$) and Learning Rate ($LR$) on Split-CIFAR100. A dash ($-$) indicates that such activation function uses the unique or baseline parameter (e.g., ReLU does not have any shape-controlling parameter since is linear on the positive right side and 0 on the negative left side).**
| Activation | i.i.d. | CL |  |  |
| --- | --- | --- | --- | --- |
|  | $HP$ | $LR$ | $HP$ | $LR$ |
| ReLU | $-$ | 0.001 | $-$ | 0.0001 |
| Leaky-ReLU | 0.01 | 0.001 | 0.7 | 0.0003 |
| RReLU | (0.125, 0.333) | 0.001 | (0.673, 2.673) | 0.0003 |
| PReLU | layer | 0.001 | neuron | 0.001 |
| Swish | 1.0 | 0.001 | 0.05 | 0.0001 |
| GeLU | 1.0 | 0.001 | 0.05 | 0.001 |
| CeLU | 1.0 | 0.001 | 2.4 | 0.001 |
| eLU | 1.5 | 0.001 | 3.9 | 0.0001 |
| SeLU | 1.673 | 0.001 | 3.7 | 0.0001 |
| Tanh | $-$ | 0.001 | $-$ | 0.0001 |
| Sigmoid | $-$ | 0.001 | $-$ | 0.001 |

Following the grid search over the hyper‑parameters listed in Table LABEL:tab:hyper_sweep, we chose the best‑performing configuration for each activation function presented in Table [B2](https://arxiv.org/html/2509.22562v1#A2.T2). Each continual‑learning run comprised 100 epochs across 20 tasks (100 classes in total, five classes per task). We adopted a standard Experience Replay (ER) framework with a random‑sampling buffer of size $|M|=10{,}000$, capped at 500 examples per task. The larger‑than‑usual buffer was intended to mitigate catastrophic forgetting and ensure consistent performance across activation functions, allowing us to isolate the effects of reduced plasticity (following the intuition presented in Dohare et al. (2024)).

### B.5 Understanding Class-Incremental Continual Learning Metrics

To properly understand the choice of metrics throughout this paper, we first need to define and explain some of the classical metrics used in continual learning—*Final Average Accuracy (ACC)*, *Backward Transfer (BWT)*, and *Forward Transfer (FWT)*—and relate them to *loss of plasticity*. Consider a learning process unfolding over $N{+}1$ sequential *phases/tasks* $t\!=\!0,\dots,N$. Between consecutive phases $t{-}1$ and $t$ ($t\!\geq\!1$), the data distribution, environment dynamics, reward signal, or task objective may change, inducing non-stationarity; we denote the distribution at phase $t$ by $E_{t}$. The model enters phase $t$ with parameters $\theta_{t-1}$ and is updated for $T_{\text{phase}}$ adaptation steps(^3^33“Steps” may be epochs, mini-batches, or timesteps, depending on the domain.), producing intermediate iterates $\theta_{t,k}$ ($k\!=\!1,\dots,T_{\text{phase}}$) and terminating at $\theta_{t}\!=\!\theta_{t,T_{\text{phase}}}$.

Then: (i) $\mathrm{ACC}_{T}\!=\!\tfrac{1}{T}\sum_{i=1}^{T}A_{T,i}$ summarizes performance after learning $T$ tasks but mixes current-task performance, retention of past tasks, and interference;
(ii) $\mathrm{BWT}_{T}\!=\!\tfrac{1}{T-1}\sum_{i=1}^{T-1}\big(A_{T,i}-A_{i,i}\big)$ quantifies forgetting/retention; and
(iii) $\mathrm{FWT}_{T}$ compares pre-training performance on future tasks to a suitable baseline, capturing positive/negative transfer.
Although $\mathrm{BWT}_{T}$ and $\mathrm{FWT}_{T}$ are widely reported in continual learning, in our setting we focus on performance metrics related to $\mathrm{ACC}_{T}$ following recent work (Kumar et al., 2023). We therefore briefly outline their differences and use-cases.

Because $\mathrm{ACC}_{T}$ averages over tasks, it cannot isolate *plasticity*—fast adaptation within the *current* phase. In our class-incremental continual learning setup with experience replay (Sec. [2](https://arxiv.org/html/2509.22562v1#S2); (Van de Ven et al., 2022)), we report $\mathrm{ACC}_{T}$ for overall performance. For the rest of our continual supervised learning benchmarks we shift to a more plasticity-focused evaluation and report lifetime aggregate of per-task *Average Online Task Accuracy* named *Total Average Online Accuracy* which track within-phase learning and, when task difficulty is comparable, reveal trends of *loss of plasticity*.

#### B.5.1 Different Accuracy metrics across Continual Learning

##### Average accuracy across seen tasks ( ACC T \mathrm{ACC}_{T} ).

This is the standard average accuracy used in (offline or online) class-incremental CL to summarize performance after training up to task $T$. Let $\mathrm{acc}_{i}^{(T)}$ be the test accuracy on task $i$ *measured after finishing training on task $T$*. Then

$$ $\mathrm{ACC}_{T}\;=\;\frac{1}{T}\sum_{i=1}^{T}\mathrm{acc}_{i}^{(T)}.$ (3) $$

High $\mathrm{ACC}_{T}$ implies some combination of plasticity (learning new tasks) and stability (retaining old ones). However, $\mathrm{ACC}_{T}$ mixes current-task performance, retention on past tasks, and the interference caused by learning the current task, so it does not isolate plasticity on the most recent shift. In our Case Studies [3](https://arxiv.org/html/2509.22562v1#S3)–[4](https://arxiv.org/html/2509.22562v1#S4), which use an experience-replay buffer, we report this metric as final accuracy.

##### Average online task accuracy (per task).

To quantify adaptation on a single task independently of past-task test performance, we use the mean online accuracy over the batches of task $T_{i}$. Let $M_{i}$ be the number of mini-batches in task $i$ and let $a_{i,j}$ denote the online accuracy on batch $j$ of task $i$. Define

$$ $\mathrm{AOA}(T_{i})\;=\;\frac{1}{M_{i}}\sum_{j=0}^{M_{i}-1}a_{i,j}.$ (4) $$

This captures how quickly the agent learns the current task (plasticity). A downward trend of $\mathrm{AOA}(T_{i})$ across tasks of equal difficulty indicates *plasticity loss*.

##### Total average online accuracy (lifetime).

For optimal hyper-parameter selection we also aggregate online accuracy over *all* batches seen so far (common in online CL Cai et al. (2021); Ghunaim et al. (2023); Prabhu et al. (2023); Kumar et al. (2023)). Let $B_{\leq T}=\sum_{i=1}^{T}M_{i}$ be the total number of processed batches up to task $T$, and let $a_{t}$ be the online accuracy at global batch index $t$. Then

$$ $\mathrm{TAOA}_{\leq T}\;=\;\frac{1}{B_{\leq T}}\sum_{t=0}^{B_{\leq T}-1}a_{t}\;=\;\frac{1}{\sum_{i=1}^{T}M_{i}}\sum_{i=1}^{T}\sum_{j=0}^{M_{i}-1}a_{i,j}.$ (5) $$

If all tasks have equal length $M_{i}\equiv M$, this reduces to

$$ $\mathrm{TAOA}_{\leq T}\;=\;\frac{1}{MT}\sum_{t=0}^{MT-1}a_{t}.$ (6) $$

We distinguish this lifetime aggregate from the per-task quantity $\mathrm{AOA}(T_{i})$. This metric is reported in Section [6](https://arxiv.org/html/2509.22562v1#S6).

### B.6 Curvature Metrics

In order to study the properties of the curvature of a neural network and how it affects the loss of plasticity, we need to work with the Hessian matrix. For a loss function $L(\theta)$ (where $\theta$ represents all the parameters of the network), the Hessian matrix $H$ is defined as:

$$ $H=\nabla^{2}L(\theta)$ $$

This is a symmetric matrix that captures the second-order derivatives (curvature) of the loss function with respect to the parameters.

#### B.6.1 Principal curvature

The principal curvature is defined as the maximum eigenvalue $\lambda_{\text{max}}$ of the Hessian $H$. In other words, it’s the largest value in the set of eigenvalues $\{\lambda_{1},\lambda_{2},\dots,\lambda_{d}\}$ where $d$ is the number of parameters (or the dimension of $H$). The largest eigenvalue indicates the steepest direction of curvature. If you move in the direction of the corresponding eigenvector (the principal direction), the loss increases most rapidly. We employ the Power iteration algorithm to find the dominant eigenvector, and thereby the dominant eigenvalue using the Rayleigh Quotient:

$$ $\lambda_{\text{max}}\approx v_{k}^{\top}Hv_{k}$ $$

The principal eigenvector is the direction in parameter space corresponding to $\lambda_{\text{max}}$. This is the steepest direction—any movement along that eigenvector direction leads to the largest second-order change in the loss.

#### B.6.2 Effective Rank

To quantify the intrinsic dimensionality of the loss‐landscape curvature in a given layer, we collect per‐sample gradients
$\{g_{i}\}_{i=1}^{m}\subset\mathbb{R}^{d}$ and stack them into a matrix $G=[g_{1},\dots,g_{m}]\in\mathbb{R}^{d\times m}$. We then form the Gram matrix

$$ $M\;=\;G^{\top}G\;\in\;\mathbb{R}^{m\times m},$ $$

compute its singular values $\sigma_{1}\geq\sigma_{2}\geq\cdots\geq\sigma_{m}\geq 0$, and define the *effective rank* at threshold $\tau\in(0,1)$ by

$$ $\mathrm{erank}_{\tau}(G)\;=\;\min\Bigl\{\,k\;:\;\frac{\sum_{j=1}^{k}\sigma_{j}}{\sum_{j=1}^{m}\sigma_{j}}\;\geq\;\tau\Bigr\}.$ $$

Effective Rank represents how many independent directions of principal directions that account for at least a fraction $\tau$ of the total “energy” in the gradient subspace Kumar et al. (2020). Therefore, effective rank tells you the number of eigenvectors (or directions) that are necessary to capture a specified fraction (for example, 99%) of the total curvature energy of the Hessian. In other words, it indicates how many independent directions contribute significantly to the curvature. If the effective rank is low, then most of the curvature is concentrated in just a few directions; if it’s high, then the curvature is spread out over many directions.

## Appendix C Expanding on Case Study 1: Negative‑Slope ‘Goldilocks Zone’

### C.1 ‘Goldilocks zone’ for negative slopes (with shape-matched comparison)

Matching each activation’s effective negative slope to that of Leaky-ReLU shows that—across all tested shape-controlling parameters—no alternative attains equal or higher final average accuracy, reinforcing the presence of a ‘Goldilocks zone’. Consistently, Fig. [C1](https://arxiv.org/html/2509.22562v1#A3.F1)b shows the fraction of dead units declining as the effective slope enters the green band, aligning with the view that dead units contribute to plasticity loss; accuracy peaks in the same region.

Figure: Figure C1: Left: Final accuracy vs. effective negative slope $\bar{s}$. Right: Dead-unit fraction vs. $\bar{s}$. Linear-leak families (blue) peak for $\bar{s}\!\in\![0.6,0.9]$. Smooth-tailed activations (ELU/CELU/SELU, Swish, GELU) are plotted on the same $\bar{s}$ axis; they underperform within the ‘Goldilocks zone’ and only approach the linear-leak peak when $\bar{s}\!>\!1$, reflecting concentrated near-zero responsiveness and vanishing tails.
Refer to caption: h1_4_acc_dead_eff_slopes.png

We also tested in Sec .[3](https://arxiv.org/html/2509.22562v1#S3) if smooth-tailed activations (ELU, CELU, SELU, Swish, GELU) replicate Leaky-ReLU’s behavior given that their average negative-side derivative ($\bar{s}$) was matched. Fig. [C1](https://arxiv.org/html/2509.22562v1#A3.F1) provides a visual argument for the explanation provided in the main text. We show that $\bar{s}$ overlays these smooth functions, and throughout the ‘Goldilocks’ $\bar{s}$ range (e.g.,$\bar{s}\approx 0.1-0.8$), they consistently underperform Leaky-ReLU in ACC by approximately 5–7 percentage points and generally with higher dead units.

On the other hand, we also check their $\lambda_{\max}$ curves (Fig. [C2](https://arxiv.org/html/2509.22562v1#A3.F2), which also sit noticeably above Leaky-ReLU’s optimal low-curvature trough. This suggests that the non-constant, often exponential or logistic, shape of their negative tails fails to flatten landscape curvature as effectively as Leaky-ReLU’s constant slope, even when average gradient throughput ($\bar{s}$) is matched. However, at extreme effective slopes ($\bar{s}>1.0$) where Leaky-ReLU’s performance collapses due to curvature explosion, functions like ELU and SELU maintain more moderate curvature and salvage higher ACC (around 22%), indicating better robustness at this specific extreme.

### C.2 Failure Modes for Slope Magnitude.

Continuing with the explanation in Sec. [3](https://arxiv.org/html/2509.22562v1#S3); the so-called ‘Goldilocks phenomenon’ highlights two distinct failure modes at opposite ends of the negative slope range. As shown in Fig. [C2](https://arxiv.org/html/2509.22562v1#A3.F2) (left) the largest Hessian eigenvalue, $\lambda_{\max}$, follows a U-shaped curve. Curvature is high when the negative slope $\bar{s}\approx 0$ (due to many "dead" units) and rises again when $\bar{s}>0.9$ (as the function behaves like an identity map). The ‘Goldilocks zone’ lies between $\bar{s}\approx 0.6$ and $0.9$, where $\lambda_{\max}$ is at its lowest. The effective rank, illustrated in Fig. [C2](https://arxiv.org/html/2509.22562v1#A3.F2) (right) offers more insight. The rank is lowest at both extremes: near $\bar{s}\approx 0$ because of widespread unit death, and at very high $\bar{s}$ where extreme curvature creates a stiff, low-dimensional learning subspace.

Optimal plasticity is achieved in the ‘Goldilocks zone’ where the network balances having active learning directions with a compact representation and flat curvature. Therefore, loss of plasticity is caused by unit death at very small negative slopes and by high curvature at very large negative slopes. A ‘Goldilocks zone’, independently of the range that it reports, is currently found empirically. However, its mere existence suggests that there are optimal configurations.

Figure: Figure C2: Left: effective rank of the gradient Gram matrix. Right: dominant Hessian eigenvalue $\lambda_{\max}$. Smooth-tailed activations (swish, celu) show spikes at large $\bar{s}$, while constant-slope leaks remain comparatively stable.
Refer to caption: h1_4_curv_eff_rank.png

We also investigated the primary drivers of functional plasticity loss, we compare across a set of representative activation configurations and all negative slope variations. As provided in the main section results, the correlation between Final Average Accuracy (ACC) and dead unit fraction is strong and significant (Pearson $(r)=-0.59$ ($p{=}5.7{\times}10^{-8}$), as seen in Fig. [C3](https://arxiv.org/html/2509.22562v1#A3.F3), right. In contrast, the correlation between ACC and final scaled gradient norms is weak and not significant (Pearson $(r){=}\,-0.28$) ($p{=}0.019$) (Fig. [C3](https://arxiv.org/html/2509.22562v1#A3.F3), left).

Settings that maintain dead unit fraction below approximately 8% consistently achieve high ACC (27–33%), whereas dead unit rates above 25% lead to ACC collapsing to 20% or lower. This indicates that while gradient magnitude is a factor, the rise in dead units (gradient starvation) is a more direct and reliable indicator of plasticity loss than the raw magnitude of gradients, which can be small in well-adapted, flat minima or large in failing networks with high curvature. The results underscore that plasticity is not about maximizing any single statistic but about achieving a delicate balance, primarily tuned by the negative slope characteristics, to ensure unit survival, maintain diverse learning directions, and promote a benign loss landscape.

Figure: Figure C3: Person (r) correlation between Plasticity Loss and diverse metrics to evaluate the primary drivers of such phenomenon for some activation functions evaluated across Case Study 1 (see Section [3](https://arxiv.org/html/2509.22562v1#S3)) Left: Dead Units; Right: Average Gradient Norm.
Refer to caption: h1_5_correlation_diagnostics.png

### C.3 Adaptive, granular slopes are useful—but need guidance to stay in-band

As seen in Sec. [3](https://arxiv.org/html/2509.22562v1#S3), PReLU-N, with its per-neuron learnable $\alpha_{i}$, demonstrates this adaptability. Fig. [C4](https://arxiv.org/html/2509.22562v1#A3.F4) (left) shows the dynamic evolution over time of how PReLU-N learns heterogeneous $\alpha_{i}$ values, ideally we will observe many neurons dynamically adjusting their slopes into a beneficial range (e.g., 0.3-0.6, approaching the Goldilocks zone), while others can remain near zero if required by their local input statistics. However, our PReLU-N configuration (with default initialization $\alpha=0.25$) undershot the empirically optimal $\alpha\approx 0.7$ for fixed Leaky-ReLU but its performance surpassed PReLU-G/L which tended towards slope decay and lower ACC ($\alpha$ evolution for PReLU-G/L shown in Figs. [C5](https://arxiv.org/html/2509.22562v1#A3.F5)). All PReLU scopes (Neuron, Layer, Global) evolve towards values outside the Leaky-ReLU-family ‘Goldilocks Zone’.

While not optimal, this might suggest that neuron-wise adaptability is a valuable trait, potentially enhanced further with optimized initialization or per-parameter learning rates if the learning can be guided towards the ‘Goldilocks zone’. Which also opens up the question of properly finding such zone for different experimental settings.

Figure: Figure C4: PReLU-N per-neuron learnable $\alpha$. Left: Distribution of all individual neurons’ $\alpha$ per layer (activation.N) and the pre-defined ‘Goldilocks zone’ representing the best-performing $\alpha$ value in Leaky-ReLU. Right: Fraction of $\alpha$’s $<\epsilon$ indicating values for which the post-activation will be ’saturated’ (too small to produce significant changes).
Refer to caption: h3_prelu_n_alpha_evol.png

Figure: Figure C5: PReLU-GŁper-neuron learnable $\alpha$. Left: PReLU-G. Distribution of all neurons’ $\alpha$ for the whole network (1 learnable parameter shared across all layers); Right: PReLU-L. PReLU-G. Distribution of all layers’ parameters $\alpha$ for the whole network (1 learnable parameter per layer).
Refer to caption: h3_prelu_LG_alpha_evol.png

### C.4 Saturation Fraction as metric for ’dead units’.

Regarding Saturation Fraction used in Fig. [C1](https://arxiv.org/html/2509.22562v1#A3.F1) we acknowledge that recent works, such as Sokar et al. (2023) or Liu et al. (2025), highlighted the ’dormant neuron phenomenon’, and ’GraMa’ (Gradient Magnitude Neural Activity Metric) in deep reinforcement learning, proposing a valuable metric based on a neuron’s average absolute activation (or gradient) relative to its layer-neuron connections to identify consistently underutilized units. While this provides a generalized approach to ’dead units’, our investigation into a diverse array of activation functions—each with unique output ranges and saturation characteristics (e.g., ReLU’s unbounded positive output versus Tanh’s strict [-1, 1] bounds, or ELU’s negative saturation plateau)—necessitated a more concrete definition of what constitutes a ’saturated’ or ’effectively dead’ state based on post-activation values. Therefore, while sharing the goal of identifying inactive units, our saturation fraction metrics employ criteria specific to each activation function family (such as proximity to +/-1 for Tanh, near-zero output for ReLU, or low output magnitude relative to batch statistics for functions like Swish or Leaky ReLU). This specific approach allows us to more precisely capture the distinct ways different activation architectures can lead to or avoid states of unresponsiveness under duress, rather than applying a single relative threshold across functions with fundamentally different output scales and properties. See code for the full criteria used for each activation function.

## Appendix D Expanding on Case Study 2: Saturation‑Threshold Stress Test

### D.1 Strees Protocol

Let $\Gamma=\{1.5,0.5,0.25,2.0\}$ be the set of shock amplitudes and let $C_{l}$ be a user–defined cycle length (we use $C_{l}=10$ for results in the main body Sec. [4](https://arxiv.org/html/2509.22562v1#S4)). During training we step through the epochs of every task as

$$ $x^{\text{pre}}\;\longleftarrow\;\gamma_{k}(t)\;x^{\text{pre}},\qquad\gamma_{k}(t)=\begin{cases}\Gamma_{k},&t\bmod C_{l}=0,\[4.0pt] 1,&\text{otherwise},\end{cases}$ $$

where the index $k$ is advanced cyclically $k\!\leftarrow\!(k+1)\bmod|\Gamma|$ *each time a shock epoch occurs*. Thus every $C_{l}$ epochs we devote exactly *one* epoch to a
scale–shock whose value alternates $1\!\to\!1.5\!\to\!1\!\to\!0.5\!\to\!1\!\to\!0.25\!\to\!1\!\to\!2.00\!\to\!1\!\to\!\dots$. The multiplicative factor is applied *after* all layers and *before* its non-linearity; all other epochs run with $\gamma\!=\!1$.

### D.2 Derivative-Floor rule.

The data presented in Figure [1](https://arxiv.org/html/2509.22562v1#S4.F1) provides strong support for the idea which posits that activation functions with non-zero derivative floors offer superior desaturation and recovery dynamics.

Fig. [1](https://arxiv.org/html/2509.22562v1#S4.F1) (left) reveals that, as expected, stronger "expanding" shocks ($\gamma$=1.5,2.0) generally induce a higher overall saturation impact (AUSC) than "shrinking" shocks ($\gamma$=0.25,0.5) across all floor types.

Critically, activations in the "Non-Zero Floor" category consistently demonstrate the most effective minimization of AUSC, maintaining the lowest values particularly under expanding shocks. Conversely, "Zero Floor" activations exhibit the highest AUSC, indicating a greater susceptibility to prolonged saturation. The "Effective Non-Zero Floor" group shows a mixed response, with AUSC values sometimes higher than "Zero Floor" for certain shrinking shocks (e.g., $\gamma$=0.5) but better than "Zero Floor" under strong expansion.

This pattern of robustness for "Non-Zero Floor" activations is further evidenced in their SF recovery capabilities, as shown in Fig. [1](https://arxiv.org/html/2509.22562v1#S4.F1) (middle) and Fig. [1](https://arxiv.org/html/2509.22562v1#S4.F1) (right). While successful recovery times tend to be quick (around 1 epoch or slightly more) for stronger shocks ($\gamma$=1.5,2.0) across most types that do recover, the "Non-Zero Floor" category stands out by also maintaining the lowest non-recovery rates, especially for expanding shocks where it approaches 0-5%. In contrast, "Zero Floor" activations not only take noticeably longer to recover SF during shrinking shocks but also suffer from extremely high non-recovery rates across all shock conditions. The ’Effective Non-Zero Floor’ group (typically smooth-tailed functions) demonstrates rapid SF recovery Fig. [1](https://arxiv.org/html/2509.22562v1#S4.F1) (middle) if recovery occurs. However, they suffer from high non-recovery rates, particularly akin to ’Zero Floor’ types during shrinking shocks, and show less consistent AUSC improvements compared to ’Non-Zero Floor’ activations. This indicates their negative tail mechanisms, despite providing an average non-zero gradient, may not consistently prevent saturation or ensure recovery across diverse shocks.

In conclusion, "Zero Floor" activations are clearly detrimental to saturation resilience. While "Effective Non-Zero Floor" functions offer fast recoveries when successful, it is the presence of a consistent and sufficiently large "Non-Zero Floor" that most effectively minimizes overall saturation impact and ensures reliable, rapid recovery from saturation-inducing shocks."

### D.3 Two-sided penalty.

We hypothesized that activations that saturate on *both* sides (Sigmoid, Tanh, Swish/GELU at very low $\beta$) will accumulate $\geq 50\%$ more saturation and take $\geq 50\%$ longer to recover than one-sided families. The data provides nuanced support for this. As seen in Fig. [2](https://arxiv.org/html/2509.22562v1#S4.F2), "Two-Sided" functions indeed accumulate more saturation, evidenced by having the highest average Peak SF Fig. [2](https://arxiv.org/html/2509.22562v1#S4.F2) (left) and high AUSC values Fig. [2](https://arxiv.org/html/2509.22562v1#S4.F2) (right). While their successful SF recoveries are very fast—around one epoch Fig. [2](https://arxiv.org/html/2509.22562v1#S4.F2) (middle)—this is severely counterbalanced by a very high non-recovery rate of 49.83% (data not shown in figure). This high failure rate means that, in practice, they are far less reliable at desaturating than "One-Sided (Kink)" activations, which had a non-recovery rate of 13.30%. If non-recovery is considered an infinite recovery time, then "Two-Sided" functions effectively "take longer to recover" on average due to frequent outright failure. Interestingly, "One-Sided (Smooth)" functions also exhibit a high non-recovery rate (48.93%), barely surpassing that of "Two-Sided" functions, despite their successful recoveries also being very fast. This suggests that while their smooth negative tail can allow for quick desaturation if conditions are right, they are also prone to getting stuck in a saturated state.

Therefore, the penalty for two-sided saturation is evident in both the magnitude of saturation experienced (Peak SF, AUSC) and the overall reliability of recovery (high non-recovery rates). The "One-Sided (Kink)" group, while having a tail of longer successful recovery times, is actually the most reliable at achieving recovery.

### D.4 The width of the dead band predicts shock sensitivity.

As studied in Sec. [4](https://arxiv.org/html/2509.22562v1#S4), the Dead-Band Sidth (DBW) critically predicts how severely a network with any activation function will be impacted by saturation-inducing shocks. As previously defined, the DBW score represents the proportion of a typical pre-activation input range where the magnitude of the activation’s first derivative, $\lvert\varphi^{\prime}(x)\rvert$ falls below certain threshold, $\epsilon<10^{-3}$. A higher Dead-Band Width Score thus indicates that a larger portion of common inputs will drive the activation into a state of significantly diminished gradient flow.

Therefore, such hypothesis states that this analytically derived DBW will positively correlate with experimentally observed adverse saturation dynamics. Specifically, activation functions with a larger DBW are expected to exhibit a greater overall saturation impact (higher average Area Under Saturation Curve, Avg. AUSC) and potentially longer average Saturation Fraction (SF) recovery times following a shock. This is because a wider intrinsic dead-band suggests that pre-activation scaling shocks are more likely to push numerous units into these unresponsive regions, and keep them there, diminishing the network’s ability to efficiently desaturate and resume learning.

Figure: Figure D1: Correlation of Dead-Band Width Score with Saturation Recovery Metrics (All Gammas Aggregated). Each point represents an activation function configuration. (Left): Average Area Under Saturation Curve (Avg. AUSC) vs. Dead-Band Width Score. A strong positive correlation (Pearson r=0.83, p=0.0016) is observed. (Middle): Average Saturation Fraction (SF) Recovery Time (for successful recoveries, measured by epochs) vs. Dead-Band Width Score. No significant correlation is found (Pearson r=-0.25, p=0.45). (Right): Average SF Non-Recovery Rate (%) vs. Dead-Band Width Score. A strong positive correlation (Pearson r=0.84, p=0.0013) is observed, indicating functions more prone to saturation are more likely to fail SF recovery."
Refer to caption: h2_3_dead_band_score.png

Figure [D1](https://arxiv.org/html/2509.22562v1#A4.F1) strongly supports this hypothesis when considering Avg. AUSC (left) and Avg. SF Non-Recovery Rate (right). Activations with a higher analytical Dead-Band Width Score do, in fact experience a greater overall saturation impact (higher AUSC), while also being much more likely to fail to recover their saturation fraction after a shock. However, for many activations, if they manage to recover, they do so very quickly (in 1 epoch on average), and this speed does not seem to strongly depend on their Dead-Band Width Score, as observed by looking at Avg. SF Recovery Time (middle).

Overall, these results suggest that the Dead-Band Width Score is a good predictor of the likelihood of getting stuck in a saturated state and the overall severity of that saturation, but not necessarily of how many steps it takes to get out if you manage to get out quickly.

### D.5 Performance recovery time on Case Study 2

To assess the functional recovery of the network after pre-activation shocks, we measured the performance recovery time $\tau_{95}$, defined as the number of epochs required to regain 95% of the pre-shock validation accuracy on the current task. This metric provides insight into the immediate resilience of the network’s learning capability for the task at hand when subjected to sudden internal perturbations.

Overall, the rate of complete performance non-recovery (failure to reach 95% of pre-shock current-task accuracy within the observation window) was very low across all experiments (1.14%). This indicates that, in most instances, the model’s ability to perform the current task bounces back effectively after the applied shocks. Figure [D2](https://arxiv.org/html/2509.22562v1#A4.F2) illustrates the mean $\tau_{95}$ for these successful recoveries, grouped by activation ’Floor Type’ (left) and ’Sidedness’ (right), across different gamma shock intensities.

Figure: Figure D2: Functional performance recovery time $\tau_{95}$ versus gamma shock intensity. Grouped by Floor Type (Left) and Sidedness (Right), showing mean epochs for successful recoveries. Most activations demonstrate rapid recovery from mild to moderate shocks; however, under strong $\gamma$=2.0 shocks, ’Two-Sided’ functions are notably fastest and most reliable (0% non-recovery), while ’Non-Zero Floor’ and ’One-Sided (Kink)’ types show slower recovery. (Overall performance non-recovery rate: 1.14%)."
Refer to caption: h2_6_perf_rec_appendix.png

For mild to moderate expanding shocks ($\gamma$=1.5) and all shrinking shocks ($\gamma$=0.25,0.5), the mean $\tau_{95}$ is consistently low (around 1.2-1.3 epochs) and remarkably similar across all floor types and sidedness categories. This suggests that for less extreme shocks, most activation functions allow for rapid recovery of performance on the current task. However, differences emerge under strong expanding shocks ($\gamma$=2.0):

- •
Impact of Floor Type (at $\gamma$=2.0): ’Non-Zero Floor’ activations, despite their strong SF recovery characteristics, surprisingly exhibit the longest average $\tau_{95}$ ($\approx$3.4 epochs) for performance and also contribute most to the few performance non-recoveries observed (2.85% non-recovery rate for this group at $\gamma$=2.0). In contrast, ’Effective Non-Zero Floor’ ($\approx$1.75 epochs $\tau_{95}$, 0.36% non-recovery) and ’Zero Floor’ ($\approx$1.9 epochs $\tau_{95}$, 0.74% non-recovery) activations recover current-task performance more quickly under these strong shocks.
- •
Impact of Sidedness (at $\gamma$=2.0): Most notably, ’Two-Sided’ activations (e.g., Sigmoid, Tanh) demonstrate exceptional current-task functional recovery. They consistently recovered performance (0% non-recovery rate in your stats) and did so fastest on average ($\approx$1.1 epochs). ’One-Sided (Smooth)’ activations also performed well. ’One-Sided (Kink)’ activations showed the slowest average $\tau_{95}$ and the highest performance non-recovery rate among sidedness categories for these strong shocks.

It is crucial to note that this $\tau_{95}$ metric reflects the immediate recovery of performance on the task currently being trained. While it indicates a certain resilience to transient internal shocks, it does not directly measure the long-term impact on catastrophic forgetting or the overall ability to learn a sequence of tasks effectively.

The ultimate test of an activation function’s suitability for continual learning lies in metrics like Final Average Accuracy (ACC) across all tasks and Average Forgetting (AF), evaluated at the end of the entire training sequence. It is in these end-to-end CL metrics that more significant and often different disparities between activation functions emerge. For instance, while ’Two-Sided’ activations show rapid current-task performance recovery here, their known issues with vanishing gradients and saturation might still lead to poorer overall ACC or higher forgetting in the full CL scenario. The current $\tau_{95}$ results offer a valuable piece of the puzzle regarding short-term resilience, but the broader impact on plasticity and stability across the entire sequence of tasks remains best assessed by holistic CL evaluation metrics.

## Appendix E Expanding on Continual Supervised Learning Experiments

Following Kumar et al. (2023), we evaluate five supervised continual image–classification benchmarks spanning two shift types: *input distribution shift* (Permuted MNIST, 5+1 CIFAR, Continual ImageNet) and *concept shift* (Random Label MNIST, Random Label CIFAR). Across all settings, training proceeds as a sequence of tasks *without task–identity signals*: the model is never told when a task switch occurs. Within each task, the learner receives mini–batches for a fixed duration (specified below) and is updated incrementally with cross–entropy on the arriving batches.
Summary of experimental settings is shown in Table [E1](https://arxiv.org/html/2509.22562v1#A5.T1).

Permuted MNIST Goodfellow et al. (2013) (input shift). We first sample a fixed subset of $10{,}000$ images from the MNIST training set. Each task is defined by drawing a new *fixed* random permutation over pixel indices and applying it to every image in the subset. The permutation is constant within a task and independent across tasks. Each task presents exactly one pass (1 epoch) over its $10{,}000$ permuted images in mini–batches of size $16$; the next task then begins with a new permutation. We train for $500$ tasks total. This setting induces strong input–space remapping while preserving label semantics within tasks, isolating rapid adaptation to input shift.

Random Label MNIST Lyle et al. (2023) (concept shift). We fix a subset of $1{,}200$ MNIST images once. For each task, we generate a fresh random label for each image in this subset, leaving the inputs unchanged but altering the input to label mapping. To encourage memorization under an arbitrary target function, the model is trained for $400$ epochs per task with batch size $16$. After $400$ epochs, a new task arrives with an independent random labeling; we run $50$ tasks in sequence. Inputs are identical across tasks; only concepts change, directly probing plasticity versus interference.

Random Label CIFAR (concept shift). Identical protocol to Random Label MNIST, but using images drawn from *CIFAR–10*. We again use a fixed subset of $1{,}200$ images and re-assign random labels independently per task. Training uses $400$ epochs per task, batch size $16$, for $50$ tasks. This mirrors the concept–shift regime on a higher–variability image domain than MNIST.

5+1 CIFAR (input shift with alternating difficulty). Tasks are constructed from *CIFAR–100* and alternate in difficulty: even–indexed tasks are *hard* and odd–indexed tasks are *easy*. A hard task contains data from $5$ distinct classes with $2{,}500$ examples total ($500$ per class); an easy task contains data from a *single* class with $500$ examples. Classes do not repeat across the sequence, ensuring non–overlapping exposure. Each task lasts $780$ timesteps, which corresponds to approximately $10$ epochs on the hard task data set with batch size $32$; easy tasks use the same $780$ time step budget for consistency. We evaluated performance on the *hard* tasks only (single-class tasks are near the ceiling for all activation functions). Alternating input diversity stresses both rapid adaptation (when diversity spikes) and retention across shifts, serving as a targeted stress test for plasticity–loss mitigation.

Continual ImageNet Dohare et al. (2024) (input shift). Each task is a binary classification problem between two distinct ImageNet classes. For every task we draw $1{,}200$ images total ($600$ per class) and *down-sample to $32{\times}32$*, following Dohare et al. (2024), to reduce compute while maintaining semantic variability. Classes do not repeat across tasks, yielding clear, non–overlapping episodes of input shift. We train for $10$ epochs per task with batch size $100$ and report task accuracy. Despite down-sampling, the setting retains ImageNet–level variability while enabling precise measurement of adaptation and retention under non–reused classes.

**Table E1: Hyperparameters and schedule per benchmark. *Timesteps* denote parameter-update steps (i.e., mini-batches) within a task. For 5+1 CIFAR, a fixed timestep budget per task implies approximate epochs depending on data size.**
| Benchmark | Per-Task<br>Data Size | Batch | Epochs | Timesteps | # Tasks |
| --- | --- | --- | --- | --- | --- |
| Permuted MNIST | $10{,}000$ images | $16$ | $1$ | $625$ | $500$ |
| Random Label MNIST | $1{,}200$ images | $16$ | $400$ | $30{,}000$ | $50$ |
| Random Label CIFAR | $1{,}200$ images | $16$ | $400$ | $30{,}000$ | $50$ |
| 5+1 CIFAR | Hard: $2{,}500$ images<br>(5 classes, 500/class) | $32$ | Hard: $\approx 10$ | $780$ | 15 |
| Easy: $500$ images<br>(1 class) | Easy: $\approx 50$ | 15 |  |  |  |
| Continual ImageNet | $1{,}200$ images/task (600/class) | $100$ | $10$ | $120$ | 500 |

### E.1 Experimental Results Expansion

Here we expand the empirical comparison of activation functions across five continual supervised benchmarks. Table [2](https://arxiv.org/html/2509.22562v1#S6.T2) reports Total Average Online Task Accuracy. Two broad patterns emerge. First, rectifiers with a learnable or randomized negative branch dominate: Leaky-ReLU, RReLU, PReLU, Smooth-Leaky, and its randomized variant consistently outperform ReLU—often by large margins (e.g., CIFAR 5+1: ReLU 4.76 vs. Rand. Smooth-Leaky 57.01). Second, smooth rectifiers (Swish/SiLU) also fare well, but tend to trail the best “leaky–family” members on the harder settings. On the memorization-friendly random-label tasks, several of these parametric/leaky activations saturate near 100%, whereas saturating sigmoid and tanh struggle on CIFAR with random labels—consistent with their known optimization brittleness under distributional churn. Overall, the strongest single performer is Rand. Smooth-Leaky, with Smooth-Leaky, RReLU, and Leaky-ReLU close behind.

Table [E2](https://arxiv.org/html/2509.22562v1#A5.T2) clarifies why these families succeed by showing the optimal shape parameters and learning rates. A striking regularity is a ‘Goldilocks zone’ for the negative linear sides: When an activation exposes a slope (or an effective initial slope) on the negative branch, the best values typically land between their respective best-performing ranges. For some it tends to be between 0.6 and 0.9, which was initially reported in Section [3](https://arxiv.org/html/2509.22562v1#S3). However, this is not a rule, and optimal ‘Goldilocks zone’ might vary between activations and settings. Nonetheless, this tends holds across Leaky-ReLU, PReLU, Smooth-Leaky, and Rand. Smooth-Leaky, and even for RReLU when considering the average of its bounds (important because that average initializes the effective leak). For these activations, we also observe that those that are within that range tend to also have higher accuracy values than the ones that are outside the preferred range. Intuitively, this regime prevents dead units and preserves gradient flow without collapsing the asymmetric gating that helps continual adaptation. We also observe a gentle LR shift: simpler datasets (Permuted/Random-Label MNIST) favor $10^{-3}$, while CIFAR 5+1 prefers $10^{-4}$, matching the increased optimization stiffness there.

Figure: Figure E1: Comparison using Total Average Online Task Accuracy across all five Continual Supervised Learning Benchmarks for all activation functions. See Tab. [2](https://arxiv.org/html/2509.22562v1#S6.T2) for the best total average online accuracies. Optimal hyperparameters can be seen in Tab. [E2](https://arxiv.org/html/2509.22562v1#A5.T2).
Refer to caption: all_benchmarks_grid.png

**Table E2: Optimal Hyperparameters for each activation function in each Continual Supervised Benchmark Problem. Represented as activation function shape parameter on the left side of the $\|$ symbol and the learning rate to the right. A dash ($-$) indicates that such activation function uses the unique or baseline parameter (e.g., ReLU does not have any shape-controlling parameter since is linear on the positive right side and 0 on the negative left side). The Total Average Online Accuracy reported for these optimal hyperparameters can be seen in Table [2](https://arxiv.org/html/2509.22562v1#S6.T2). PReLU’s $\alpha$ indicates initial parameter value. Smooth-Leaky triplets indicate $c,p,\alpha$, while Rand. Smooth-Leaky indicates $c,p,$ and bounds $[l,u]$. The tuple of values from Rational indicates Version, ((P), (Q)), Function Approx. where (P) and (Q) are the numerator and denominator degrees respectively of the polynomial.**
| Activation | Permuted<br>MNIST | Random<br>Label<br>MNIST | Random<br>Label<br>CIFAR | CIFAR<br>5+1 | Continual<br>ImageNet |
| --- | --- | --- | --- | --- | --- |
| ReLU | $-$ $\|$ 0.001 | $-$ $\|$ 0.0001 | $-$ $\|$ 0.0001 | $-$ $\|$ 0.0001 | $-$ $\|$ 0.0001 |
| Leaky-ReLU | 0.6 $\|$ 0.001 | 0.8 $\|$ 0.001 | 0.6 $\|$ 0.001 | 0.4 $\|$ 0.001 | 0.6 $\|$ 0.001 |
| Sigmoid | $-$ $\|$ 0.001 | $-$ $\|$ 0.001 | $-$ $\|$ 0.001 | $-$ $\|$ 0.0001 | $-$ $\|$ 0.001 |
| Tanh | $-$ $\|$ 0.001 | $-$ $\|$ 0.0001 | $-$ $\|$ 0.0001 | $-$ $\|$ 0.001 | $-$ $\|$ 0.0001 |
| RReLU | [0.6, 0.8] $\|$ 0.001 | [0.125, 0.333] $\|$ 0.001 | [0.6, 0.8] $\|$ 0.001 | [0.673, 2.673] $\|$ 0.001 | [0.6, 0.8] $\|$ 0.001 |
| PReLU | neuron, $\alpha$ = 1.2 $\|$ 0.001 | neuron, $\alpha$ = 0.1 $\|$ 0.001 | global, $\alpha$ = 0.65 $\|$ 0.0001 | global, $\alpha$ = 0.9 $\|$ 0.0001 | neuron, $\alpha$ = 0.65 $\|$ 0.001 |
| Swish (SiLU) | 0.05 $\|$ 0.001 | 0.01 $\|$ 0.001 | 0.01 $\|$ 0.0001 | 0.1 $\|$ 0.001 | 0.05 $\|$ 0.001 |
| GeLU | 0.5 $\|$ 0.001 | 0.8 $\|$ 0.001 | 0.05 $\|$ 0.001 | 1.0 $\|$ 0.0001 | 1.0 $\|$ 0.001 |
| CeLU | 3.6 $\|$ 0.001 | 3.6 $\|$ 0.0001 | 2.0 $\|$ 0.0001 | 3.3 $\|$ 0.001 | 3.3 $\|$ 0.001 |
| eLU | 1.0 $\|$ 0.001 | 3.6 $\|$ 0.0001 | 3.6 $\|$ 0.0001 | 3.6 $\|$ 0.001 | 1.0 $\|$ 0.001 |
| SeLU | 1.0 $\|$ 0.001 | 3.0 $\|$ 0.0001 | 3.7 $\|$ 0.0001 | 3.7 $\|$ 0.001 | 3.7 $\|$ 0.001 |
| CReLU | $-$ $\|$ 0.001 | $-$ $\|$ 0.001 | $-$ $\|$ 0.001 | $-$ $\|$ 0.001 | $-$ $\|$ 0.001 |
| Rational | A, (5, 4), Leaky-ReLU $\|$ 0.001 | D, (5, 4), Tanh $\|$ 0.0001 | A, (5, 4), Tanh $\|$ 0.001 | B, (5, 4), Swish $\|$ 0.001 | C, (5, 4), ReLU $\|$ 0.001 |
| Smooth-Leaky | 0.1, 0.3, 0.3 $\|$ 0.001 | 0.3, 0.1, 0.3 $\|$ 0.001 | 0.3, 0.5, 0.65 $\|$ 0.001 | 0.1, 3.0, 0.9 $\|$ 0.001 | 3.0, 2.0, 0.65 $\|$ 0.001 |
| Rand. Smooth-Leaky | 0.8, 1.0, 0.3, 0.6 $\|$ 0.001 | 2.0, 0.8, 0.3, 0.6 $\|$ 0.001 | 0.8, 3.0, 0.5, 0.5 $\|$ 0.001 | 0.5, 0.5, 0.673, 2.673 $\|$ 0.001 | 0.5, 0.5, 0.3, 0.3 $\|$ 0.001 |

## Appendix F Expanding on Continual Reinforcement Learning Experiments

**Table F1: Mean plasticity score across 5 seeds with 95% bootstrap confidence intervals (higher is better). Best mean per column is bolded. We also report optimal learning rate (LR) and optimal hyperparameters (HP) per activation function. In the case of bounded activations we provide the lower and upper bounds. PReLU provides granularity level at the number of parameters per activation layer. A dash ($-$), in the Optimal HP column, indicates that such activation function uses the unique or baseline parameter (e.g., ReLU only has slope $\alpha=0$).**
| Activation | Mean $\pm$ 95% CI<br>Plasticity Score | LR | Optimal HP |
| --- | --- | --- | --- |
| ReLU | 168.59  $\pm$ 222.92 | 0.0001 | $-$ |
| Leaky-ReLU | -127.40  $\pm$ 73.09 | 1e-05 | 0.8 |
| Sigmoid | 993.61  $\pm$ 190.38 | 0.0001 | $-$ |
| Tanh | 337.61  $\pm$ 20.01 | 1e-05 | $-$ |
| RReLU | 435.00  $\pm$ 434.30 | 0.0001 | Bounds: $[0.125,0.333]$ |
| PReLU | 684.88  $\pm$ 115.90 | 0.0001 | Layer. $\alpha$=0.65 |
| Swish (SiLU) | 1120.28  $\pm$ 229.22 | 0.0001 | 0.01 |
| GeLU | 331.30  $\pm$ 126.07 | 0.0001 | 1.0 |
| eLU | -7.78  $\pm$ 194.69 | 1e-05 | 1.0 |
| CeLU | 115.78  $\pm$ 173.85 | 1e-05 | 1.0 |
| SeLU | 306.16  $\pm$ 4.27 | 1e-05 | 1.673 |
| CReLU | -69.09  $\pm$ 41.11 | 1e-05 | $-$ |
| Rational | 603.39  $\pm$ 274.26 | 1e-05 | C, 5, 4, Leaky-ReLU |
| Smooth-Leaky | 1239.80  $\pm$ 212.96 | 0.0001 | C:0.5, P:2.0, $\alpha$=0.1 |
| Rand. Smooth-Leaky | 1310.94  $\pm$ 98.37 | 0.0001 | C:0.1, P:1.0. Bounds: $[0.01,0.1]$ |

Previously, we defined $\mathrm{GAP}_{c,e}=R^{\text{train}}_{c,e}-R^{\text{test}}_{c,e}$, with $R$ the expected return measured at the end of cycle $c$. For each activation and environment, we summarize change across cycles as
$\Delta(\mathrm{GAP}_{e})=\mathrm{GAP}_{3,e}-\mathrm{GAP}_{1,e}$ (abbreviated $\Delta$).
Thus, $\Delta<0$ means the train–test gap *shrinks* across cycles (better transfer), whereas $\Delta>0$ means it *widens* (worse transfer).

We summarize across environments using both the *median* (robust) and the *mean* (sensitive to outliers). This distinction matters: for instance, GeLU exhibits an extremely large *negative* mean $\Delta$ driven by a single Humanoid-v5 outlier, whereas its median $\Delta$ is modest and *positive* (i.e., a typical slight widening), indicating that the outlier dominates the mean while the median reflects the typical behavior.

Separately, we report a Plasticity Score that captures late-cycle *functional* performance on the training environments. We align runs to a common timeline, average rewards, and—within the last cycle—take the median of the mean returns over the final $15\%$ of each cycle. This metric answers: “can the agent still perform well after repeated shifts on the data it now collects?”—it is not a train–test gap.

The highest Plasticity Scores are obtained by *Rand. Smooth-Leaky* (1310.94), *Smooth-Leaky* (1239.80), *Swish* (1120.28), *Sigmoid* (993.61), and *PReLU* (684.88) (see Tab. [F1](https://arxiv.org/html/2509.22562v1#A6.T1) for full hyperparameter disclosure, we do not show Smooth-Leaky in favor of just showing the best of our custom activations). These activations reliably recover strong late-cycle returns across tasks. However, several also show *positive* $\Delta$ (e.g., Swish, Sigmoid, PReLU, Smooth-Leaky, Rand. Smooth-Leaky), indicating that while train performance rose, transfer to the perturbed tests did not keep pace (the gap widened).
Conversely, *Tanh* and *SeLU* achieve *negative* median $\Delta$ (gaps narrow on typical environments) but only moderate Plasticity Scores ($\approx 337$ and $\approx 306$).
Among ELU-family variants, *eLU* yields a small improvement in $\Delta$ with low train-side performance ($\approx 17$), whereas *CeLU* shows a larger gap reduction while showing modest competitiveness on Plasticity Score ($\approx 498.06$).
*ReLU* and *Leaky-ReLU* produce low Plasticity Scores ($\approx 168.59$ / $\approx-80$), consistent with instability under repeated shifts, despite mixed signs in $\Delta$.
Overall, a high Plasticity Score *does not imply* improved transfer: the score reflects adaptation on the training distribution, while $\Delta(\mathrm{GAP}_{\text{end}})$ reveals whether that adaptation *carries* to perturbed tests.

We therefore present both viewpoints: (i) Plasticity Score for functional performance under non-stationarity, and (ii) Generalization Gap for evaluation of the adaptation carried to perturbed tests. We use the *median* $\Delta$ as the primary cross-environment summary and the *mean* $\Delta$ only to highlight outlier-driven effects. Full per-activation, per-environment results are in Tab. [F2](https://arxiv.org/html/2509.22562v1#A6.T2), and the cycle-by-cycle evolution is visualized in Fig. [F1](https://arxiv.org/html/2509.22562v1#A6.F1).

**Table F2: First-to-last cycle $\Delta$ of the Generalization Gap per environment and activation function (rounded to 2 decimals). Lower is better (more transferability / higher plasticity). Best (lowest) per column is bolded.**
| Activation | HalfCheetah-v5 | Humanoid-v5 | Ant-v5 | Hopper-v5 | mean $\Delta$ | median $\Delta$ |
| --- | --- | --- | --- | --- | --- | --- |
| ReLU | 124.81 | -1249177.00 | 183.00 | -287.15 | -312289.00 | -81.17 |
| Leaky-ReLU | 65.64 | -13959.34 | -56.82 | -0.48 | -3487.75 | -28.65 |
| Sigmoid | 1521.45 | 18.92 | 276.48 | 152.14 | 492.25 | 214.31 |
| Tanh | -782.69 | 48.28 | -388.85 | 91.95 | -257.83 | -170.29 |
| RReLU | 527.09 | -19274.62 | -26.83 | 21.89 | -4688.12 | -2.47 |
| PReLU | 839.60 | 128132.00 | 94.17 | -22.35 | 32260.85 | 466.88 |
| Swish (SiLU) | 627.24 | -127118.70 | 1035.22 | 20.88 | -31358.83 | 324.06 |
| GeLU | 317.71 | -9766168.00 | 258.21 | -32.29 | -2441406.00 | 112.96 |
| eLU | 103.83 | -80.15 | -207.62 | 14.92 | -42.25 | -32.61 |
| CeLU | -341.13 | -49.31 | -281.51 | 3.56 | -167.10 | -165.41 |
| SeLU | -839.65 | 15.91 | -339.88 | 51.90 | -277.93 | -161.99 |
| CReLU | 367.50 | -65.10 | -5.26 | 1.65 | 74.70 | -1.81 |
| Rational | 550.81 | 9852.05 | 270.39 | 54.72 | 2681.99 | 410.60 |
| Smooth-Leaky | 621.48 | -666714.76 | -210.37 | -8.04 | -166577.9 | -109.20 |
| Rand. Smooth-Leaky | 103.80 | -1236645.00 | 563.09 | 19.22 | -308989.80 | 61.51 |

### F.1 Interpreting Negative Generalization Gaps in Continual RL

A strongly negative generalization gap at the end of training ($\mathrm{GAP}_{c,e}<0$) is not paradoxical under non-stationary streams (e.g., randomized MuJoCo friction Abbas et al. (2023)). As the training environment keeps shifting, the agent can lose *plasticity*—it struggles to re-fit the *current* regime—so $R^{\text{train}}_{c,e}$ is depressed. Yet the policy may retain robust, regime-invariant skills that carry to perturbed test conditions, where evaluation is noise-free and does not suffer on-policy update instability. Consequently $R^{\text{test}}_{c,e}$ can exceed $R^{\text{train}}_{c,e}$, yielding a negative $\mathrm{GAP}_{\text{c,e}}$. Our summary $\Delta(\mathrm{GAP}_{e})=\mathrm{GAP}_{3,e}-\mathrm{GAP}_{1,e}$ becomes highly negative when transferability improves over the training cycle even as within-cycle adaptation degrades—an acceptable and informative outcome in this setting.

Figure: Figure F1: The heatmap reports end-of-cycle $\mathrm{GAP}_{c,e}$ per activation (rows) and cycle (columns). Colors are centered at 0 (green = negative values, test $>$ train; red = positive values, train $>$ test). Values are computed on a held-out friction variant of the training environment (Abbas et al., 2023). See Tab. [F2](https://arxiv.org/html/2509.22562v1#A6.T2) for the across-cycle summary $\Delta(\mathrm{GAP}_{e})$. Together, these views reveal when apparent trainability gains translate (or fail to translate) into generalization.
Refer to caption: gap_sequence.png

## Appendix G Novel Activation Function Formulations

Our characterization study enables the principled design of many novel activation functions; representative examples include:

**Table G1: Binary property grid (✓= present, – = absent). Abbreviations. HDZ: hard dead zone; NZG: non‑zero gradient for $x<0$; Sat$\pm$: two‑sided saturation; Sat$-$: negative‑side saturation; $C^{1}$: first derivative continuous; NonM: non‑monotonic segment; SelfN: self‑normalizing output; L/R${}_{\text{slp}}$: learnable or randomized slope; $f^{\prime\prime}$: non‑zero second derivative. ^△ Proposed in this work.**
| Activation | HDZ | NZG | Sat$\pm$ | Sat$-$ | $C^{1}$ | NonM | SelfN | $L/R_{slp}$ | $f^{\prime\prime}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RSeLU^△ | – | ✓ | – | ✓ | – | – | ✓ | ✓ | ✓ |
| Bo-PReLU^△ | – | ✓ | – | – | – | – | – | ✓ | – |

### G.1 Bounded PReLU (Bo-PReLU)

The Bounded Parametric Rectified Linear Unit (Bo-PReLU) is designed to combine the adaptability of PReLU with enhanced stability by constraining its learnable negative slope. Our case studies found that extreme slope values can be detrimental to performance. Bo-PReLU addresses this by forcing the slope to remain within a predefined "Goldilocks" range, preventing it from becoming excessively large or small.

Figure: Figure G1: Bo-PReLU where $\alpha_{min}=0.6$, $\alpha_{max}=0.8$ and $\alpha_{init}=0.65$.
Refer to caption: boprelu_sample.png

The function follows the standard PReLU formulation:

$$ $f(x)=\begin{cases}x&\text{if }x\geq 0\\ \alpha x&\text{if }x<0\end{cases}$ (7) $$

The key innovation lies in how $\alpha$ is learned. It is constrained to the range $[\alpha_{\min},\alpha_{\max}]$. To ensure this constraint is met without interfering with gradient-based optimization, we employ the reparameterization trick. An unconstrained parameter, $\alpha_{\text{raw}}$, is learned, and the final slope is derived during the forward pass as:

$$ $\alpha=\alpha_{\min}+(\alpha_{\max}-\alpha_{\min})\cdot\sigma(\alpha_{\text{raw}})$ (8) $$

where $\sigma$ is the sigmoid function. This makes Bo-PReLU a robust and stable learnable rectifier.

### G.2 Randomized-Slope SELU (RSELU)

The Randomized-Slope Scaled Exponential Linear Unit (RSELU) is a hybrid activation function designed to merge the stochastic regularization benefits of RReLU with the training stability of SELU’s self-normalization property.

The function has two modes of operation. During training, it introduces randomization to the negative slope:

$$ $f(x)=\begin{cases}\lambda x&\text{if }x\geq 0\\ \lambda r(\exp(x)-1)&\text{if }x<0\end{cases}\quad\text{where }r\sim\mathcal{U}(l,u)$ (9) $$

During inference, the randomization is removed to ensure deterministic output, and the random variable $r$ is fixed to the mean of its distribution, $(l+u)/2$.

Figure: Figure G2: R-SeLU with bounds $r\sim\mathcal{U}(l,u)$ where $r=(0.9232,2.4232)$.
Refer to caption: rselu_sample.png

A crucial feature of this design is that the bounds $l$ and $u$ are chosen to be symmetric around the original SELU alpha parameter ($\approx 1.6732$). This ensures that the self-normalizing property of SELU is preserved "in expectation" throughout training, providing a stable foundation while the slope randomization encourages robust learning and plasticity.

## Appendix H The Use of Large Language Models (LLMs)

We used an LLM as a tool for early brainstorming, code debugging, and writing/editing of the earlier drafts of this paper. During ideation, we used it to enumerate experiment variants and sanity-check design choices; for code, we requested bug-finding hints and refactoring suggestions that we implemented only after manual review and testing; for text, we used it to improve clarity, organization, and grammar. The LLM did not generate novel research ideas, experiments, or results on our behalf; all methodological innovations, analyses, and conclusions are our own. We verified any technical claims, equations, and citations suggested during assisted drafting and we did not include uncited, model-generated text verbatim. No proprietary or personally identifiable data was provided to the LLM. The authors retain full responsibility for the content of this paper, and we affirm the novelty and originality of the work.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 1
questions: - Could you further clarify the reasoning behind the criteria for dead units defined for non-ReLU activation functions? For ReLU it makes sense because there is a region where the value of the activation function is 0 with slope 0. That doesn’t seem to be the case with most of the other activation functions studied, and it seems a bit odd to talk about dead-unit fractions.
soundness: 2
strengths: - The initial part of the paper with the case studies on the negative slope and dead band width are valuable experiments for the community.
- The diversity of metrics and analyses presented for the experiments is quite high.
summary: This paper looks at the effect of various characteristics of activation functions on the plasticity of neural networks on supervised continual learning and RL benchmarks. These characteristics include things like the slope of the activation function for negative inputs, where they identify a “Goldilocks zone,” and dead-band width which is the portion of the activation function domain that has a small gradient. They propose two different activation functions, Smooth-Leaky and Randomized Smooth-Leaky which are modifications of the Leaky-ReLU activation function with a smooth transition region rather than a kinked one.
weaknesses: - The presentation of the paper can use some work. Most of the results for the experiments in Sections 3 and 4 are not presented in the main paper, and are instead presented in an Appendix. This is fine for one or two auxiliary results, but here, most of the main results for a significant portion of the paper are presented elsewhere. I’d ask that either the authors figure out a way to present a coherent story with results in the main paper, either by moving some parts of the paper to the appendix (not just the graphs), or if that is not possible in the page limit, consider another venue.
- The paper introduces two activation functions Smooth-Leaky and Randomized Smooth-Leaky. These activation functions introduce 3 and 4 new hyperparameters respectively which are searched for in a grid, with 175 and 450 combinations respectively. Compare this to the baseline of Leaky-ReLU which had 12 configurations evaluated. Furthermore, these hyperparameters were individually tuned on each setting. The amount of extra tuning given to the new methods is an unfair advantage, and could definitely be the cause of any improvement in performance rather than something inherent about the methods themselves.
- The metrics used to measure plasticity in the RL section seem to be flawed. If we look at just the rewards, Sigmoid seems to significantly outperform the other activation functions on 2/4 environments. Yet the plasticity score seems designed in a way to remove that effect, taking the median across environments to result in a number where the proposed method does better. A better approach could be to do some kind of normalization of the return and presenting the mean. Taking the median across 4 environments specifically excludes the environment where none of the activation functions except for Sigmoid are able to learn.
- For generalizability, there’s another metric presented which is a non-standard metric. It looks at the gap between train performance and a test environment which is some perturbation of the train environments. The metric reported compares the difference between the gap after cycle 3 and cycle1. First, it’s unclear what the perturbations are. Second, this metric has no notion of what the actual reward obtained in each of the settings was. A small gap with low absolute rewards is less preferable to a large gap with high absolute rewards, but this metric doesn’t capture that. Finally, measuring generalizability is much easier in the supervised setting, and has been done in several prior works. The results on those settings should be reported before creating such a metric.
- Tables 1, 2, and 4 are missing standard deviations.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Given the recent popularity of the SwiGLU, why isn't this activation function evaluated?
soundness: 3
strengths: - The empirical results clearly show that the proposed activation function randomized smooth-leaky consistently attains superior average online task accuracy. 
- The experiments consider a comprehensive set of baseline or existing activation functions.
- The discussion and experiments regarding the generalization gap, specifically, that plasticity loss can manifest in reduced training loss and test loss in different ways, is a positive touch to the paper which is not usually evaluated in this space. As the authors point out, maximizing plasticity via maximizing training loss may result in over-fitting. While earlier papers have argued that solely analyzing test loss in continual settings can be confounded by issues of regularization and generalization. The focus that the authors put here is a strength of this paper.
summary: The authors explore the role of activation function design on plasticity in continual learning. The authors introduce the Smooth-Leaky and the Randomized Smooth-Leaky activation functions.The authors evaluate their proposed activation functions in the supervised class incremental continual learning benchmarks and in RL environments and show that the randomized smooth leaky activation function consistently outperforms existing activation functions.
weaknesses: - There is a paper by Lewandowski et al. "Plastic Learning with Deep Fourier Features" which also examines the role of activation function design on plasticity. It would be useful if the authors could comment on this line of work, and ideally, incorporate this paper's contribution as a baseline competitor. 
- It would be worthwhile to apply a few existing SOTA methods for plasticity loss such as reset-based: SNR and regularization-based: L2 Init, and to generate a table with results for a reasonable set of METHOD X + ACTIVATION FUNCTION Y. When given an additional continual learning intervention how important is the choice of activation function?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to the comments in the weaknesses section.
soundness: 3
strengths: - The main strength of the paper lies in its rigorous evaluation of activation functions across continual learning benchmarks (supervised and RL).
- Proposed activation functions, and the concept of a "Goldilocks zone" for negative-side responsiveness, offer a interesting perspective on mitigating plasticity loss in continual learning. ​The findings highlight the role of activation functions in maintaining plasticity.
- Metrics like saturation fraction and dead-band width provide valuable insights into activation function behavior under non-stationary settings.
- The paper is well-structured and clearly written, with visualizations, explanations.
summary: The authors provide a thorough investigation on the role of activation functions in addressing plasticity loss in continual learning. They propose two activation functions, Smooth-Leaky and Randomized Smooth-Leaky, which involve maintaining a non-zero derivative floor, optimizing negative-side responsiveness and ensuring smooth transitions at the origin. They evaluate these activations across supervised continual learning benchmarks and RL environments, showing their effectiveness in maintaining plasticity.
weaknesses: - While the paper discussed impact unconstrained adaptive slopes (e.g., PReLU), it does not explore alternative methods to guide parameters into the "Goldilocks zone" which itself is primarily identified empirically, but lacks theoretical grounding or formal analysis. Therefore, explaining why this range is optimal would help the paper overall.
- The paper does not explore how activation functions interact with different optimizers or learning rate schedules. For example, adaptive optimizers like Adam or RMSProp may behave differently with the proposed activations. Are the proposed activations universally effective across all continual learning settings? Since the dynamics of plasticity loss may vary significantly between tasks, a more nuanced analysis of task-specific performance could provide deeper insights.
- The reinforcement learning experiments focus solely on MuJoCo tasks, Have authors tried running experiments on domains like sparse reward or high-dimensional tasks?

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 1
questions: What distribution over pre-activations is used for the effective negative slope in Figure C-1(per-layer? per-epoch? across tasks?), and how sensitive are the conclusions to that choice? Please report variability across layers/epochs and any normalization applied before computing.
soundness: 2
strengths: The paper provides a substantial empirical study on the properties of many activation functions under the lens of continual learning. The case studies are well thought out and well conducted, they isolate negative-branch slope, saturation, and “dead-band width” as key predictors under shift. The study of desaturation dynamics under shocks is an important diagnostic of activation functions and ties well activation shape to recovery behavior after controlled shifts.
The suggested drop-in replacements are intuitive and easy to implement.
summary: The paper argues that activation choice is a primary lever for sustaining plasticity in continual learning. It presents a property-level analysis (negative-branch responsiveness, saturation, “dead-band” width) of activation functions and proposes two drop-in $C^1$ activations, Smooth-Leaky and Randomized Smooth-Leaky, that maintain a non-zero derivative floor. The paper evaluates the activation functions on standard Continual Learning tasks.
weaknesses: In general, the paper is verbose and somewhat difficult to parse. Table 2 reports statistical significance but does not provide confidence intervals or which statistical tests were used. There also appears to be an imbalance in the hyper-parameter sweeps: the two proposed activation functions are explored across 175 and 450 combinations, respectively, whereas other hyper-parameters are examined over only a handful of settings. This imbalance could bias the experimental results.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - Do the proposed activation functions result in greater wall-clock time compared to existing activation functions?
- Could the authors provide the results of the **Smooth-Leaky** activation function in the continual reinforcement learning experiments?
soundness: 3
strengths: - The case studies are interesting and thoughtfully executed, providing solid and well-supported conclusions. 
- The large-scale empirical studies on continual supervised learning and continual reinforcement learning clearly demonstrate the superiority of the proposed activation function over prior alternatives.
- The proposed activation functions show promise for broader application in domains that rely on network plasticity.
- The authors also provide code to improve reproducibility.
summary: This work presents two case studies to investigate the activation functions in the context of continual learning and identifies three principles for designing plasticity-friendly nonlinearities. Building upon these insights, the authors propose two new activation functions: **Smooth-Leaky** and **Randomized Smooth-Leaky**. Empirical studies on continual supervised learning and continual reinforcement learning demonstrate that the proposed activation functions achieve superior performance compared to existing alternatives.
weaknesses: - As the experiments already include multiple runs, I recommend that the authors report not only the mean results but also measures of statistical significance—such as standard deviation or 95% confidence intervals—for example, in Table 1 and Table 2.
- The continual reinforcement learning evaluation appears limited to a single experimental configuration. Expanding this analysis to include additional environments or settings would provide stronger support for the generality of the proposed activation functions.
- *Case Study 1* and *Case Study 2* are important parts of the paper, but too many related results and figures are placed in the appendix. To enhance readability, the authors might consider reorganizing the content despite the page limit.
- The term $C^1$ transition first appears in the introduction (line 60), but without explanation.


Decision: Accept (Poster)