Adaptive Rational Activations to Boost Deep Reinforcement Learning
Authors: Quentin Delfosse
Patrick Schramowski
Martin Mundt
Alejandro Molina
Kristian Kersting
Venue: ICLR 2024 spotlight (2024)
Keywords: Deep Reinforcement Learning
Neural Plasticity
Activation Functions
Rational Functions
Primary area: reinforcement learning
TL;DR: We exhibit the importance of plasticity for reinforcement learning, and propose to use learnable rational activation functions to augment agents' plasticity.

Abstract: Latest insights from biology show that intelligence not only emerges from the connections between neurons, but that individual neurons shoulder more computational responsibility than previously anticipated. Specifically, neural plasticity should be critical in the context of constantly changing reinforcement learning (RL) environments, yet current approaches still primarily employ static activation functions. In this work, we motivate the use of adaptable activation functions in RL and show that rational activation functions are particularly suitable for augmenting plasticity. Inspired by residual networks, we derive a condition under which rational units are closed under residual connections and formulate a naturally regularised version. The proposed joint-rational activation allows for desirable degrees of flexibility, yet regularises plasticity to an extent that avoids overfitting by leveraging a mutual set of activation function parameters across layers. We demonstrate that equipping popular algorithms with (joint) rational activations leads to consistent improvements on different games from the Atari Learning Environment benchmark, notably making DQN competitive to DDQN and Rainbow.

Published as a conference paper at ICLR 2024
ADAPTIVE RATIONAL ACTIVATIONS TO BOOST DEEP REINFORCEMENT LEARNING
Quentin D Patrick Schramowski 123, Martin Mundt
Alejandro Molina' & Kristian Kersting!... 1,2,,44
Computer Science Dept., TU Darmstadt
German Center for Artificial Intelligence Centre for Cognitive Science, Darmstadt
3 Hessian Center for Artifical Intelligence
ABSTRACT
Latest insights from biology show that intelligence not only emerges from the connections between neurons, but that individual neurons shoulder more compu- tational responsibility than previously anticipated. Specifically, neural plasticity should be critical in the context of constantly changing reinforcement learning (RL) environments, yet current approaches still primarily employ static activation functions. In this work, we motivate the use of adaptable activation functions in RL and show that rational activation functions are particularly suitable for augmenting plasticity. Inspired by residual networks, we derive a condition under which rational units are closed under residual connections and formulate naturally regularised version. The proposed joint-rational activation allows for desirable degrees of flexi- bility, yet regularises plasticity to an extent that avoids overfiuting by leveraging mutual set of activation function parameters across layers. We demonstrate that equipping popular algorithms with (joint) rational activations leads to consistent improvements on different games from the Atari Learning Environment benchmark, notably making DQN competitive to DDQN and Rainbow.'
INTRODUCTION
Neural Networks' efficiency in approximating any function has made them the default choice in many machine learning tasks. This is no different in deep reinforcement learning (RL., where the DQN algorithm's 's introduction (Mnih et al., 2015) has sparked the development of various neural solutions. In concurrence with former neuroscientific explanations of brainpower residing in combinations stemming from trillions of connections (Garlick, 2002), present advances have emphasised the role of the neural architecture (Liu et al., 2018; Xie et al., 2019). As such, RL improvements have first been mainly obtained through enhancing algorithms (Mnih et al., 2016; Haarnoja et al., 2018; Banerjee et al., 2021) 1) and only recently by searching for performing architectural patterns, via automatic deep policy search (Pang et al., 2021; Krishnan et al., 2023), or via decoupling object detection (Lin et al., 2020: Delfosse et al., 2023b) and policy search (Delfosse et al., 2023a; Wu et al., 2024).
However, research has also progressively shown that individual neurons shoulder more complexity than initially expected, with the latest results demonstrating that dendritic compartments can compute complex functions (e.g. XOR) (Gidon et al., 2020), previously categorised as unsolvable by single- neuron systems. This finding seems have renewed interest in activation functions (Georgescu et al., 2020; Misra, 2020). In fact, many functions have been adopted across different domains (Redmon et al., 2016; Brown et al., 2020; Schulman et al., 2017). To reduce the bias introduced by fixed activation function and achieve higher expressive power, one can further learn which activation function is performant for particular task (Zoph & Le, 2017: Liu et al., 2018), learn to combine arbitrary families of activation functions (Manessi & Rozza, 2018), or find coefficients for polynomial activations as weights to be optimised (Goyal et al., 2019).
Whereas these prior approaches have all contributed to their respective investigated scenarios, there exists a finer approach that elegantly encapsulates the challenges brought on by reinforcement
Rational library: github.comm//ntt/actiittinoo -tt.s///iiiiiiiiioonnuuntccsssss Experiments: hataubccommmmmreeeee/aaaaaa rl.

---

Published as a conference paper at ICLR 2024
Environment (Time Pilot]
Shared activation functions
Layer 3
Layer 4
epoch 0
epoch 50
epoch 200
epoch 500
Figure 1: Neural plasticity due to trainable activation functions allows RI agents to adapt to environments of increasing complexity. Rational activations (bottom), with shared parameters in each of the last two layers, evolve together with their input distributions (shaded blue) when learning with DQN on Time Pilot. Each column corresponds to training state where new, more challenging part of the environment (top, c.g. increasing enemy speed and movement complexity) has been uncovered and is additionally used for training.
learning problems. Specifically, at each layer, we can learn rational activation functions (ratio of polynomials) (Molina et al., 2020). Not only can rationals converge to any continuous function, but they have further been proven to be better approximants than polynomials in terms of convergence (Telgarsky, 2017). Even more crucially, their ability to adapt while learning equips a model with high neural plasticity ( ("capability to adjust to the environment and its transformations" (Garlick, 2002)). We argue that adapting to environmental changes essential, making rational activation functions particularly suitable for dynamic RL environments. To provide visual intuition, showease exemplary evolution of two rational activation functions together with their respective changing input distributions the dynamic "Time Pilot" environment in Fig. 1.
In this work, we show that plasticity is of major importance for RI agents, as a central element to satisfy the requirements originating from diverse and dynamic environments and propose the of rational activation functions to augment deep RL agents plasticity. Apart from demonstrating the suitability adaptive activation functions for Deep RL, we also evaluate how many additional layer weights can be replaced by rational activations. Our specific contributions are:
(i) We motivate why neural plasticity is key aspect for Deep RL agents and that rational activations are adequate as adaptable activation functions. For this purpose, we not only highlight that rational activation functions adapt their parameters over time, but further prove that they can dynamically embed residual connections, which we refer to as residual plasticity.
additional representational capacity can hinder generalisation, especially RL (Farebrother et al., 2018; Roy et al., 2020; Yarats et al., 2021), propose joint-rational variant, that uses weight-sharing rational activations across different layers.
(iii) We empirically demonstrate that rational activations bring significant improvements to DQN and Rainbow algorithms on Atari games and that our joint variant further increases performance.
(iv) Finally, investigate the overestimation phenomenon of predicting too large return values, which has previously been argued to originate from an unsuitable representational capacity of the learning architecture (van Hasselt al., 2016). As result of our introduced (rational) neural and residual plasticity, such overestimation can practically be reduced.
We proceed as follows. We start off by arguing in favour of plasticity for deep RL, then show how rational functions are particularly suitable candidates to provide plasticity neural networks and present empirical evaluation. Before concluding, touch upon related work.
2

---

Published as a conference paper at ICLR 2024
Human normalized scare [%)
Fig
Conaltantal ReLU
Nvtrational Plaiidly (PEJU)
Revional Pastic
Tomponal Rational Plaikity
Rapalebbl Ratioal Pladiily
Statienary Env.
Dynamic Env.
Progressive Env.
10?
10
Kangaroo
Space
Tennis
Asterix
Enduro
Jamesbond
Seaquest
Time Pilot
Figure 2: Neural plasticity is essential for reinforcement learning. Human normalised mean scores for rigid (LReLU and CRELU) DQN agents, agents with non-rational, rational, tempered, and regu- larised plasticity are shown with standard deviation across 5 random seeded experimental repetitions. Larger scores are better. Tempered plasticity, allowing initial adaptation to the environments, but not their transformations in experimental repetitions, performs better on stationary environments. Regu- larised plasticity performs well across all environment types. Best viewed in colour. A description of the environments' types is provided in Appendix A.5,
RATIONAL PLASTICITY FOR DEEP RL
Let us start by arguing why deep reinforcement learning agents require extensive plasticity and show that parametric rational activation functions provide appropriate means to augment plasticity.
As motivated in the introduction, RL is subject to inherent distribution shifts. During training, agents progressively uncover new states (input drift) and, as the policy improves, the reward signal is modified (output drift). More precisely, for input drifts, can distinguish environments according to how much they change through learning. For simplicity, we categorise according to three intuitive stationary, dynamic and progressive environments. Consider the example of Atari 2600 games. Kangaroo and Tennis can be characterised as stationary since the games' distributions do not change significantly through learning. Asterix, Enduro and Qere are dynamic environments, as different distribution shifts (e.g. Cauldron, Helmet, Shield, etc. in Asterix) are provided to the agents in the first epochs, with no policy improvement required to uncover them. On the contrary, Jamesbond, Seaquest, and Time Pilot are progressive environments: agents need to master early stages before being provided with additional states, i.e. exposed to significant input shifts.
categories:
How do efficiently improve RL agents' ability to adapt to environments and their changes? To deal with distribution shifts, our agents require high neural plasticity and thus benefit from adaptive architectures. To elaborate further in our work, let consider the popular DQN algorithm (Mnih et al., 2015), that employs a o-parameterised neural network to approximate the Q-value function of state S, and action a. This network is updated following the Q-learning equation: Q (S,, a: 0) = R1+1 + 7 max, Q (St+1, a; 0). In addition to network connectivity playing an important role, we now highlight the importance of individual neurons by modifying the network architecture of the algorithm via the use of learnable activation functions, to show that they are presently underestimated component. To emphasise the utility of the upcoming proposed rational and joint-rations activation functions, we will interleave early results into this section. The latter serves the primary purpose to not only motivate the suitability of the rational parameterisation to provide plasticity, but also discern the individual benefits of (joint-) rational activations, in the spirit of ablation studies.
2. RATIONAL NEURAL PLASTICITY
Q(x) = En azad
P(x)
Rational functions are ratio of polynomials, defined on I by R(x)
1 D byck
where æ E R. {az} and {bx} } are m++1 and n (real) Iearnable parameters per layer. To test rational functions' plasticity, we use the discrete distribution shifts of the Permutted-MNIST continual learning experiment. We show in Appendix 4 that rational activation functions improve the plasticity over both ReLU and over CReLU (Shang et al., 2016), used to augment plasticity by Abbas et al. (2023). For RL we show in Fig. that the rational parametrisations substantially enhances RL agents. More precisely, by comparing agents with rigid networks (a fixed Leaky ReLU baseline) to agents with rational plasticity (f.e. with rational activation function at each layer), we that rational functions boosts the agents to super-human performances on out of 9 games. The acquired extra neural plasticity seems to play significant role in these Atari environments, especially in progressive ones.
3

---

Published as a conference paper at ICLR 2024
In order to discern the benefits of general plasticity through any adaptive activation function, over the proposed use of the rational parametrisation, Fig. 2 additionally includes agents with Concatenated ReLU and with Parametrised Exponential Linear Unit (Trottier et al., 2017). CReLU, was used in RL to address plasicity loss (Abbas et al., 2023), outperforms LRELU on 6 out of 9 games, but is always outmatched by rational plasticities. PELU that uses 3 parameters to control its slope, saturation and exponential decay, and has been shown to outperform other learnable alternatives on classification tasks (Godfrey, 2019). However, in contrast to the rational parameterisation, it seems fall behind and only boosts the agents to super-human performance on 3 out of 9 games (contrary to 7), implying that the type of plasticity provided by rational activations is particularly suitable.
To highlight the desirability of rational activations even further, we additionally distinguish between the plasticity of agents towards their specific environment and the plasticity allowing them to adapt while the environment is changing. To this end, we show agents equipped with rational activations that are tempered in Fig. 2. Such agents are equipped with the final, optimised rational functions agents. They correspond to frozen functions from agents that already
of trained rational-equipee adapted to their specific environment. The plasticity of the rationals is thus tempered (i.e. stopped) in repeated application (i.e. another training session) to emphasise the necessity to continuously adapt the activation functions, together with the layers' weights during training. Whereas providing agents with such tempered, tailored to the task, activations already boosts performances, rational plasticity at all times is essential, particularly in dynamic and progressive environments.
2.2 RATIONAL RESIDUAL PLASTICITY
The prior paragraphs have showcased the advantage of agents equipped with rational activation functions, rooted in their ability to update their parameters over time. However, we argue that the observed boost in performance is not only due to parameters adapting to distributional drifts. Rational activations can embed one of the most popular techniques to stabilise deep networks training; namely, they can dynamically make use of a residual connection. We refer to this as residual plasticity.
Rationals closed under residual connection and provide residual plasticity. We here show that rational functions with strictly higher degree in the numerator embed residual connections. Recall that residual neural networks (ResNets) were initially introduced following the intuition that it is easier to optimise the residual mapping than to optimise the original, unreferenced mapping (He et al., 2016). Formally, residual blocks of ResNets propagate an input X through two paths: a transforming block of layers that preserves the dimensionality (F) and a residual connection (identity).
Theorem: Let R be rational function of order (m, n). R embeds a residual connection <> m m>> n.
Proof: Let us consider a rational function R = P/Q of order (m, n), with coefficients Alm) (az) E Rm++1 of P and Bin] = (b:) E R of Q (with bo = 1). We denote by & (respectively be tensor corresponding to
②) the Hadamard product (respectively division). Let X E I
the input of the rational function of an arbitrary layer in a given neural network. We derive x
8 i-1 X. Furthermore, we use GV( = [1, x ,..,, X E R(m x..*xx,)x*++++ to denote the tensor containing the powers up to l of the tensor X. Note that GV[K) can be understood as a generalised Vandermonde tensor, similar as introduced in (Xu et al., 2016). For Vlb) = (w)) E RR 6+1
let GVIM) -iM== " X be the weighted sum over the tensor elements of the last dimension.
Now, we apply the rational activation function R with residual connection to X:
y(X)= R(X) + X GV/ml (x).Alm) (m)
GVL*) (m) (X). + X
= (GVIml (X) [m] + X x GV (X) B')) o Gylm Blan
= (GVVml(X (X)..1 [mm) + GV
By
2 GVin (VVN)(X))BBl BI).
= -VVlmax(m..++ 1)) ())))))lmmxxmmm)m (mlaxxmmmmm)) 2 GVlll(X))
= =R(X),
where B BB (bo. E 2+2 (with by,o = 0 and by. = b for E (1, ..., n 13).
clnax(m,a+1) ma)) (cj oj + bj- a, -0 00j (0. .... b, Vij (0.....) n.).
is a rational function of order (m', n²). with m' > n'. In other words, rational activation functions of order m > embed residual connections. Using the same degrees for numerator and denominator
4

---

Published as a conference paper at ICLR 2024
certifies asymptotic stability, but our derived configuration allows rationals to implicitly use residual connections. Importantly, note that these residual connections are not rigid, as these functions can progressively learn a =0 for all j >n, n, i.e. we have residual plasticity.
Rational plasticity to replace residual blocks' plasticity. In very deep ResNets, it has been observed that feature re-combinations does not occur inside the blocks but that transitions to repre- sentations occur during dimensionality changes Veit et al. (2016); Greff et al. (2017).
To investigate this hypothesis, Veit al. have conducted lesioning experiments, where residual block is removed from the network, and surrounding ones are fine-tuned to recover. Whereas we emphasize that we do not claim that the residual in rationals can replace entire convolutional blocks or that they are generally equivalent, we hypothesize that under the conditions investigated by Veit et al. of very deep networks, residual blocks could learn complex activation function-like behaviours. To test this conjecture, we repeat the lesioning experiments, but also test replacing the lesioned block with rational function that satisfies the residual connection condition derived above. Results are provided in appendix (ef. A.2) and show that rational functions' plasticity can efficiently compensate for the lost capacity of lesioned residual blocks in very deep residual networks.
2.3 NATURAL RATIONAL REGULARISATION
We have motivated and shown that the combination of neural and residual plasticity form the central pillars for why rational activation functions are desirable in deep RL. In particular for dynamic and progressive environments, rational plasticity has been observed to provide substantial boost over alternatives. However, if we circle back to Fig. 2 and take more careful look at the stationary environments, we can observe that our previously investigated tempered rational plasticity (for emphasis, initially allowed to tailor to the task but later "stopped" in experimental repetition) can also have an upper edge over full plasticity. The extra rational plasticity at all times might reduce generalisation abilities, particularly non-diverse stationary environments. In fact, prior works have highlighted the necessity for regularisation methods in deep reinforcement learning (Farebrother al., 2018; Roy et al., 2020; Yarats al., 2021).
We thus propose a naturally regularised rational activation version, inspired from residual blocks. In particular, Greff al. have indicated that sharing the weights can improve learning performances, as shown in Highway (Lu & Renals, 2016) and Residual Networks (Liao & Poggio, 2016). In the spirit of these findings, we propose the regularised coint-rationals. where we constrain the input to propagate through different layers but always be activated by the same learnable rational activation function. Rational functions thus share a mutual set of parameters across the network, (instead of layers, cf Fig. 1 1). As observable in Fig. 2, this regularised form of plasticity increases the agents' scores the stationary environments and does not deteriorate performances in the progressive ones.
3 EMPIRICAL EVIDENCE FOR PLASTICITY
Our intention here investigate the benefits of neural plasticity through rational networks for deep reinforcement learning. That is, we investigated the following questions:
(Q1) Do neural networks equipped with rational plasticity outperform rigid baselines?
(Q2) Can neural plasticity make up for more heavy algorithmic RL advancements?
(Q3) Can plasticity address the overestimation problem?
(Q4) How many more parameters would rigid networks need to measure up to rational ones?
To this end, we compare" our rational plasticity using the original DQN algorithm and its convolutional network (Mnih al., 2015) on 15 different games of the Atari 2600 domain (Brockman et al., 2017). We compare these architectures to ones equipped with Leaky ReLU (as experiments on Breakout and SpaceInvaders showed that agents with Leaky ReLU outperform ReLU ones), the learnable x sigmoid(z:) and its derivative dSiLU. Elfwing
PELU function, as well as SiLU (SiT.U(z) et al. showed that SiLU or a combination of SiLU (on convolutional layers) and its derivative (on fully connected layers) perform better than ReLU in DQN agents on several games (2018). SiLU and dSiLU are -to our knowledge- the only activation functions specifically designed for RL applications. More details on the architecture and hyperparameters can be found in Appendix A.8.
330.000 GPU hours, carried out DGX-2 Machine with Nvidia Tesla V100 with 32GB.
5

---

Published as a conference paper at ICLR 2024
LReLU SILU
rational
PELU
1000
LReLU
X rational
runs with scora
d+
DDON
joint-rat
DDQN
X joint-rational
.5
Human nomalized scores
x
100
Human Normalized Score (t)
LReLU
SILU
d+SiLU
DDQN
retional
jointrat
0.2
0.6
Asterix
Battlezone
Breakout
Enduro
Jamesstond
Kangaroo
Pong
Obert
Seaquest
Spacelnv.
Rennic
Timepplat
Katankham
VideoPinh
Superhuman Probability
Figure 3: Learnable functions' plasticity boosts RL. agents. For reliable evaluation, we report the performance profiles (top left) as well as superhuman probabilities (with CIs, bottom left) of baselines (i.e. DQN and DDQN with Leaky ReLU, DQN with SiLU and SiLU + dSiLU), with plasticity: using PELU, rational and joint-rational (5 random seeds). While the learnable PELU already augment performances of its agents, rational and joint-rational ones lift them above human performances on more than 70% of our runs. Detailed score tables are provided in Appendix A.3.
well as DQN
We then compare increased neural plasticity provided by improvements, namely the Double DQN (DDQN) method (van Hasselt et al., 2016), that tackles DQN's overestimation problem, as well as Rainbow (Hessel et al., 2018). Rainbow incorporates multiple algorithm improvements brought to DQN-Double Q-learning, prioritised experience replay, duelling network, multi-step target, distributional learning and stochastic networks and is widely used also as a baseline (Lin et al., 2020; Hafner et al., 2021). We further explain how neural plasticity help readdress overestimation. Finally, evaluate how many additional weights rigid networks need to approximate rational ones.
networks to algorithm
In practice, used safe rational functions (Molina al., 2020), i.e. we used the absolute value of the sum in the denominator to avoid poles. This stabilises training and makes the function continuous. Rational activation functions shared across layers (adding only 10 parameters per layer) through the whole network for the regularised (joint) version, with their parameters optimised together with the of the weights. For ease comparison and reproducibility, we conducted the original DQN experiment (also used by DDQN and SiLU authors) using the mushroomRL (D' Eramo et al., 2020) library, with the same hyperparameters (cf. Appendix A.8) across all the Atari agents, for a specific game, but we did not use reward clipping. For a fair comparison, we report final performances using (ef. Eq. Appendix) mean and standard deviation of the scores obtained by fully trained agents over five seeded reruns for every (D)DQN agent. However, since often only the best performing RL agent is reported the literature, we also provide tables of such scores (cf. Appendix A.3). For the Rainbow algorithm, single runs. A single run took more than 40 days on an NVIDIA Tesla V100 GPU; Rainbow is computationally quite demanding Oobando-Ceron & Castro, 2021).
unfortunately can only report the results of
(Q1) DQN with activation plasticity is better than rigid baselines. To start off, we compared RL agents with additional plasticity (from PELU and rationals) to rigid DQN baselines: Leaky ReI.U, as well as agents equipped with SiLU and SiLU+dSiLLU activation functions.
The results summarised in Fig. 3 confirm what our earlier figure had shown, but on larger seale. While RL agents with functions of the SiLU family do not outperform Leaky ReLU ones in our games, plastic DQN agents clearly outperform their rigid activation counterparts. DQN with regularised plasticity even obtains higher superhuman probability and highest mean scores 64% of the time. Scores on (difficult credit assignment) Skiin are in Appendix A.3. This clearly shows that plasticity, and above all rational plasticity, pays off for deep agents, providing affirmative answer Q1.

---

Published as a conference paper at ICLR 2024
Breakout
Enduro
Jamesbond
Kangaroo
1500
1000
1000
500
500
2K
0
0K
Obert
TimePilot
Seaquest
Spacelnvaders
15K
1000
5C
10K
5K
500
5K
0K
0K
200
400
0
200
400
epochs
200
400
200
400
DQN (fot. Platiiit))
DQN (Rog. Pasticity)
Rigld DQN
Riglt DOON
Rainbow (Rat. Plasticit)
Rainbow (Reg. Masticitt)
Rigid Rainbow
Figure 4: Networks with rational (Rat.) and regularised (Reg.) rational plasticity compared to rigid baselines (DQN, DDQN and Rainbow) over five random seeded runs on eight Atari 2600 games. The resulting mean scores (lines) and standard deviation (transparent area) during training are shown. As one can see, DDQN does resolve performance drops but only delays them (e.g. particularly pronounced on Seaquest). A figure including the evolution of every agent on all Atari 2600 games is provided in Appendix A.4, Figure best viewed in colour.
(Q2) Neural plasticity can boost performances of complex deep reinforcement learning ap- proaches, such as Rainbow. In Fig. 4, we show the learning curves of Rainbow and DQN agents, both with Leaky ReLU baselines, well as with full and regularised rational plasticity types. While Rainbow is computationally much heavier (~~ times slower than DQN in our experiments, with higher memory needs), its rigid form never outperforms the much simpler and more efficient DQN with neural plasticity, and its rational versions dominate in only out of 8 games (Enduro). In our experiments, Rainbow even lost to vanilla DQN on 3 games. These results show that augmenting the plasticity of an RL agent's 's modeling architecture can be of higher importance than bringing complex and computationally expensive improvements to the learning algorithm.
Therefore, DQN with rational plasticity is a competitive alternative to the complicated and expensive Rainbow method. Plasticity also improves Rainbow agents, answering question (Q2) affirmatively.
(Q3) Neural plasticity directly tackles the overestimation problem. Revisiting Fig. 4, one can see that Rainbow variants are worst on dynamic environments such as Jamesbond, Time Pilot and particularly Seaquest. For these games, the performance of rigid (Leaky ReLU) DQN progressively decreases. Such drops are well known in the literature and are typically attributed to the overestimation problem of DQN. This overestimation is due to the combination of bootstrapping, off-policy learning and a function approximator (neural network) operating by DQN. van Hasselt et al. showed that inadequate flexibility of the function approximator (either insufficient overestimations of state-action pairs (2016). The max operator in the update rule of DQN then propagates this overestimation while learning with the replay buffer. The overestimated states can stay in the buffer long before the agent revisit (and thus update) them. This can lead to catastrophic performance drops. To mitigate this problem, van Hasselt et al. introduced a second network to separate action selection from action evaluation, resulting in Double DQN (DDQN).
excessive) can lead to
We have compared rigid DDQN (equipped with Leaky ReL.U,, to vanilla DQN with neural plasticity on Atari games. As one can see in Fig. 3, DQN with rational plasticity outperforms the more complex (rigid) DDQN algorithm on every considered Atari game. This reinforces the affirmative answer to (QI) from earlier on. More importantly, we have computed the relative overestimation values of the (D)DQN, both with and without neural plasticity, following: overestimation O-value-ft where the return R corresponds to R Di= 7'r,, with the observed reward r. and the discount factor 7-
The results are summarised in Fig. 5. Plasticity helps to reduce overestimation drastically. DDQN substantially reduces overestimation on Jamesbond, Kangaroo, Tennis, Time Pilot and Seaquest. For these games, DDQN obtains the best performances among all rigid variants only on Jamesbond (cf. Tab. 3 in Appendix). Moreover, Fig. 4 reveals that the DDQN agents' performance drops are only delayed and not prevented. The performance drops thus happen after the 200th epoch, after which the agents" training usually stopped, no more performance increase seems achievable.
7

---

Published as a conference paper at ICLR 2024
10r
1266
Stationary Env.
Dynamic Env.
Progressive Env.
10
109
10-1
Kangaroo
Space Inw
Tennis
Asterix
Endure
@+bert
Jamnesoond
Seaquest
Time Pilet
Rigid DQN
Rational Plasticity on DQN
Regularized Rational Plasticity on DON
Rigid DDQN
Figure 5: Plasticity naturally reduces overestimation. Relative overestimation values (4, log scale) of rigid DQN and DDQN, as well as DQN with rational and regularised rational plasticity. Each trained agent is evaluated 100 completed games (5 seeds per game per agent). Agents with rational plasticity lower overestimation values as much or further than rigid DDQN ones, which has specifically been introduced this end. Figure best viewed in colour.
Overestimation might play a role in the performance drops on progressive environments (ef. Fig. Jamesbond, TimePilot and Seaquest), but cannot fully explain the phenomena. RI. agents with higher plasticity handle these games much better while embedding only a few more parameters. Hence, we advocate that neural plasticity better deals with distribution shifts of dynamic and progressive envi- ronments. Perhaps surprisingly, (regularised) rational plasticity not only works well on challenging progressive environments but also simpler flexibility is likely to hurt. Flexibility does not lead to overestimation (ef. Fig. 5). The rational func- tions for these games have simpler profile than of more complicated games like Kangaroo and Time Pilot (ef. Appendix A.6). The rational functions seem to adapt to the environment's complexity and the policy they need to model. This clearly provides an affirmative answer (Q3).
such Enduro, Pong and Qebert, where more
(Q4) Adding parameters through rationals efficiently augments plasticity. Compared rigid alternatives. the joint- roitonaational networks embed in total 10 additional parameters and always out- perform (cf. Fig. PELU (12 more parameters) ones. Our proposed method add plasticity via rational functions thus efficiently augments the capacity of the network. However, ReLU layers can theoretically approximate rational functions (Telgarsky, 2017). Augmenting the number of layers (or neurons per layer) is thus, theoretically, costly alternative augment the plasticity. How many parameters are practically needed rigid networks to obtain similar performances? Searching for bigger equivalent architectures for RL agents is tedious, RL training curves possess considerably more variance and noise than SL ones (Pang et al., 2021). but this question is not restricted to RL. We thus answer it by highlighting the generality of our insights. demonstrated by further investigation a classification scenario (cf. Appendix A.1). In short, rigid baselines need up to 3.5 times as many parameters the architectures that rational functions in order obtain similar performances.
All experimental results together clearly show that increasing neural plasticity, particularly through the integration of rational activations functions, considerably benefits deep reinforcement learning agents in highly computational efficient manner.
RELATED WORK
Next the related work discussed throughout the paper, our work on neural plasticity is also related to several research lines on neural architecture search and to the choice of activation functions, particularly in deep reinforcement learning settings.
The choice of activation functions. Many functions have been adopted across domains (e.g. Leaky ReLU in YOLO (Redmon et al., 2016), hyperbolic tangent in PPO (Schulman et al., 2017), GELU in GPT-3 (Brown et al., 2020)), indicating that the relationship between the choice of activation functions and the performances is highly dependent on the task, architecture and hyper-parameters.. As shown, parametric functions augment on plasticity. Molina et al. showed that rational functions can outperform other learnable activation function types supervised learning tasks (2020). Telgarsky (2017) showed that rationals are locally better approximants than polynomials. Loni et al. (2023) showed that searching for activation functions mitigates the performance drops of sparsity in networks.
8

---

Published as a conference paper at ICLR 2024
Neural Architectures for Deep Reinforcement Learning. Cobbe showed that the architecture of IMPALA (Espeholt et al., 2018), notably containing residual blocks, improved the performances over the original Nature-CNy, network used in DQN (2019). Motivated by these findings. Pang et al. (2021) recently applied neural architecture search to RL tasks and demonstrated that the optimal architecture highly depend the environment. Their search provides different architectures across environments, with varying activation functions across layers and potential residual connections. Continuously modifying the complexity of the neural network based on the noisy reward signal in complex architectural space is extremely resource demanding, particularly for large scale problems. Many reinforcement learning specific problems, such as noisy rewards (Henderson et al., 2018), input interdependency (Mnih et al., 2015), policy instabilities (Haarnoja et al., 2018), sparse rewards, difficult credit assignment (Mesnard et al., 2021), complicate an automated architecture search.
Plasticity in deep RL. A lot of attention has recently been brought to the plasticity of RL agents' learning structures. Abbas et al. (2023) have also identified their loss of plasticity and answered it using concatenated ReLU (CReLUU in Rainbow. Nikishin et al. (2022) periodically reset parts of the networks, Sokar et al. (2023) improved the resets by targeting identified dormant neurons. Similarly, Nikishin et al. (2023) inject plasticity via incorporating new trainable weights. Lyle et al. (2022) mitigate capacity (or plasiticity) loss, regularizing some features back to their starting values, and later showed that layer normalization help with plasticity (Lyle et al., 2023). Dohare et al. (2023) tackle dynamics with continual backprop and apply it to RL on PPO (Dohare et al., 2021), also varying between different non-learnable activation functions. Dynamically adapting the hyperparameter landscape is also improves agents' adaptability to distribution shifts (Zahavy et al., 2020; Mohan al., 2023). Testing how much much of these techniques can be covered by the use of rational plasticity is an interesting line of future work, as rational functions dynamically change the weights optimisation landscape. Fuks et al. (2019) adjust sub- cub-policies on sub- games find suitable hyperparameters that bootstrap main evolution- based optimised agent. Apart from using CReLU, all of these techniques are complementary to the use of rational plasticity.
LIMITATIONS, FUTURE WORK AND SOCIETAL IMPACT
We have shown the benefits of rational activation functions for RL,, as a consequence of both their neural and residual plasticity. In our derivation for closure under residuals, we have deduced that the degree of the polynomial in the numerator needs to be greater than that of the denominator. Correspondingly, we have based our empirical investigations on the degrees (5, 4). Interesting future work would be to further automatically select suitable such degrees, or even integrating rationals into dynamic hyperparameters' optimisation techniques. One should also explore neural plasticity in more advanced RL. approaches, including short term memory (Kapturowski et al., 2019), neurosymbolic approaches (Delfosse et al., 2024), finer exploration strategy (Badia et al., 2020), and in continual learning (Kudithipudi et al., 2022) techniques. Finally, the noisy optimisation performed in our RL experiments contribute to carbon emissions. However, this is usually a means to an end, as RL algorithms are also used to optimise energy distribution and consumption in several applications.
6 CONCLUSION
In this work, we have highlighted the central role of neural plasticity in deep reinforcement learning algorithms, and have motivated the use of rational activation functions, as lightweight way of boosting RL agents performances. We derived a condition, under which rational functions embed residual connections. Then the naturally regularised joint-rational activation function was developed, inspired by weight sharing in residual networks.
The simple DQN algorithm equipped with these (regularised) rational forms of plasticity becomes a competitive alternative to more complicated and costly algorithms, such as Double DQN and Rainbow. Fortunately, the complexity of these rational functions also seem to automatically adapt to the one of the environment used for training. Their use could be substitute for more expensive architectural searches. We thus hope that they will be adopted in future deep reinforcement learning algorithms, as they can provide agents with the necessary neural plasticity required by stationary, dynamic and progressive reinforcement learning environments.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The authors thank Elisa Corbean for her help on the manuscript revisions, as well as the anonymous reviewers of ICLR 2024 for their valuable feedback. This research work has been funded by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science within their joint support of the National Research Center for Applied Cybersecu- rity ATHENE, via the "SenPai: XReLeeS" project, from the German Center for Artificial Intelligence (DFKT). This work was also supported by the project "safeFBDC: Financial Big Data Cluster (FK7: 01MK21002K)" funded by the German Federal Ministry for Economies Affairs and Energy as part of the GAIA-x initiative. It benefited from the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK; project "The Third Wave of Al")
REFERENCES
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plasticity in continual deep reinforcement learning. ArXiv, 2023.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In 8th International Conference on Learning Representations (ICLR), 2020.
Chayan Banerjee, Zhiyong Chen, and Nasimul Noman. Improved soft actor-critic: Mixing prioritized off-policy samples with on-policy experience. ArXiv, 2021.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. ArXiv, 2017.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, (ICML), 2019.
Quentin Delfosse, Hikaru Shindo, Devendra Singh Dhami, and Kristian Kersting. Interpretable and explainable logical policies via neurally guided symbolic abstraction. 2023a.
Quentin Delfosse, Wolfgang Stammer, Thomas Rothenbacher, Dwarak Vittal, and Kristian Kersting, Boosting object representation learning via motion and object continuity. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML), 2023b.
Quentin Delfosse, Sebastian Sztwiertnia, Wolfgang Stammer, Mark Rothermel, and Kristian Kersting. Interpretable concept bottlenecks align reinforcement learning agents. ArXiv, 2024.
Carlo D' D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Mushroomrl: Simplifying reinforcement learning research. ArXiv, 2020.
Shibhansh Dohare, Ashique Rupam Mahmood, and Richard S. Sutton. Continual backprop: Stochastic gradient descent with persistent randomness. ArXiv, 2021.
Shibhansh Dohare, J. Fernando Hermandez-Garcia. Parash Rahman, Richard S. Sutton, and A. Rupam Mahmood. Maintaining plasticity in deep continual learning. ArXiv, 2023.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weigheet linear units for neural network function approximation in reinforcement learning. Neural Networks, 2018.
10

---

Published as a conference paper at ICLR 2024
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.
Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and regularization in DQN, ArXiv, 2018
Lior Fuks, Noor H. Awad, Frank Hutter, and Marius Thomas Lindauer. An evolution strategy with progressive episode lengths for playing games. In International Joint Conference on Artificial Intelligence, 2019.
D Garlick. Understanding the nature of the general factor of intelligence: the role of individual differences in neural plasticity as an explanatory mechanism. Psychological review, 2002.
Mariana-Iuliana Georgescu, Radu Tudor Ionescu, Nicolae-Catalle Ristea, and Nicu Sebe. Non-linear neurons with human-like apical dendrite activations. ArXiv, 2020.
Albert Gidon, Timothy Adam Zolnik, Pawel Fidzinski, Felix Bolduan, Athanasia Papoutsi, Panayiota Poirazi, Martin Holtkamp, Imre Vida, and Matthew Evan Larkum. Dendritic action potentials and computation in human layer 2/3 cortical neurons. Science, 2020.
Luke B. Godfrey. An evaluation of parametric activation functions for deep learning, IEEE Interna- tional Conference on Systems, Man and Cybernetics (SMC), 2019.
Mohit Goyal, Rajan Goyal, and Brejesh Lall. Learning activation functions: A new paradigm of understanding neural networks. ArXiv, 2019.
Klaus Greff, Rupesh Kumar Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. In 5th International Conference Learning Representations (ICLR), 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.
Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In 9th International Conference on Learning Representations (ICLR), 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (Cr (CVPR), 2016.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the Thirty-Second Conference on Artificial Intelligence AAAI, 2018.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the Thirty-Second Conference Artificial Intelligence (AAAI), 2018.
Steven Kapturowski, Georg Ostrovski, John Quan, Rémi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In 7th International Conference on Learning Representations (ICLR), 2019.
Srivatsan Krishnan, Amir Yazdanbaksh, Shvetank Prakash, Jason J. Jabbour, Ikechukwu Uchendu, Susobhan Ghosh, Behzad Boroujerdian, Daniel Richins, Devashree Tripathy, Aleksandra Faust, and Vijay Janapa Reddi. Archgym: open-source gymnasium for machine learning assisted architecture design. Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 10 (canadian institute for advanced research). URL http://wwww...ooooooooodott/i/ia

---

Published as a conference paper at ICLR 2024
Dhireesha Kudithipudi, Mario Aguilar-Simont Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh C. Bongard, Andrew Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, Anurag Reddy Daram, Stefano Fusi, Peter Helfer, Leslie M. Kay, Nicholas A. Ketz, Zsolt Kira, Soheil Kolouri, Jeffrey I.. Krichmar, Sam Kriegman, Michael Levin, Sandeep Madireddy, Santosh Manicka, Ali Marjaninejad, Bruce I.. McNaughton, Risto Miikkulainen, Zaneta Navratilova, Tej Pandit, Alice Parker, Praveen K. Pilly, Sebastian Risi, Terrence J. Sejnowski, Andrea Soltoggio, Nicholas Soures, Andreas Savas Tolias, Darío Urbina-Melêndez, Francisco J. Valero-Cuevas, Gido M. van de Ven, Joshua T. Vogelstein, Felix Wang, Ron Weiss, Angel Yanguas- Xinyun Zou, and Hava T. Siegelmann. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence. 2022.
Qianli Liao and Tomaso A. Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. ArXiv, 2016.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. In 8th International Conference on Learning Representations (ICLR), 2020.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shiens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In 15th European Conference on Computer Vision (ECCV), 2018.
Mohammad Loni, Aditya Mohan, Mehdi Asadi, and Marius Thomas Lindauer. Learning activation functions for sparse neural networks. ArXiv, 2023.
Liang Lu and Steve Renals. Small-footprint deep neural networks with highway connections for speech recognition. In 17th Annual Conference of the International Speech Communication Association (INTERSPEECH), 2016.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. ArXiv, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Ávila Pires, Razvan Pascanu, and Will Dabney, Understanding plasticity in neural networks. ArXiv, 2023.
Franco Manessi and Alessandro Rozza. Learning combinations of activation functions. In 24th International Conference on Pattern Recognition (ICPR), 2018.
Thomas Mesnard, Theophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Thomas S. Stepleton, Nicolas Heess, Arthur Guez, Eric Moulines, Mareus Hutter, Lars Buesing, and Rémi Munos. Counterfactual credit assignment in model-free reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
Diganta Misra. Mish: A self regularized monotonoo activation function. 31st British Machine Vision Conference 2020, (BMVC), 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness. Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King. Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mchdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016.
Aditya Mohan, C. F. Benjamins, Konrad Wienecke, Alexander Dockhom, and Marius I Lindauer. Autorl hyperparameter landscapes. ArXiv, 2023.
Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Padé activation units: End-to-end learning of flexible activation functions in deep networks. In 8th International Conference on Learning Representations (ICLR), 2020.
12

---

Published as a conference paper at ICLR 2024
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron C. Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, 2022.
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. ArXiv, 2023.
Johan S. Obando-Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
Dong Pang, Xinyi Le, and Xinping Guan. RL-DARTS: differentiable neural architecture search via meta-optimizer. Knowl. Based Syst., 2021.
lanfforcement-eerringg--.-
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Uni- fied, real-time object detection. In JEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Julien Roy, Paul Barde, Félix G. Harvey, Derek Nowrouzezahrrai, and Chris Pal. Promoting coordi- nation through policy regularization multi-agent deep reinforcement learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS), 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 2017.
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In International Conference on Machine Learning, 2016.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning, 2023.
Matus Telgarsky. Neural networks and rational functions. In Proceedings of the 34th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, 2017.
Ludovic Trottier, Philippe Giguère, and Brahim Chaib-draa. Parametric exponential linear unit for deep convolutional neural networks. In IEEE International Conference on Machine Learning and Applications (ICMLA), 2017.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth Conference on Artificial Intelligence (AAAI), 2016.
Andreas Veit, Michael Wilber, and Serge J. Belongie. Residual networks behave like ensembles of relatively shallow networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems (NeurIPS), 2016.
Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. Read and reap the rewards: Learning to play atari with the help of instruction manuals. Advances in Neural Information Processing Systems, 2024.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search.
In 7th International Conference on Learning Representations (ICLR), 2019.
Changqing Xu, Mingyue Wang, and Xian Li. Generalized vandermonde tensors. Frontiers of Mathematics in China, 2016.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep
reinforcement learning from pixels. In 9th International Conference on Learning Representations, (ICLR), 2021.
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, H. V. Hasselt, David Silver, and Satinder Singh. A self-tuning actor-critic algorithm. arXiv: Machine Learning, 2020.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In 5th International Conference Learning Representations (ICIR), 2017.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
As mentioned in the main body, the appendix contains additional materials and supporting information
for the following aspects: rational activation functions improving plasticity (4), comparison of rational and rigid networks with different sizes supervised learning experiments (A.1), results replacing residual blocks with rational activation functions (A.2), every final and maximal scores obtained by the reinforcement learning agents used our experiments (A.3), the evolutions of these scores (A.4), the different environment types with illustrations of their changes (A.5), graphs of the learned rational activation functions (A.6) and technical details for reproducibility (A.8).
Rational functions improve plasticity
To prove that rational can help with plasiticy, we tested them in continual learning settings (with more abrupt distribution shifts). We included Con- catenated RELU and rational functions to an ex- isting implementation of continual AI in which 4 layers (2 convolutional ones and to fully con-
100
MNIST
PERM.
PERM. 2
80
Thakk
Accuracy
60
Task
Rall
40
CReLU
Rational
nected ones) networks are trained on MNIST. The
20
network then continues training on PERM.1, variation of the dataset, for which a fixed random permutation is applied every image. Another permutation is used for PERM. 2. used after the training on PERMI. As shown in Fig. 6, networks with rationals are both better at modelling the new data (higher accuracies on the currently trained data), but are also able to retain more information about the data previously trained on. Networks with Continual ReLU (Shang et al., 2016) better retain information on Task 1, while performing on with ReLU ones for the 2 other tasks.
100 150
250
350
Training Images (K)
Figure 6: Rational function improve plasticity on the permutted MNIST experiment. Ratio- nal networks obtain better accuracies each currently and previously trained datasets.
A.1 RATIONAL EFFICIENT PLASTICITY CAN REPLACE LAYER'S WEIGHT PLASTICITY
We here show that networks with rational activations not only outperform Leaky ReLU ones with the same amount of parameters, but also to outperform deeper and more heavily parametrised neural networks (indicated by the colours). For example, a rational activated VGG4 not only performs better than rigid Leaky ReLU VGG4 at 1.37M parameters, but even performs similarly to the 4.7IM parameters rigid VGG6. Activation's 's plasticity allowing to reduce the number of layers weights is also shown by the experiments summarized in Tab. 2 in the next section, where blocks from pretrained ResNet are replaced by a rational function, and the resulting networks are able to recover and surpass their accuracies.
Architecture
VGG4
VGG6
VGG8
Activation function
IRReLU
Rational
I.ReLU
Rational
LRReIU
Rational
CIFAR 10
Training Acc@
Testing Acc @
83.0s.3
80.0=1.
87.11.6 84.3*.5
83.1
86.9+.2
89.2+.2
85.4..6
90.1t.1 85.0+11
86.9+.3
42.22
Training Acc@
64.62.8
70.41.9 58.94.6
70.74.6
86.02.9
87.7+.2
87.8 59.9+.4 4
CIFAR 100
Testing Acc@
56.S+.9
59.04.5
59.92.9
60.02.9
# Network parameters
1.37M
4.71M
9.27M
Table 1: Shallow rational networks perform as deeper Leaky ReLU ones. VGG networks training and testing top-1 accuracies with different numbers of layers are evaluated on CIFARIO and CIFARI00. Rational VGG4 has similar performances VGG6 network, with 3.5 times less parameters, and Rational VGG6 outperforms VGG8, with two times less parameters. Shaded colour pairs included for emphasis.
hwwpe//wg..attaac.omooooooooooae split_mmist.iyybb
14

---

Published as a conference paper at ICLR 2024
A.2 RESIDUAL BLOCK LEARN OF DEEP RESNET LEARN ACTIVATION FUNCTION-LIKE BEHAVIOUR.
We present in this section lesioning experiments, where a residual block is lesioned from a pretrained Residual Network, and the surrounding blocks are fine-tuned (with a learning rate of 0.001) for 15 epochs. These lesioning experiments were first conducted by Veit al. (2016). We also perform rational lesioning, where we replace block by an (identity initialised) rational activation function (instead of removing the block), and train the activation function along with the surrounding blocks. The used rational functions have the same order as in every other experiment ((m, n) (5, 4)). that satisfies the rational residual property derive in the paper). We report recovery percentages, computed following:
recovery 100 finetuned surgered
original surgered
We also provide the amount of dropped parameters of each lesioning.
Table 2: Rational functions improve lesioning. The recovery percentages for finetuned networks after lesioning (Veit al., 2016) of a ResNet layer's (L. block (B) are shown. Residual blocks were lesioned, i.e. replaced with the identity (Base) or a rational from a pretrained ResNet101 (44M parameters). Then, the surrounding blocks (and implanted rational activation function) are retrained for 15 epochs. Larger percentages are better, best results are in bold.
Recovery (%) Lesioning
L2B3
L3B19
L3B22
L4B2
Original (Veit al., 2016)
100.9
90.5
100
58.9
Training
Rational (ours)
101.1
104
120
91.1
Original (Veit et al., 2016)
93.1
97.1
81.6
81.7
Testing
Rational (ours)
90.5
97.6
91.5
85.3
% dropped params
0.63
2.51
2.51
10.0
As the goal is to show that flexible rational functions can achieve similar modelling capacities to the residual blocks, we did not apply regularisation methods and mainly focused on training accuracies. We can clearly observe that rational activation functions lead to performance improvements that even surpass the original model, or are able to maintain performances when the amount of dropped parameters rises.
A.3 COMPLETE SCORES TABLE FOR DEEP REINFORCEMENT LEARNING
Through this work, we showed the performance superiority of reinforcement learning agents that embed additional plasticity provided by learnable rational activation functions. We used human normalised scores (cf. Eq. 2) for readability. For completeness, we provide in this section the final raw scores of every trained agent. As many papers provide the maximum obtained score among every epoch and every agent, even if we consider it to be an inaccurate and noisy indicator of the performances, for which random actions can still be taken (because of e-greedy strategy also being used in evaluation). A fairer indicator to compare methods is the mean score. We thus also provide final mean scores (of agents retrained among 5 seeded reruns) with standard deviation. We start off by providing the human scores used for normalisation (provided by van Hasselt et al., in Table 5), then provide final mean and maximum obtained raw scores of every agent.
all weights initially but aj (and bp), both to 1.
15

---

Published as a conference paper at ICLR 2024
Algorithm
DQN
DDQN
DQN with Plasticity
Activation
LReLU
SiLU
d+SiLU
LReLU
PELU
rational
joint-rational
Asterix
1,85
0,52106
2.14
48.
25.8
242
168 132.6
Battlezone
11.4
21.2
11.3+67
68. 38
46.6#195
70.1+2a1
77.4
Breakout
158
93.9
11.7
286
788
1134
1210#360
Enduro
16.3+22.
37.0
0.37
47.
24.5++42.6
141 +:50
1494
Jamesbond
8.62.164
6.08
5.28
10.7
74.2:51.5
308
312.199.5
Kangaroo
11.8
128
13.9
17.2+145
17.7
107
1931868
Pong
101
96. 1:120
104
91.3
106.4
107.0#2.44
107.3
Qbert
55,4
14.2
2.74
74.0
1:66
120
117:4990
Seaquest
0.57
3,67
0.18
2.17
9.21
16.3
18,4+3.3
Skiing
-90.7
-111+-07
-85.5
-86.9+466
-111
-59.5=607
-60.2=56.1
Space Inv.
33.9+43
33.1
32.4+124
31.0+10
50.1
42.3
95.1+1777
Tennis
8.94
26.31533
78.5
32.1
106
257.81288
258.3
Timepilot
14.9
19.3
18.3
6.61
124
341
253:::1000
Tutankham
0,03
58.2+436
2.89
24.4+
91.6++29..
130
134
Videopinball
440#123
55.8
-4.03
626+24
299
1616#1025
9:6
# Wins
0/15
0/15
0/15
0/15
0/15
6/15
9/15
Super-Human 3/15
1/15
1/15
2/15
6/15
11/15
11/15
Table 3: Neural plasticity leads standard deviations (in percentage, cf. Appendix A.8 for the equation) of rigid baselines (i.e. DQN and DDQN with Leaky ReLU, DQN with SiLU and SiLU + dSiLU), as well as DQN with plasticity: using PELU, rational (full) and joint-rational (regularised), are reported over five experimental random seeded repetitions (larger mean values are better). The best results are highlighted in bold and runner-ups denoted with markers. The last rows summarise the number of times best mean scores were obtained by each agent and the number of super-human performances.
performance improvements. Normalised mean scores and
Final mean and maximum obtained scores of Rainbow agents:
Evaluation
Final Mean Scores
Max. Obtained Scores
Plasticity
rigid
full
regularised
rigid
full
regularised
Breakout
52
279
303
383
569
569
Enduro
844
1473
1470
1388
1973
1964
Kangaroo
40
2157
2139
6300
6000
4800
Qebert
149
11931
11551
16125
23550
23550
Seaquest
82
247
282
920
1280
1280
Space Inv.
595
1263
1157
2070
3395
2875
Time Pilot
3926
5386
6411
12700
15900
15900
Table 4: Final mean and maximum obtained scores obtained by rigid Rainbow agents (i.e. using Leaky ReLU), as well as Rainbow with full (i.e. using rational activation functions) and regularised (i.e. using joint-rational ones) plasticity (only run because of computational cost, larger values better).
16

---

Published as a conference paper at ICLR 2024
Final mean scores of all agents:
Algorithm
Random
DQN
DDQN
DQN with Plasticity
Network type
LReLU
SiLU
d+SiLU
LReLU
PELU
full
regularised
Asterix
67.9=22
206++9
107
228+1105
3723
1998
18109-1755
12621+248
Battlezone
38
4464
7612.14877
4429
22775
15807 -320
23403
25749128m
Breakout
0.14
155146
26.2
3.9
79.4
219
315
33
Enduro
Oze
121
274
2.77=341
353=1144
181
1043:
957
Jamesbond
6.39
37.6+2336
28.4-13.8
25.5+1662
45.2+-407
275+157
1122+176
1137=-216
Kangaroo
14.2+00
335
3500+-207
+.3++.504
484-2005
1586
2940
5266:-236
Pong
-20.211
15.9.2
14.1
16.9112
12.4
17.8
18209
18.111
Q*bert
40.6
6715
1754
371
8954
12143=795
14436
1 14080.1595
Seaquest
10.4
150
1504
94.62187.2
898=A58
3740
6603
7462
Skiing
-16104+92
+773
-29890+4
-26725
-26892
-29912+10
-23487+7704
2-3582+7708
Space Inv.
51.6=1.1
531+62
520+189
509+:76
490=15
759+4s
650+45
1395::251
Tennis
-23.910.0
-22.4130
-19.419.2
-10
-18.4188
-5,6192
20.5
20.6.10
TimePilot
688
1428
1644
1994
1016::401
6818
17632
13261::566
Turankham
2.54
3.55
81.9-m
7:41
36.422
127
179
184:140
VideoPinb.
6795.1461
15683=1183
11730
6439
62151 1:179
42051
11536
149712.19121
86942.14814
Table 5: Final mean raw scores (with std. dev.) of rigid baselines (i.e. DQN and DDQN with Leaky ReLU, DQN with SiLU and SiLU dSiLU), well DQN with full plasticity (i.e. using rational activation functions) and regularised plasticity (i.e. using joint-rational ones) on Atari 2600 games, averaged over 5 seeded reruns (larger mean values are better).
Maximum obtained scores:
Algorithm
Random
DQN
DDQN
DQN with Plasticity
Network type
LReLU
SiLU
d+SiLU
LReLU
PELU
full
regularised
Asterix
71
9250
3400
3800
20150
9300
84950
49700
Battlezone
843
88000
81000
70000
97000
68000
78000
94000
Breakout
427
370
344
411
430
864
864
Enduro
0
1243
928
1041
1067
1699
1946
1927
Jamesbond
5600
5750
700
7500
6150
9250
13300
Kangaroo
14800
15600
10200
13000
12400
16200
16800
Pong
-20
21
21
21
21
21
21
21
2crt
45
19425
11700
5625
19200
18900
24325
25075
Seaquest
20
7440
8300
740
15830
14860
9100
26990
Skiing
-15997
-5987
-6505
-6267
-5359
-5495
-5368
-5622
Space Inv.
2435
2205
2460
2290
2030
2490
3790
Tennis
-23
8
1
-1
4
-1
24
36
Time Pilot
730
11900
15500
12500
12200
16300
72000
28000
Tutankham
249
267
267
274
397
334
309
VideoPinb.
7599
998535
950250
338512
991669
322655
997952
998324
Table 6: Maximum obtained scores (with std. dev.) of rigid baselines (i.e. DQN and DDQN with Leaky ReLU, DQN with SiLU and SiLU + dSiLU), as well DQN with full plasticity (i.e. using rational activation functions) and regularised plasticity (i.e. using joint-rational ones) on Atari 2600 games, averaged over seeded reruns (larger values are better).
Human scores used for normalisation:
Asterix: 7536,
Battlezone: 33030,
Breakout: 27.9,
Enduro: 740.2, Jamesbond: 368.5.
Kangaroo: 2739,
Pong: 15.5. Qe 12085, Seaquest: 40425.8, Skiing: 3686.66 Space
Invaders: 1464.9,
Tennis: -677, Time Pilot: 5650,
Tutankham: 138.3, Video Pinball: 15641.1
17

---

Published as a conference paper at ICLR 2024
A.4 EVOLUTION OF THE SCORES ON EVERY GAME
The main part present some graphs that compares performance evolutions of the Rainbow and DQN agents with plasticity, well Rigid DQN, DDQN and Rainbow agents. We here provide the evolution of the scores of every tested DQN and the DDQN agents on the complete game set. DQN agents with higher plasticity are always the best-performing ones, Experiments on several games (c.g. Jamesbond, Seaquest) show that using DDQN does not prevent the performance drop but only delays it.
336
Setl
25C
DOON (RAL
226
(PON PELU
(POR 563--d540
DON SELL
106
200
Enduro
Jameshond
Kangaraa
1200
2430
18.
1201
2350
203
602
48
433
209
-202
0K
Pong
Obert
Seaquest
14
1.20
08
Skiing
Spacelnvaders
Tonnis
154
3990
-304
330
530
-34
300
150
200
200 Tutankhamn
300
TimmePllot
VideoPinball
2508
28c
2004
1.04
38C
330
200
300
300
100 200 200
700
100
353
175
200
Figure 7: Smoothed (ef. Eq. 3) evolutions of the scores on every tested game for DQN agents with full (i.e. using rational activation functions) and regularised (i.e. using joint-rational ones) plasticity, and original DQN agents using Leaky ReLU, SiLU and SüLU+dSiLU, as well as for DDQN agents with Leaky ReLU,
18

---

Published as a conference paper at ICLR 2024
A.5 ENVIRONMENTS TYPES: STATIONARY, DYNAMICS AND PROGRESSIVE
The used environments have been separated in 3 categories, describing their potential changes through agents learning. This categorisation is here illustrated with frames of the tested games. As can see: Breakout, Kangaroo, Pong. Skinng, Space Invaders, Tennis, Tutankham and VideoPinball can be categorised stationary environment, changes are minimal for the agents in these games. Asterix, BattleZone, Qebert and Enduro present environment changes, that are early reached by the playing agents, and are thus dynamic environments, Finally, Jamesbond, Seaquest and Time Pilot correspond to progressive environments, as the agents needs to master early changes to access new parts of these environments.
Asterix
Battle Zone
Breakout
Enduro
Jamesbond
Kangaroo
Pong
Qebt
Seaquest
Skiing
Space
Tennis
Time Pilot Tutankham Video Pinb.
000300
EATI
Figure 8: Images extracted from DQN agents with full plasticity playing the set of 15 Atari 2600 games used this paper. Stationary environments (e.g. Pong. Video Pinball) do not evolve during training, dynamic ones provide different input/output distributions that are early accessible in the game (e.g **bert, Enduro) and progressive ones (e.g. Jamesbond, Time Pilot) require the agent to improve for the it to evolve.
19

---

Published as a conference paper at ICLR 2024
A.6 LEARNED RATIONAL ACTIVATION FUNCTIONS
We have explained in the main text how rational functions of agents used different games can exhibit different complexities. This section provides the learned parametric rational functions learned by DQN agents with full plasticity (left) and by those with regularised plasticity (right) after convergence for every different tested game of the gym Atari 2600 environment, Kernel Density Estimations (with Gaussian kernels) of input distributions indicates where the functions are most activated. Rational functions from agents trained on simpler games (e.g. Enduro, Pong, Q*bert) have simpler profiles (i.e. fewer distinct extremas).
Asterix
-10
10
BattleZone
20
-10
10
Breakout
-50
-00
10
-20
-10
Enduro
Jamesbond
Kangaroo
Pong
Obert
Seaquest
-2
3.0
-25
0.0
-10
20

---

Published as a conference paper at ICLR 2024
Skilin
-20
-10
20
Spacelnvader
-20
10
-5
Tennis
-10
TimePlloc
Tutankham
VideoPinbail
Figure 9: Profiles (dark blue) and input distributions (light blue) of rational functions (left) and joint-rational ones (right) of DQN agents on the different tested games. (Joint-)rational functions from agents of simpler games have simpler profiles (i.e. fewer distinct extrema).
21

---

Published as a conference paper at ICLR 2024
A.7 EVOLUTION OF RATIONALS ON THE
CONTINUAL LEARNING EXPERIMENT
Figure 10 depicts the evolutions of rational functions through the permuted MNIST experiment. One can see that while the function the first layer remains stable through the successive datasets, second one flatten at its most activated region (around 0), while the third one increase its slope in this region, leading to higher gradients. This suggests that rational functions can help adapting the gradient scales each layer. Further investigating this is interesting line of future work.
Evolution of the rational functions on the Permuted MNIST experiment
Before training
0.15
0.4
0.2
0.05
0
0.0
0.00
0.55
0.5
0.0
-0.4
After MNIST
2-
1-
2.5
-2
-1.0
-0.5
0.0
0.5
1.0
1.5
-2
0
-5
5
After PERM1
2
5-
2
0
0
-5
-2-
-1
-5
After PERM2
2-
1-
-2
Layer
Layer
Layer
MNIST
PERMI
PERM2
Figure 10: Evolution of the rational activation functions on the permuted MNIST experiment (cf. 4). The 3 rational activation functions used for training (and retraining) are adapting fit the data (depicted semi transparent).
A.8 TECHNICAL DETAILS TO REPRODUCE THE EXPERIMENTS
We here provide details on our experiments for reproducibility. We used the seed 0, 1, 2, 3, for every multi-seed experiment.
SUPERVISED LEARNING EXPERIMENTS
For the lesioning experiment, we used an available" pretrained Residual Network. We then remove the corresponding block (and potentially replace it with an identity initialised rational activation function) (surgered). We finetune the new models, allowing for optimisation of the previous and next layers (and potentially the rational function) for 15 epochs with SGD (learning rate of 0.001).
22

---

Published as a conference paper at ICLR 2024
For the classification experiments, License), we let every network learn for 60 epochs. We use the code provided by Molina et al. (2020), with only one classification layer in these smaller VGG versions (VGG4, VGG6 and VGG8, against 3 for VGG16 and larger). We use SGD as the optimisation algorithm, with a learning rate of 0.02 and 128 batch size. The VGG networks contain successive VGG blocks that all consist of convolutional layers, input channels and output channels, stride 3 and padding 1, followed by an activation function, and Max Pooling layer. For each used architecture, the (n, o) parameters of the successive blocks are:
CIFARIO and CIFARIO0 (Krizhevsky et al., MIT
VGG4: (1,3, 64) + (1,64, 64, 128) + (2, 121, 256)
VGG6: (1,3, 64) - (1,64, 128) - (2, 128, 256) + (2, 256, 512)
VGG8: (1,3, 64) (1, 64, 128) (2, 128, 256) + (2, 256, 512) - (2, 512,
The output of these blocks is then passed on to a classifier (linear layer). Only activation functions differ between the Leaky ReLU and the Rational versions.
REINFORCEMENT LEARNING EXPERIMENTS
To ease the reproducibility of the reinforcement learning experiments. we used the Mushroom RL library (D" Eramo et al., 2020) on the Arcade Learning Environment (GNU General Public License). We used states consisting of consecutive grey-scaled images, downsampled to 84 84. Computing the gradients for rational functions takes longer than e.g. ReLU. However, we used a CUDA optimized implementation of the rational activation functions that we open source along with this paper. In practice, we did not notice any significant training time difference.
Network Architecture. The input to the network is thus 84x84x4 tensor containing rescaled, and gray-scaled, version of the last four frames. The first convolution layer convolves the input with 32 filters of size 8 (stride 4), the second layer has 64 layers size 4 (stride 2), the final convolution layer has 64 filters of size 3 (stride 1). This is followed by a fully-connected hidden layer of 512 units. All these layers are separated by the corresponding activation functions (cither Leaky ReLU, SiLU, SiLU for convolution layers and dSiLU for linear ones, PELU, rational functions (at each layer) and joint-rational ones (shared accross layers) of order m = 5 and n. = 4, initialised to approximate Leaky ReLU). We used the default PeLU initial hyperparameters (a=1, b=1, c=1) and let the weights optimizer tune them through training, as for rational functions. For CRELU, we took the implementa- tion from ML Compiled and halves the number of filters in the following convolutional layers to keep the same network structure intact, as done by Shang et al. (2016).
Hyper-parameters. We evaluate the agents every 250K steps, for 125K steps. The target network is updated every 10K steps, with a replay buffer memory of initial size 50K, and maximum size 500K, except for Pong. for which all these values are divided by 10. The discount factor y is set to 0.99 and the learning rate is 0.00025. We do not select the best policy among seeds between epochs. We use the simple c- e-greedy exploration policy, with the decreasing linearly from 0.1 over 1M steps, and an of 0.05 is used for testing.
The only difference from the evaluation of Mnih et al. (2015) and of van Hasselt et al. (2016) evaluation is the use of the Adam optimiser instead of RMSProp, for every evaluated agent.
Normalisation techniques. To compute human normalised scores, we used the following equation:
agent score random
scorenormaliee
100 score human score random
For readability, the curves plotted in the Fig. 4 and Fig. 8 are smoothed following:
score, = a score, + (1 - ax) x scorest;
with a = 0.9.
Overestination computation. We used the following formulae to compute relative overestimation.
Q-value R
overestimation
R
23

---

Published as a conference paper at ICLR 2024
RL NETWORK ARCHITECTURE
The DQN, DDQN and Rainbow agents networks architecture, rational plasticity (with rational activations functions at each layer) and of the regularized ones (with one joint-rational activation function shared across layers). For the other activation functions, the "Rat." " blocks are replaced with Leaky ReLU, CReLU, SiLU, or PELU. For the d+SiLU networks, SiLU is used on the convolutional layers (i.e. first two), and dSiLU in the fully connected ones (i.e. last two).
Convolation
Convolution
Add
[6]
(4)
(6)
Add
(4)
(64)
644 32,
Rat.
3oint
164, 32,
(64)
Convolution
Convolution
Add
(6)
(4)
Add
(64)
64, 64, 3, 3
Rat.
64, 64, 3, 3
Joint Rat.
(64]
Convelation
Convoluticn
View
(6)
Ratt.
S1 2, 3136
612, 31:36)
Joint Rat..
(512)
View
Mult
(512)
Mult
View
16, 512)
Ratt.
(8)
MuLi
Joint Rat.,
(G)
Addmm
Addmm
Figure 11: left: The DQN agents' neural network equipped with Rational Activation Functions (Rat.). Any other network with classical activation functions (as Leaky Relu or SiLU) would be similar, with the corresponding activation function instead of the rational one. right: The agents' network using the regularized joint-rational version of the network. The same activation is used across the layers. The parameters of the rational activation (in orange) function are shared. In both graphs, operations are placed in the grey boxes and parameters in the blue ones, (or orange for the rationals' ones).
24


Reviews:
[{"review_id": "94QoptUQhW", "reviewer": "Reviewer_cKgE", "summary": "The paper argues for plasticity in RL through adaptive activation functions. It proposes using (safe) rational activation functions as a plug-in modification to existing value-based RL methods (DQN-style algorithms).\n\nThe authors first motivate the need for plasticity through activation functions in constantly changing reinforcement learning environments. They categorize environments into static, dynamic, and progressively evolving environments. The authors then show how existing methods, such as DQN and non-rational plasticity variants (PELU), fail to adapt to all changes, whereas rational activations can.\n\nHaving established Rational activation functions as promising candidates for imbuing existing neural networks with plasticity, the authors show that rational activation functions can embed residual connections. Using this insight, they develop a joint rational activation function that is a regularized form of rational plasticity through weight-sharing across activations in different layers.\n\nThe authors finally demonstrate through experiments that equipping popular algorithms such as a DQN with rational activations leads to consistent improvements in performance on Atari games, elevating them to be competitive alternatives to highly engineered and computationally expensive methods such as RAINBOW.", "questions": "- How is this rational plasticity related to previous works on plasticity in RL? [Sokar et. al, 2023, Abbas et. al, 2023,Lyle et al., 2023,Nishikin et al., 2022]\n- Parametric forms to capture existing activation functions have previously been used in supervised learning [Loni et al., 2023]. How is the rational form placed regarding these works? Does the comparison to SiLU and PELU capture this already?\n- It is argued in sections 1 and 2 that RL can significantly benefit from adaptive neural architectures. However, another critical aspect of RL algorithms is hyperparameter optimization, with potentially the requirement to adapt hyperparameters as an RL agent dynamically trains [Mohan et al., 2023]. To what extent can the adaptable rational activation functions in RL mitigate these requirements?\n- Since the benefit of applying rational activation functions lies in their ability to adjust their parameters dynamically, shouldn't they be additionally compared to Baselines of dynamic hyperparameter optimization, such as hyperparameter schedules [Zahavy et al., 2020], progressive episode lengths [Fuks et al., 2019]? I think it makes sense to at least justify whether relationships between the use of rational activation functions for improving adaptability and methods to tackle the same issue in this literature does exist or not.\n\n### References\n- [Sokar et. al, 2023] Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci.  The dormant neuron phenomenon in deep reinforcement learning. arXiv preprint arXiv:2302.12902, 2023.\n- [Abbas et. al, 2023] Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.07507, 2023.\n- [Lyle et al., 2023] Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. arXiv preprint arXiv:2303.01486, 2023.\n- [Nishikin et al., 2022] Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville.  The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, pages 16828–16847. PMLR, 2022.\n- [Loni et al, 2023] Loni, M., Mohan, A., Asadi, M., & Lindauer, M. (2023). Learning Activation Functions for Sparse Neural Networks. arXiv preprint arXiv:2305.10964.\n- [Mohan et al, 2023] Mohan, A., Benjamins, C., Wienecke, K., Dockhorn, A., & Lindauer, M. (2023). Autorl hyperparameter landscapes. AutoML Conference 2023 (https://openreview.net/forum?id=Ec09TcV_HKq).\n- [Zahavy et al., 2020] Zahavy, T., Xu, Z., Veeriah, V., Hessel, M., Oh, J., van Hasselt, H. P., ... & Singh, S. (2020). A self-tuning actor-critic algorithm. Advances in neural information processing systems, 33, 20913-20924.\n- [Fuks et al., 2019] Fuks, L., Awad, N. H., Hutter, F., & Lindauer, M. (2019, August). An Evolution Strategy with Progressive Episode Lengths for Playing Games. In IJCAI (pp. 1234-1240).\n\nI have increased my score based on the changes made by the authors", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "### Originality\nThe proposed application of the activation function to the RL pipeline is a novel contribution. \n\n### Quality\nThe quality of the work is generally high.\n\n### Clarity\nThe work is generally presented clearly.\n\n### Significance\nActivation function optimization has shown considerable success in other areas, such as pruning. Therefore, applying them as an alternative to heavy architectural search and per-environment hyperparameter optimization methods is a significant step.", "weaknesses": "- The work seems to miss some previous work on plasticity in RL. Please see the questions section for more information on related works. Overall, I think discussing this would further concretize the argumentation for rational plasticity while placing this work better in the RL literature. \n- Hyperparameters are not at all discussed as a requirement for adaptation. I believe this is a critical aspect of designing RL pipelines, and therefore, a focus on only adaptable Neural architectures is limiting, in my opinion. It would make a lot of sense if the authors explicitly state how they see their work as related to this literature.\n- I think the baselines need more justification. While it is understandable that the focus is on activation functions, so comparisons with static methods make sense, the authors also argue that the dynamically changing nature of rational activations allows RL agents to adapt to changing environmental circumstances. This usability argument, in my opinion, is not captured by the considered baselines since, most of the time, when we are talking about addressing changing aspects of environments, the focus is either on algorithmic improvements, continual and curriculum-based methods, or dynamic hyperparameter adaptation mechanisms. I believe either making strong arguments as to why the authors do not consider these approaches or performing comparisons to one of them would significantly boost this line of argumentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aOj6KdzDuM", "reviewer": "Reviewer_7Gju", "summary": "The manuscript argues that plasticity is important for RL. The paper proposes rational activation functions for RL. It also proposes a regularized version of the activation function (joint rational). The results show that rational activation functions significantly outperform rigid activation functions like leaky-ReLU.", "questions": "Can you please clarify what exactly is the rational activation function? The definition says $R(x)$ where x is a real number, but the equation uses $x^j$, what is $x^j$?\n\n------------------\nI've updated my score in light of the new results.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea of using rational activation functions to inject plasticity is very interesting, and the results are promising. DQN with rational activation functions significantly outperforms vanilla DQN. The use of rational activation in RL seems novel. I'm not aware of any work that uses these activations in RL. Most of the paper is clearly written and easy to understand.", "weaknesses": "Although the paper contains exciting ideas and promising results, there are many significant weaknesses in the paper that stop me from recommending a full acceptance. \n- **Missing literature on plasticity.** The manuscript needs to include the entire literature on plasticity. There are already many papers that show that there is a loss of plasticity when deep learning systems face a non-stationary data stream, such as in RL (Dohare et al., 2021; Lyle et al., 2022; Nikishin et al., 2022; Abbas et al., 2023; Lyle et al., 2023; Sokar et al., 2023; Nikishin et al., 2023; Dohare et al., 2023). Missing this literature also means that the paper is missing important baselines in the paper methods like CReLUs, selective reinitialization, layer normalization, etc. I realize that doing a full comparison in Atari with all of these methods might not be computationally feasible, but they should be compared in a smaller problem.\n- **Unsupported conclusions.** The manuscript claims that rational activation functions help because they inject plasticity. However, the paper does not provide any direct evidence of this claim. The paper shows that deep RL methods with rational activations outperform deep RL methods with rigid activations. Better performance does not mean better plasticity. However, this problem can be somewhat reduced by comparing rational and rigid activations in a continual supervised learning problem like random label MNIST (Lyle et al., 2023) or permuted MNIST (Dohare et al., 2023). Then, we will have direct evidence that rational activations inject plasticity.\n\nDohare, S., Sutton, R. S., & Mahmood, A. R. (2021). Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325.\n\nNikishin, E., Schwarzer, M., D’Oro, P., Bacon, P.-L. & Courville, A. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, 16828–16847 (PMLR, 2022).\n\nLyle, C., Rowland, M. & Dabney, W. Understanding and preventing capacity loss in reinforcement learning. In\nInternational Conference on Learning Representations (2022).\n\nAbbas, Z., Zhao, R., Modayil, J., White, A., & Machado, M. C. (2023). Loss of plasticity in continual deep reinforcement learning. CoLLAs 2023.\n\nSokar, G., Agarwal, R., Castro, P. S. & Evci, U. The dormant neuron phenomenon in deep reinforcement learning. In Krause, A. et al. (eds.) Proceedings of the 40th International Conference on Machine Learning, vol. 202 of Proceedings of Machine Learning Research, 32145–32168 (PMLR, 2023).\n\nLyle, C. et al. Understanding plasticity in neural networks. In Krause, A. et al. (eds.) Proceedings of the 40th International Conference on Machine Learning, vol. 202 of Proceedings of Machine Learning Research, 23190–23211 (PMLR, 2023).\n\nNikishin, E., Oh, J., Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., & Barreto, A. (2023). Deep Reinforcement Learning with Plasticity Injection. arXiv preprint arXiv:2305.15555.\n\nDohare, S., Hernandez-Garcia, J. F., Rahman, P., Sutton, R. S., & Mahmood, A. R. (2023). Loss of Plasticity in Deep Continual Learning. arXiv preprint arXiv:2306.13812.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RPPoABaa0o", "reviewer": "Reviewer_mxps", "summary": "Neural plasticity has been shown to be crucial to reinforcement learning (RL), especially to environments that change constantly and/or rapidly. This work proposes the use of adaptive activation functions to increase plasticity. Particualrly, it employs a form of rational activation functions, i.e., joint-rational activation, to have greater flexibility while avoiding overfitting. Experimentally, the work shows that such functions can make DQN competitive with DDQN and Rainbow without all the additional techniques.", "questions": "1. I am not very familiar with the area of adaptive activation functions. But what in particular do rational activation functions offer that make it superior over some of the options out there? E.g., I came across [1]\n2. Can the authors provide some intuition on why improved plasticity lead to reduced overestimation?\n\nJagtap, A. D., Kawaguchi, K., & Karniadakis, G. E. (2020). Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics, 404, 109136.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The work tackles the important problem of plasticity in deep RL with a novel method based on adaptive activation functions.\n2. Showing using rational functions can compensate for residual block is an intriguing and insightful finding", "weaknesses": "1. Missing related works in the study of plasticity in deep RL (e.g., [1])\n2. There are works out there that tackles plasticity in deep RL. They should be added as baselines to compare with the proposed method (e.g., [2], [3])\n3. Results would be a lot stronger if it can show that the results transfer across algorithms too (e.g., policy gradient methods)\n\n[1] Abbas, Z., Zhao, R., Modayil, J., White, A., & Machado, M. C. (2023). Loss of plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.07507.\n\n[2] Nikishin, E., Oh, J., Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., & Barreto, A. (2023). Deep Reinforcement Learning with Plasticity Injection. arXiv preprint arXiv:2305.15555.\n\n[3] Dohare, S., Sutton, R. S., & Mahmood, A. R. (2021). Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lJPa2kx2YJ", "reviewer": "Reviewer_qikv", "summary": "The authors study the use of rational activation functions for RL.  They show that these functions embed residual connections.  They study them empirically.", "questions": "See clarity concerns above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "An interesting, well-motivated idea, with a large number of various, potentially significant contributions.", "weaknesses": "**Small edits and minor recommendations:**\n\n- \"emphasise\" spelling error in 2.1\n- In the “Network Architecture” section of the appendix, there is a parentheses mismatch.\n- Does “resp.” mean respectively?  This stumped me for a couple minutes, I suggest being more clear and not using abbreviations like this, or at least defining them before use.\n- 2.3: “However, if we circle back to this figure” Which figure? Be more clear; don’t make your reader work unnecessarily hard to understand.\n\n**Clarity Weaknesses:**\n- 2.1: the authors say very little about the architecture.  Is it conv nets?  Is the number of units between the baselines and their approach fixed, and they replace each unit’s activation function with a rational plasticity unit (resulting in many more parameters for their approach)?  Or do they replace whole layers with a single rational plasticity unit (the parenthetical in 2.1 seems to hint at this, but it’s unclear)?  Or do they keep all units from each layer intact, and add a singe additional rational plasticity unit (if I recall correctly, this is what the original rational activation function paper did)?  These things are critically important pieces of information about this paper’s results (this is essentially an architecture paper, but the reader is left guessing about the most basic aspects of the architectures used), and this critical information should not be relegated to an appendix.  Worse, the last last question (how exactly rational plasticity units replace traditional units in an architecture) is unclear to me, even after reading the Network Architecture section of the appendix.  Update on this last: more details are given, almost parenthetically, on page 6 and again on page 8 (in the Q4 discussion).  However, the long delay hinders clarity, and even with these details, the exact details of the approach/architecture remain unclear.  I suggest clarifying all of these questions in 2.1; a figure could be very helpful.\n\n- “To this end, we also show agents equipped with rational activations that are tempered in Fig. 2. They have been extracted from agents with rational plasticity, that adapted to their specific environment. The plasticity of the rational activation functions is then tempered (“stopped”) in repeated application to emphasise the necessity to continuously adapt during training.” I am not sure exactly what this means; it is too vague and can think of several different interpretations with very different meanings.  In the second sentence, it sounds like they freeze these rational activations after a while, but let the other network weights continue to be optimized.  But then the third sentence seems to contradict this (what does “repeated application” mean, and why would freezing some parameters “emphasize the necessity to continuously adapt”?)\n\n- 2.2 Theorem concern: The precise meaning of this theorem is unclear.  As stated, it can easily be taken to mean that all rational activation functions implicitly contain a residual connection.  But if I’m understanding the proof correctly, it seems to indicate that it depends on the weights; that is, that a residual connection can be learned if necessary.  This is confusing and potentially misleading.  I suggest stating the Theorem more precisely.\n\n- 2.3: “regularized joint-rationals, where the key idea is to constraint the input to propagate through different layers but always be activated by the same learnable rational activation function”.  This is incredibly vague, and does not tell the reader what this approach means (and this sentence seems to constitute the entire description of joint rationals).\n\n**Other weaknesses:**\n- Is plasticity actually causing the better results?  Q3 is a very nice analysis, and provides evidence that the proposed approach reduces overestimation.  However, it is not clearly demonstrated that: 1) the plasticity in particular is what causes this reduction in overestimation OR 2) that the reduction in overestimation is what causes the improved performance in Q1 and Q2. BOTH of these things would need to be shown to support the authors’ claims.  In fact, Figure 2 shows evidence against the former: rational activation functions in general seem to simply perform better even for stationary environments, which is nice, but seems to contradict the core thesis of the paper (rational activation functions cause better plasticity which in turn causes better performance).\n- The above is even more concerning in light of the lack of details about architectures used.  Perhaps the better performance is simply due to their approach using networks with more parameters and thus more capacity?  Unfortunately, it is not even clear how the different approaches’ architectures vary, so it is difficult for a reader to even begin to evaluate this concern.\n\n\nNote: Despite the many significant concerns I have about this paper, I think it already has many nice results and strengths as well, in addition to being a well-motivated work.  If a rating of 4 were an option, I would give it a 4 instead of a 3.  I encourage the authors to work to address the clarity issues and the \"Is plasticity actually causing the better results?\" concern, and resubmit; I think this is a potentially very strong contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)