Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimization
Authors: Abdullah Akgül
Gulcin Baykal
Manuel Haussmann
Melih Kandemir
Venue: Accepted by TMLR ()

Abstract: Continuous control of non-stationary environments is a major challenge for deep reinforcement learning algorithms. The time-dependency of the state transition dynamics aggravates the notorious stability problems of model-free deep actor-critic architectures. We posit that two properties will play a key role in overcoming non-stationarity in transition dynamics: (i)~preserving the plasticity of the critic network and (ii) directed exploration for rapid adaptation to changing dynamics. We show that performing on-policy reinforcement learning with an evidential critic provides both. The evidential design ensures a fast and accurate approximation of the uncertainty around the state value, which maintains the plasticity of the critic network by detecting the distributional shifts caused by changes in dynamics. The probabilistic critic also makes the actor training objective a random variable, enabling the use of directed exploration approaches as a by-product. We name the resulting algorithm \emph{Evidential Proximal Policy Optimization (EPPO)} due to the integral role of evidential uncertainty quantification in both policy evaluation and policy improvement stages. Through experiments on non-stationary continuous control tasks, where the environment dynamics change at regular intervals, we demonstrate that our algorithm outperforms state-of-the-art on-policy reinforcement learning variants in both task-specific and overall return.

Published in Transactions on Machine Learning Research (11/2025)
Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimization
Abdullah Akgill
akgul@imada.sdu.c du.dk
Deportment of Mathematics and Computer Science University of Southern Denmark
Gulcin Baykal
baykalg Simada.ss...a dk
Department of Mathemalics and Compuler Science University of Southern Denmark
Manuel Haußmann
houssmannn@imaaa....uu
Department of Mathematics and Computer Science University of Southern Denmark
Melih Kandemir
kandemir@imadaa.sde
Deportment of Mathematics and Computer Science Universily of Southern Denmark
Reviewed on OpenReview:
Abstract
Continuous control non-stationary enviromments a major challenge for deep reinforcement learning algorithms. The time-dependency of the state transition dynamics aggravates the notorious stability problems of model-free deep actor-critic architectures. We posit that two properties will play key role in overcoming non-stationarity in transition dynamics: (i) preserving the plasticity of the critic network and (ii) directed exploration for rapid adaptation to changing dynamics. We show that performing on-policy reinforcement learning with an evidential critic provides both. The evidential design ensures a fast and accurate approximation the uncertainty around the state value, which maintains the plasticity of the critic network by detecting the distributional shifts caused by changes dynamics. The probabilistic critic also makes the actor training objective a random variable, enabling the use of directed exploration approaches Evidential Prorimal Policy Optimization (EPPO) due to the integral role of evidential uncertainty quantification in both policy evaluation and policy improvement stages. Through experiments on non-stationary iontinuous control lasks, where the environment dynamics change at regular intervals, we demonstrate that our algorithm outperforms saaee-f--thearr on-policy reinforcement learning variants both task-specific and overall return.
by-product. We name the resulting algorithm
1
Introduction
Most deep reinforcement learning algorithms developed assuming stationary transition dynamics, even though in many real-world applications the transition distributions time-dependent, iee, non-stationary (Thrun, 1998). The non-stationarity of state transitions makes it essential for the agent to keep updating its policy. For example, a robotic arm may experience wear and lear, leading to changes in the ability of its joints to apply torque, or an autonomous robot navigating a terrain with varying ground conditions, such as friction, inclination, and roughness. In such environments, an agent can maintain high performance only by continually adapting its policy to changes. On-policy algorithms, such as Proximal Policy Optimization (PPO) (Schulman al., 2017), are particularly well-suited for non-stationary environments because they rely

---

Published in Transactions on Machine Learning Research (11/2025)
solely data from the most recent policy, ensuring policy improvement through sufficiently small updates (Kakade & Langford, 2002). This makes PPO an attractive choice for applications ranging from physical robotics (Melo & Máximo, 2019) to fine-tuning large language models (Touvron et al., 2023; Achiam et al., 2023; Zheng et al., 2023). Agents designed for open-world, non-stationary environments have to contimually learn throughout their entire lifecycle, not just during a fixed training phase. Time-dependent changes in state transition dynamics result in non-stationary Markov decision processes (MDPs), where existing reinforcement learning algorithms often struggle to adapt elfectively.
We posit that the simultaneous presence of two key features is essential for overcoming the challenges caused by non-stationarity in deep reinforcement learning:
( Maintaining the plasticity of the critic network: Plasticity refers to the ability of a neural network to change its wiring in response to new obscrvations throughout the complete learning period. Deep reinforcement. learning algorithms have been reported to sulfer from the loss of plasticity in non-stationary settings by a vast body of earlier work (Dohare et al., 2021; Lyle et al., 2022; Nikishin al., 2022; Abbas et al., 2023; Dohare et al., 2023; Lyle et al., 2023; Dohare et al., 2024; Lee et al., 2024; Moalla et al., 2024; Chung et al., 2024; Kumar et al., 2025; Lyle et al., 2025).
(ii) Ensuring directed exploration for rapid adaptation to changing dynamies: Directed exploration determines the degree of exploration based on an estimated uncertainty of an unobserved state. In this way, the agent prioritizes underexplored, hence more informative, areas of the state-action space, thereby improving its sample efficiency. Directed exploration is instrumental in fast-changing non-stationary environments where the agent has limited time to adapt to each new condition (Kaufmann et al., 2012; Besbes et al., 2014; Zhao et al., 2020).
We hypothesize that both sustained plasticity and directed exploration can be achieved by quantifying the uncertainty around the value function. An agent equipped with a probabilistic value function will systematically reduce the uncertainty of its value predictions it collects more data. When confronted with change in environment dynamics, the value function output will make predictions with reduced confidence. The incrcased uncertainty will increase the critic training loss, thereby keeping the training process active. Furthermore, the probabilistic value predictor will make it possible to assign uncertainty estimates to the policy training objective, which can in turn be used as an exploration bomus to direct the policy search toward underexplored arcas of the state-action space.
Our Hypothesis: Equipping an agent with a mechanism to quantify the uncertainty of the value function enables it to (i) preserve plasticity and (t) explore effectively under non atn-staaionary dynamics.
Guided by the above hypothesis, we adopt Evidential Deep Learning (Sensoy et al., 2018) a well-suited framework for learning probabilistic value functions. Evidential deep learning suggests modeling the uncertainty of each data point by a Bayesian data-generating process where the hyperparameters of the prior distribution are determined by input-dependent functions. The likelihood and priors are chosen as conjugate pairs to keep the calculation of the data point-specific posterior and the marginal likelihood analytically tractable. The prior hyperparameter functions are modeled as deep neural net works, the parameters of which are learned by empirical Bayes Evidential approaches are observed to deliver high-quality uncortainty estimates in both regression (Amini et al., 2020) and classification (Kandemir et al., 2022) settings.
Figure 1 illustrates the learning profiles of on-policy deep actor-critics in continuous control task with non-stationary dynamics. Plain PPO, its recent extension to non-stationary environments (Moalla et al., 2024), and a state-of-the-ar variant equipped with directed exploration (Yang et al., 2024b) all lose their adaptation capability at early stages of training. Conversely, our evidential version and its extension to directod exploration quickly adapt to new tasks. We posit that our new method, called Evidential Proximal Policy Optimization (EPPO), brings such a performance boost as it fulfills both requirements of our hypothesis above. Our contributions are as follows:
2

---

Published in Transactions on Machine Learning Research (11/2025)
(i) We apply evidential deep learning for the first time to uncortainty-aware modcling of the value func- tion in on-policy deep actor-critic architecture. Our solution prescribes a hierarchical Bayesian generative process that maps state observations to hyperpriors.
paralyais-cheetah-ffoo--
4000
3000
Episode Return
2000
(ii) We evidential value learning to develop two methods for constructing probabilistic extension of the generalized advantage estimator (Schulman et al.,
1000
Plain PPO Noe-stationary PPO
PPO with Directed Exploration
2016). We demonstrated that performing directed ex- ploration based on the probabilistic advantage estima- tors brings a consistent performance improvement.
Exidntial PPO with Directed Exploration
0.0
0.5
1.0 Timesteps
1.5
2.0 x10
(ii) Due to the absence of a widely adopted bench- mark, we introduce two new experimental designs tai- lored to evaluate the adaptation capabilitics of contin- uous control agents to rapidly changing environment conditions. We benchmark our approach against two state-of-the-art PPO variants and observe that it out- performs them in the majority of cases.
Figure 1: PPO, its non-stationary extension, and PPO equipped with directed exploration all lose their adap- lation capability after million steps. In contrast, evidential PPO variants continue to improve, and di- rected exploration further enhances evidential PPO's performance. See Section B.4 for details.
2 Background
2.1
On-policy deep actor-critics
We define an infinite-horizon MDP as a tuple . 2 Prr, Po,7), 7); where S represents the state space and A denotes the action space. Let P be the state transition probability distribution such that s' (()(s,a) where 8 E and a E A. We assume a deterministie, lime-homogeneous reward function r : x A + R to facilitate presentation but without loss of generality. We denote the initial state distribution as so po(-) and the discount factor y E (0, 1). We consider non-stationary environments with time-dependent state transition probabilities, i.e., Pr(-Is, a) for time index i, where the non-stationarity arises from these changes. We consider stationary stochastic policies defined as T--ss). We use the following standard definitions of the action-value function Q²,, the value function V*, and the advantage function A*:
Σ S
Σ Extron AT(s), & (8t, as) V (st),
Q" (st, A Eee- Gr-1:00
V (st) A Es041:00 Gs:o0
where expectations are taken over trajectories induced by the policy w and r4-1 2 r(8 a(+1). The colon notation a b refers to the inclusive range (a, + b).... We denote by G. A Eo Tre the discounted sum of rewards.
We focus study on-policy deep actor-critic algorithms. We adopt PPO (Schulman et al., 2017) as the stato-of-the-art representative of the conservative policy iteration approaches (Kakade & Langford, 2002). This algorithm family has been adopted in real-world scenarios due to its relative robustness stemming from the conservative policy updates that promote slower but more stable training. Prime examples include the control of physical robotic platforms (Lopes et al., 2018; Melo Máximo, 2019) and fine-tuning large language models (Christiano et al., 2017: Bai et al., 2022; Touvron et al., 2023; Achiam et al., 2023; Zhong et al., 2023). PPO is policy gradient method that updates the poliey using a surrogate objective, ensuring that policy updates remain constrained to ensure average policy improvement (Schulman et al., 2015). We follow the established practice and adopt the clipped objective as the surrogate function. PPO updates its policy ne, parametrized by 0 E O:
Lelip (s,
min
Te(a|s) ToM (s, clip Totd (a|s)
Tota Tala 8) e, 1 +e)) ATou ATom(s,aa)]
(a(ss))
where ATou (s, a) is an estimate of the advantage function, and clip( xala|s) ratio within the range
[a]a]] 1- 1 1e) bounds the probability e, 1 +c] for 0. PPO approximates the value function with Vo parametrized by
3

---

Published in Transactions on Machine Learning Research (11/2025)
Φ E Φ. It uses the squared-error loss Lvr(Ø) Es, (Vo( G.) to learn Vg. The learned Vo is then used to compute advantage estimates, guiding policy updates for more stable and efficient learning.
Modern PPO implementations use Generalized Advantage Estimation (GAE) (Schulman et al., 2016), which is a technique for computing advantage estimates. This method helps reduce the variance in the return estimate while enabling step-wise updates via bootstrapping. GAE constructs the advantage function using a weighted sum of multi-step temporal-difference errors. Let the temporal-difference residual at time step t be 6, rr + (V4(+ - (s)).. The GAE estimate is defined as the exponentially weighted sum of temporal difference residuals:
A AA = (6A))6
(1)
where E [0, 1] is a hyperparameter that controls the bias-variance trade-off. GAE provides a flexible mechanism for estimating advantages, allowing reinforcement learning algorithms to achieve improved stability and faster convergence (Schulman al., 2015; 2017).
Loss of plasticity mitigation. Many deep reinforcement learning algorithms suffer from loss of plasticity under non-stationarity, attributed to several factors. Dohare et al. (2021; 2023; 2024) treat the loss of plasticity in PPO primarily as an optimization challenge. To address this, Dohare et al. (2021; 2024) propose a continual backpropagation method to counteract decaying plasticity, while Dohare et al. (2023) introduce a non-stationary adaptation of the Adam optimizer (Kingma & Ba, 2015), balancing exponential decay rates for moment estimates (Lyle et al., 2023). Another line of work employs regularization to preserve plasticity, such as constraining learned features to remain close to their initial values (Lyle al., 2022), penalizing parameter drift relative to initializations (Kumar et al., 2025), promoting weight-matrix orthogonality to stabilize learning (Chung et al., 2024), and discouraging excessive feature shifts during training (Moalla et al., 2024). Some approaches balance adaptation and retention by maintaining separate fast and slow learners, periodically resetting the fast learner to the slow one (Tee et al., 2024)- Other methods target specific causes, such as activation sparsity reducing gradients (Abbas et al., 2023), while others show that combining mechanisms like layer normalization and weight decay strengthens plasticity preservation (Lyle et al., 2025). Recent studies have also linked data diversity to plasticity preservation: Mayor et al. (2025) improve diversity through multiple parallel environments, and McLean et al. (2025) do through multi-task training. While these approaches rely optimizer adaptations, regularization, architectural modifications, data diversity to preserve plasticity, our method achieves both diverse data collection and plasticity preservation within a single probabilistic framework using evidential value learning. We compare our method with Dohare et al. (2024) and Moalla et al. (2024), which represent state-of-the-art solutions for loss of plasticity in PPO.
Directed exploration encourages agents to seek out novel or informative states. Prior work has improved the exploration schemes of PPO in both stationary and non-stationary settings (Burda et al., 2019; Wang et al., 2019; Zhang et al., 2022; Steinparz et. al., 2022; Yang et al., 2024b). PPO is also widely applied in continuous control, particularly in robotics (Schwarke et al., 2025; Mittal et al., 2024), where non-stationarity often arises due to dynamic euvironments and changing task requirements. To address this, Schwarke et al. (2023) extended random network distillation (Burda et al., 2019) to non-stationary robotic tasks, while Yang et al. (2024b) further advanced it with a distributional formulation. We compare against Yang et al. (2024b), which represents the state-of-the-art in directed exploration for PPO, with its non-distributional variant demonstrating strong performance in non-stationary robotics task (Schwarke el al., 2023).
Relation of non-stationary reinforcement learning to curriculum, meta-, and continual reinforce- ment learning. Continual learning aims to learn a single policy that maximizes the expected return in a stream of tasks (Khetarpal et al., 2022). This is a useful learning setup when the tasks have common properties. The challenge is to improve performance in each new task by exploiting these commonalities without causing a performance drop in the previously seen lasks, the phenomenon called calastrophic forgelling (Rusu et. al., 2016; Kirkpatrick et al., 2017; Traoré et al., 2019; Kaplanis et al., 2019). Non-stationary reinforcement learning addresses setups where some factors of the environment change slower than the step-level transition dynamics. In such setups, the agent is tasked to adapt to the recent change as quickly as possible and does not have to remember all the history of the changes. Instead, it uses all the model capacity to capture the
4

---

Published in Transactions on Machine Learning Research (11/2025)
trend of the change and rapidly adapt to it. Our setup can thus be viewed a special case of a continual reinforcement, learning problem with weaker assumptions than standard benchmarks (Wolezyk et al., 2021; Tomilin et al., 2023), such as the agent having no access to task identifiers, task-specific replay buffers, or network heads. These relaxed assumptions make our non-stationary setting an even more challenging instance of continual reinforcement learning. In curriculum learning, task order is typically chosen, either explicitly or implicitly, to promote transfer or accelerate mastery (Bengio et al., 2009). In our case, tasks change at short, fixed intervals, independent of success, difficulty, or skill acquisition. The sequences are structured but not pedagogically curated; they reflect arbitrary shifts in environment dynamics rather than progression intended to guide the learner. Meta-learning trains agents across task distributions for fast generalization at test time (Al-Shedivat et al., 2018; Berseth et al., 2021; Bing et al., 2023). Methods in these areas often stress retention by constraining updates, which can reduce plasticily. In non-stationary settings, however, plasticity is crucial for effective long- term adaptation with minimal regret. Despite its importance, non-stationary reinforcement learning has been less studied than meta- or continual learning (Khetarpal et al., 2022). Compared to continual learning formulations such as Dohare al. (2024), our setting differs in two ways: (i) changes occur at shorter intervals, requiring faster adaptation, and (fi) changes structured rather than random, enabling realistic strategies. In domains like robotics or control under changing conditions, retaining performance across all past dynamics is often infeasible; the main challenge is rapid adaptation to current conditions, highlighting non-stationary reinforcement, learning as a distinct and practical setting.
2.2 Evidential deep learning
Bayesian inference (Bishop, 2006; Gelman et al., 2013) infers a posterior distribution over model parameters from a given likelihood function evaluated on data and a prior distribution chosen without access to data. Evidential deep learning (Sensoy et al., 2018) applies the classical Bayesian framework a particular way where posteriors are fit data-speeific random variables from data-specific prior distributions, the parameters of which are amortized by input observations. The amortized prior and the likelihood are chosen from conjugate families to ensure analytically tractable computation of the posterior and the marginal likelihood, the latter of which is used as training objective." See Ulmer et al. (2023) for a recent survey paper introducing the concept in greater detail.
We build our solution on Amini et al. (2020)'s adaptation of the cvidential framework to regression problems, as typical continuous control task has real-valued reward functions. Amini al. (2020)'s approach assumes that the output label y corresponding to an input observation æ follows a normally distributed likelihood with mean μ and variance o². This distribution is assigned a Normal Inverse- Gamma (NIG) distributed evidential prior:
(u. ( ~ NIG (H, o (()) = =N (nlw(z), (f v((x) Inngam(o? ((()) B(x)),
where the hyperparameters w, 12 a. B are modeled ieput-dependent functions, specifically neural networks with weights Φb Throughout the paper, suppress the dependency of the variables on o and x for notational clarity, c.g,, w = w (x), and refer to them jointly as the evidential parameters m - mm = (w, M, a, 8). Due to its conjugacy with the normal likelihood p(yla, o²) = (N(yl,, o²), the posterior p(u, o²ly, m) and the marginal likelihood p(y)m) are analytically tractable. This marginal is the well-known Student-t distribution:
y/m St. (w)
w 3(1 B(1+v) +v) 20x
The parameters of this distribution can be fit by minimizing the negative logarithm of the marginal likelihood
CNLL log (m) -a log (I)) + (a+ -) log (w-
w) w) log
(T((TT))
[(a+ ((a+t))
[(a)
(2)
where Ω = 23 (1 11+1) v) and r(-) is the Gamma function. See Section A for the derivation of the posterior
distribution. A Marginal likelihood optimization also known as Type If Mozimuum Lskeldbood or Emptrioal Bayes (Elron, 2012).
5

---

Published in Transactions on Machine Learning Research (11/2025)
Evidential deep learning in deep reinforcement learning. Evidential deep learning has been tensively used in numerous machine learning frameworks and practical tasks (Gao et al., 2024). It has also been integrated into deep reinforcement learning for recommendation systems to provide uncertainty- aware recommendations (Wang et al., 2024), modcling policy network uncertainty to guide midence-basee exploration behavioral analysis (Wang et al., 2023), incorporating uncertainty measures as rewards for decision-making in opinion inference tasks (Zhao et al., 2019), and calibrating prediction risk in safety-critical vision tasks through fine-grained reward optimization (Yang et al., 2024a). However, we instead it to model uncertainty in value function estimates, which enables confidence- based exploration and helps preserve the plasticity of the neural network.
3 Method
We present method that adapts the evidential approach to learn a distribution over the value function V(s)). The inferred distribution induces a corresponding distribution over the GAE, which enables the model to detect distributional shifts resulting from the non-stationarity of the dynamics and to guide directed exploration, thereby promoting rapid adaptation.
3.1 Evidential value learning
We assumne our value function estimates (st) to normally distributed with unknown mean μ and variance o²; which are jointly WIG dig--dittiiuuedd We shorten the notation V/2 = Vs,,) when the relation clear from context. Although evidential deep learning has demonstrated promising results in epistemic uncertainty estimation, including for unscen out-of-distribution data, naïvely following Amini et al. (2020)'s method often results in training instabilities even on standard supervised regression tasks (Oh & Shin, 2022; Meinert el al., 2023). Prior work relied on non non-Bayesian heuristics to solve these instabilities. We are the first to introduce a fully Bayesian hierarchical design to solve this which we summarize in the plate diagram in Figure 2. Introducing hyperpriors on each of the four evidential param- eters, our model is:
(ss) N (w(s)l (o.))
9
v(s) Gam (v(s)
a(s) Gam (a(s))a (2),
((8) Gam (B(8) (g),
8² In (o (2)) B(s))
2lo² N (ulw(s), v())) 02)))
Vln,o ~N(V),,0²)
Figure 2: Plate diagram of our evidential value learning model.
where Gam(-) is the Gamma distribution, and 1. p....., 33 are fixed hyperparameters. We adopt fixod set of hyperpriors to provide relatively flat and uninformative priors for all experiments. See Table 12 in the Appendix for further details. Following our notational convention, suppress the dependency 8, e.g., = w(s), and combine the evidential parameters into m = (w, a, Marginalizing over (H, yields
(V, = / (VVl,,o?)nnn (m)),,, =Man))=((mmmii) pV(mm))(m)
where p(| ([))) is a Student-t distribution parameterized as in Section 2.2. The hyperprior p(m) acts regularizer in the log-joint objective. The training objective of evidential value learning is regularization coefficient.
C(m) = (m) ((g)(mm, where
2 AAs deterministic transformation of the state s, the notation w(s) A((-) implies that its parameters are random variables such that w(s) is normally distributed.

---

Published in Transactions on Machine Learning Research (11/2025)
The mean and variance of the state-value function output V can be computed analytically as
Evim [V] )m [Eviu [V] 2) [x] w,
and
vaIv/m pm [V] = (( varryuu.0
10,0² [V]] + var(u [vino [V]
[o-] + (a [ec]
B + v(a a -1-1)==-((111)))
(11
3
= -
-
where we assume a > 1.3 The first equality follows from the law of total variance, which splits the marginal variance into aleatoric and epistemic uncertainty components. Reliance on vargim [y] therefore provides us with a principled way of incorporating irreducible uncertainty inherent in the environmental structure and reducible uncertainty due to improvable approximation errors in EPPO.
Distributional reinforcement learning. Evidential value learning belongs to broader research field that incorporates distributional information into reinforcement learning models, which can be roughly divided into two sub-fields. The first aims to account for aleatoric uncertainty in the MDP caused by the inherent stochasticity of the environment. returns G., e.g.; to infer risk-averse policies (Keramati et al., 2020). See Bellemare et al. (2023) for a recent textbook introduction. The second focuses on accounting for sccond-degree epistemic uncertainty inherent in value function inference, usually relying methods from Bayesian inference (Ghavamzadeh et al., 2015; Luis et al., 2024), e.g., to use it guide for exploration (e.g., Deisenroth & Rasmuussn,, 2011; Osband et al., 2019). Evidential value learning differs from standard distributional reinforcement learning approaches in several key ways. While methods in the first category focus solely on modeling alcatoric uncertainty for risk-sensitive, usually risk-averse, control, evidential value learning is related to the second area of research. It uses an evidential model over the value function to induce distribution over an advantage function that simultancously incorporates both alcatoric and epistemic uncertainty. To make this fcasible requires the assumption of parameterized densily model instead of the model-free particle-based distribution approximation used in distributional reinforcement learning. This dual uncertainty quantification enables both regularization benefits and optimistic exploration strategies, distinguishing it from approaches that target only adaptive or risk-sensitive settings through aleatoric uncertainty modeling alone.
focuses on accurately modeling the resulting distribution over the
3.2 Directed exploration via probabilistic advantages
Evidential value learning provides an uncertainty quantifier for the value function that detects shifts in the data distribution caused by non-stationary state transition dynamics. By parameterizing the value function as a distribution p(((m) rather than as a point estimate, our model naturally increases uncertainty regions of distributional shift, thereby maintaining gradient flows and preserving plasticity. This uncertainty propagates through the advantage calculation, turning the generalized advantage estimator ÂGAE into a random variable. An Upper Confidence Bound (UCB) exploration strategy emerges naturally from this probabilistic treatment, drawing theoretical justification from the exploration-exxloitatoo trade-off in multi-armed bandit theory ( (Auer et al., 2002). The UCB estimator
A =
E [A
AGAE
+ +
provides an optimistic estimator that balances expected advantage with uncertainty, where k > 0 controls the confidence radius. This ensures that the policy is directed towards state-action regions where the value function exhibits high uncertainty, enabling duriosity-driven exploration with firm theoretical grounding.
The mean estimate for GAE is
E B[AGAR AGAE
Σ
(7)))'
We enforce this condition by adding one to the neural network's output.
7

---

Published in Transactions on Machine Learning Research (11/2025)
and remains tractable due to the linearity of expectations and because the mean of the temporal difference E. [6s] = + +EE [V++1] F [V]] is tractable. We propose two variants of EPPO that differ in how the variance term var [ in Equation (3) is computed.
(EPPOcor) Exploration via correlated uncertainties. We derive the variance of Ae by focusing on its definition as the exponentially weighted average of the k-step estimators A = V₁ + - + > 2 7're+t- Because the rewards are deterministic in our setup and therefore have zero variance, we combine them into a generic constant term and obtain
AGAE - (1-
Σ
x 1-1 (1 (1- 11
Σ 1=0
X Σ N2-
const
1=1
1=1
=
(7A)
const.
1=1
Given the assumed conditional independence of the states, the resulting variance is
var
ACAAE]
var [V]] +
var [V++]..
(4)
We use this variance to construct the UCB in Equation (3) and refer it EPPO,or in the experiments.
(E Exploration via uncorrelated uncertainties. We also consider the case where the k-step estimators 1(() are assumed to be independent of each other. We then construct the overall variance as the exponentially weighted sum of the individual k-step estimators. It can be shown casily (sce Section A.2) that the resulting variance approximation is
var AGAE
1 1+ var [7]
1
(7 var [V++]]
(5)
1=1
i.c., the influence of the current value variance is down-scalod by a factor (1- (1-A)/(1+3)) A) relative to the future time steps in EPPOid compared with EPPO This adjustment makes EPPO.na more far-sighted for the same We use the variance estimate in Equation (5) to construct the UCB in Equation (3).
Practical implementation of EPPO. We provide pseudocode in Algorithm 1 illustrating how to implement EPPO variants by overlaying color-coded modifications on top of standard PPO implementatoon, where each color corresponds to a specific EPPO variant. As shown, EPPÒ variants require only minimal changes to PPO with a clipped objective and GAE-based advantage calculation. The key additions include an evidential value estimator, its update rule, and probabilistic advantage computation with UCB, All other components remain identical to those in PPO.
4 Experiments
We design experiments to benchmark EPPO variants against state-of-the-art on-policy deep actor-critic algorithms in non-stationary continuous control environments. To amplify the effect of non-stationarity on model performance, we define tasks over short time intervals and introduce changes in the environment. dynamics. In each interval, agents are required to detect the change, explore effectively, and adapt rapidly maximize the overall return during learning. We focus on non-stationaritics that affect environment dynamics in a structured manner, based on identifiable patterns of change, and exclude scenarios where changes occur randomly. This design ensures that observed performance improvements stem from enhanced learning capabilities rather than increased robustness to noise. We run our simulations on the Ant and Hal HalfCheetan environments using the 'v5' versions of MuJoCo environments (Todorov et al., 2012). For further details on the experimental pipeline and hyperparameters, see Section B, The implementation of the EPPO variants and the full experimental pipcline is available at https: /ttp::///iii thub. com/ cooo/addig
8

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: Performance evaluation on slippery environments. Area Under the Learning Curve (AULC) and Final Return (meanise) scores are averaged over 15 repetitions. The highest averages are highlighted in bold, and results that are not significantly different from them (see Section B.2) are underlined. The average score represents the mean across all environments, while the average ranking is based on the mean scores.
decreasing
increasing
Average
Metric
Model
Ant
Hhheeah
Ant
#Cheeth
Score
Rank ing
PPO
2355
2495:201
2237:224
2536=207
2406
5.3
РЕO
2522=109
2800=189
2485=90
1809=450
2279
5.0
2185+663
1519#183
2288+118
1833=150
1956
6.5
AULC (1)
PPODRND
2475
2633 180
2307+:348
2021
2359
4.5
EPP mean
2504=127
2432:299
2875=77
2822=219
2658
3.5
EPPO Done
2561 126
2699 256
10
2962
1.5
EPP Cid
2614+138
2866=218
2779=86
3374=220
2908
1.8
PPO
2357:2230
2483
2341::2700
2720 =310
2475
5.5
PFO
2613=110
2346=214
2620=29
1906=442
2371
5.3
CB
2303#772
1527=193
2398+116
1920=172
2037
6.5
FINAL RETURN (t)
PPODRND
2583:1141
2672
2/28
2115
2449
4.5
EFFO meean
2660=181
2522+333
3002=92
2978=227
2790
3.0
EPP
EpP Door
2714#
2821 4:74
3071188
3872#248
3120
1.5
EPP Oind
2741:115
2970
2941 89
3559
3053
1.8
Baselines. To validate our hypothesis that uncerlainly-aware value estimation enhances both plasticity preservation and directed exploration, we benchmark against four reprosentative basclines, summarized in Table 2. These baselines are selected to isolate and test the contribution of each hypothesis component. (i) PPO (Schulman et al., 2017): A widely used on-policy deep actor-critic reinforcement learning algorithm that serves as the foundation for EPPO. We follow the most recent implementation practices to repre- sent the state of the art. In particular, we use GAE (Schulman et al., 2016) to estimate value function targets. (ti) PFO (Moalla et al., 2024): A recent PPO variant that addresses the plasticity problem under non-stationarity by extending the trust region constraint to the feature space. (tö) CB (Dohare al., 2024): A PPO variant for continual learning that mitigates plasticity loss through continual backpropagation, where fraction of less-used units are randomly reinitialized. (iv) РPOμии (Yang et al., 2024b): A state-of-the-art PPO variant designed for directed exploration using random network distillation, where the distillation signal acts as a pseudo-count to generate intrinsic rewards that guide exploration. We also evaluate the EPPO variant with x = 0, which performs evidential value learning without directed exploration. We refer this model EPPO Its relative performance highlights the contribution of directed exploration. We use the recommended hyperparameters for the baselines from the original papers, except for CB, where we conduct grid search due to its high sensitivity to two hyperparameters reported in the original work. We provide the full details in Section B.3.
Table 2:
Plasticity and exploration. Comparison of baselines highlighting the presence of plasticity preservation mechanisms and directed exploration.
Plasticity Mcchanism
Directed Exploration
Model
PPO
X
PFO
CB
PPOPANt
EPPOpean
EPPOcar
EPPO
Experimental setup. We propose two experimental setups to assess the ability of the models to adapt to non-stationarity. In both setups, encourage fast adaptation by limiting task durations to short intervals. We also preserve agent learnability by introducing changes gradually and avoiding abrupt transitions. The sctups are as follows:
(i) Slippery environments. Inspired by Dohare et al. (2021; 2024). we construct non-stationary envi- ronments by varying the friction coefficient of the floor in locomotion tasks using the Ant and Ha fCheetah
9

---

Published in Transactions on Machine Learning Research (11/2025)
Table 3: Performance evaluation on paralysis enwironments. Area Under the Learning Curve (AULC) and Final Return (meanise) scores are averaged over 15 repetitions. The highest averages are highlighted in bold, and results that are not significantly different from them (see Section B.2) are underlined. The average score represents the mean across all environments, while the average ranking is based on the mean scores.
Model
Metric
Enviromment
Strategy
PPG
PFO
CR
PPODAND
EP EP PPmon
E.POcor
RP POud
back-one
2009
2259
2298
2562+124
2455=
2008+129
2724:174
Front-ene
2054:2200
2098
2035
2547
2407=992
2749:112
274
back-twa
192
2136+57
1808
2226
2203
2099
2088=94
Ant
front=two
1975
2000
2023
2144::00
2259
2294:993
2275=76
parallel
2162#1176
2298
1918:122
2358
2350==108
2348+127
2558:159
AULC (*)
cro9s
1898:186
2161+71
18:0
2012+90
2167=991
2197
2281
Average
2004
2159
1988
2308
2307
2383
2445
back-one
2444g223
2181:998
2258
2131
3160=22
35022473
3515:131
front-one
2076,1209
2485
2151
25921331
3384=227
3558
36951241
HalfCheetah
cross-vi
2311::25
2314
2255
2487
3002
3205-1244
3120
cress-v2
2477:2200
1903+245
2318+96
2371 2321
3039=195 195
3250=195
3283:212
Average
2327
2221
2246
2395
3146
3379
3403
Overall Average
2133
2184
2091
2343
26-13
2781
2828
Overall Average Ranking
5.2
3.8
3.1
back-one
2261=3326
2503+114
2478:1126
2784+147
2709
1811
2977:160
Iront-one
2253+2244
2337 #:7
2130
2802+122
26:19
3020=107
2956++185
back-twe
2188.:206
2454.cc2
2103
2512:991
2533
24000
2327
Ane
front-two
2282:119
224
2310
2425
2536
2633:97
2605
parallel
2397
2601496
2118:117
2647
26-19=114
2653+1144
2883
FINAL RETURN (1)
cross
2144
2467 2:79
2043180
23101104
2495
2500
2570
Average
2254
2435
2197
2580
2505
2683
2720
back-one
2504g200
2275::300
2344:152
2204
3320=287
3696
3718:
front-cne
2115:2305
2577::99
2184g1
2625.1302
35400225
3724
3892
Hal @Cheetah
cross-vi
2405=221
2349
2333
2648++246
3159=254
3420#=281
3341
cross-v2
2550==2a6
1953+226
2434=106
2511+:250
3217-204
3450=2288
3468 2228
Average
2394
2288
2324
2497
3309
3573
3605
Overall Average Overall Average Ranking
2310
2376
2248
2547
2881
3039
3074
5.8
5.3
6.4
3.9
3.9
1.9
1.7
environments. We induce non-stationarity in them by changing friction every 000 steps. To create challenging task changes, we implement two strategies: decreasing, where friction starts at its maximum value and gradually decreases, and increasing, where friction starts at its minimum value and gradually increases. This setup ensures that agents encounter non-stationarity in both increasing and decreasing friction scenarios. The minimum friction is set to 0.5 and the maximum to 4.0, based on the feasibility of solving the exsks-extreme friction values may make movement too difficult due to slipping or an inability to move forward. We define 15 tasks by changing the friction with a positive or negative offset of 0.25.
(ti) Paralysis environments. We design a new set of non-stationarity experiments by dynamically altering the torque capabilities of the leg joints in the Ant and halfCheetah environments, inspired by Al-Shedivat al. (2018). Each experiment involves paralyzing different joints to diversify the control tasks across experiments. We generate six torque modification schemes for Ant and four for Hal faafehheeah, In each scheme, we select specific joints and progressively reduce their torque capability until they become fully paralyzed. Then, gradually restore their functionality, returning to the fully operational state. This yields sequence of nine tasks, where each joint either loses or regains 25% of its torque capacity in each step, following the pattern: [100, 75, 50, 25, 0, 25, 50, 75, 100).
Evaluation metrics. We assess model performance using Iwo metrics: (i) Area Under the Leurning Curve (AULC) and (it) Final Return. AULC is the average return collected over the entire training trajectory. It captures not only the agent's final performance but also how quickly and consistently it improves throughout. training. In non-stationary environments, where the task dynamics change over time, AULC reflects the agent's ability to continually adapt to new conditions and recover from changes. A higher score indicates
10

---

Published in Transactions on Machine Learning Research (11/2025)
slippery
Mean
Median o Outliers
Effective Rank
Stable Rank
Dormant Units (%)
C8
PRO
EPPO
Ed
20
30
50
100
150
200
20
40
paralysis
Mean
Median
Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPO
PFO
CB-
PRO
EPPO
10
20
30
40
50
100
150
200
10
20
30
40
Figure 3: Plasticity preservation analysis using critic network metrics. We evaluate three metries: effective rank, stable rank, and dormant unit percentage, shown from left to right. The top row shows results from the slippery environments, and the bottom row shows results from the paralysis environments. Each box plot summarizes the distribution of the respective metric across training seeds: the red line indicates the mean, the black line indicates the modian, and the individual points represent outliers. These metries quantify the prediction capacity of the critic networks learning progresses. EPPO variants consistently preserve plasticity better than PPO variants, as shown by higher ranks and lower dormant unit percentages.
stronger overall adaptation and learning stability across the full training horizon. Final Return is calculated at the completion of each individual task by averaging the returns during the last evaluation steps of that task. averaged over all tasks at the end of training. This metric mcasurcs how well the agent adapts and performs on individual tasks after it has observed and interacted with them. A higher final return suggests more effective task-specific adaptation. Together, these metrics assess distinct but complementary aspects of adaptation: AULC evaluates an agent's learning efficiency and dynamic adaptability to changing environments over time, while final return assesses the agent's 's stable performance after task-specific adaptation has occurred. We use statistical tests to identify significant. improvements; see Section B.2 for details.
These evaluations
4.1 Results and Discussion
We present the detailed results of the experiments Table 1 and Table We also provide experimental result visualizations in Section B.4, illustrating episode returns throughout the changing tasks and demonstrating both quantitative and qualitative performance differences between EPPO variants and our baselines.
Plasticity preservation analysis. We analyze the plasticity of the agents' critic networks using three metrics: (i) Effective rank (Roy & Vetterli, 2007) quantifies the number of significant dimensions in the feature matrix. A high value indicates that most matrix dimensions contribute, suggesting that the network generates diverse features and maintains plasticity. (ii) Stable rank (Yang et al., 2020) measures the effective dimensionality of the feature matrix. A low rank suggests limited diversity in the learned representations, implying that the network struggles to preserve plasticity. (iii) Dormant wnil percenlage (Sokar el. al., 2023) refers to the proportion of inactive neurons. A high percentage suggests impaired gradient flow and reduced learning capacity, indicating loss of plasticity. We compute these motrics at the end of cach task and report the average over the entire training process. Figure summarizes the results across both experimental setups. As shown, EPPO variants consistently achieve higher effective and stable ranks and exhibit fewer dormant.
11

---

Published in Transactions on Machine Learning Research (11/2025)
units than plain PPO and its recent variants for handling non-stationarity and directed exploration. Pairwise t-tests on these plasticity metries confirm that EPPO and its variants significantly outperform plain PPO (p < 0.05), indicating statistical significance; see Section B.2 for details. These findings indicate that the evidential value learning framework helps preserve plasticity and that the addition of directed exploration does not compromise relative to plain PPO. We provide a full analysis in Section B.1.3.
Plasticity preservation through evidential value lcarning. EPPO preserves plasticity by making the loss (Equation (2)) geometry (i) smoother (fewer extrema per unit area) and (ti) more convex (less saddle points), which facilitates gradient-base training. Table 4 decomposes the evidential learning objective (see Equation (2)) and reports the sign of the derivative of each loss component with respect to the evidential parameters.
Let A = ly - w| (absolute value for simplicity) denote the value function approximation error. The derivatives of the loss term that includes the value function approximation error (see the third row in Table 4) with respect to the evidential parameters have negative signs, decreasing the evidential parameters except w which is driven toward the target y. Conversely, the derivatives of the other loss terms with respect to the evidential parameters have positive signs, increasing the evidential paramcters. As a result, when the value function approximation error drives the evidential parameters downward, the other loss lerms penalize this change, balancing the tendencies and constraining the solution in the function space. This regulatory effect smooths the loss goometry.
For gradient-based method to perform well, the loss geometry should exhibit similar con- vexity in all dimensions; in other words, the eigenvalues of its Hessian should be close to each other (Belkin, 2021). We examine the gradient of the term that includes the value function approximation error with respect to the neural network parameters, compared with the mcan squared error as in plain PPO and other baselines. The gradient term for PPO is 2ow, where f denotes the network parameters. In contrast, the derivative of Equation (2), considering only the value func- tion approximation error, (2A Vow where (a+
Table 4: Sign of loss derivalives with respect to evidential parameters. For cach loss term listed in the rows (dccomposod from Equation (2)), the table reports the sign of the derivative of the loss with respect to each evidential parameter. '+' and '_' denote positive and negative signs of derivatives, respectivcly, while '0' indicates no dependency. All evidential- related terms have positive derivatives except in the third row, which includes the value function approximation error term ((y w))), highlighting their role regularizers.
y
wl
@
B
1 log
0
+
0
0
2
a log (23 (1 +1))
0
+
+
+
(o + 2) log (Iv w)² v + 2 (1+11) v))
<- D4 + 23(1 + D)
In effect, ( adaptively rescales the gradient, reducing the step size when the value function approximation error A is large. This dynamic scaling functions as an effective adaptive learning rate and limits excessively large updates that could move the weights into regions with inactive gradients, which may otherwise create dormant units.
-
log
I(a)
+
r(a+
As shown in Figures& 8 to 9, crities trained with evidential value learning maintain lower gradient norms, higher effective and stable ranks, and fewer dormant units compared to crities trained with mean squared error (plain PPO and PPODRND). In these baselines, increasing gradient norms causes plasticity loss, leading to lower ranks and more dormant units. Since all models use layer normalization and gradient clipping, as is common in the plasticity-preservinn literature (Lyle et al., 2023; 2025), these factors are controlled. Therefore, EPPO's improvements in plasticity are due solely to its evidential value learning loss.
Exploration analysis. We evaluate the exploration ability of the agents using the percentage coverage metric (Mayor et al., 2025), which quantifies the spatial dispersion of state representations. Specifically, we project the representations into 2D space and compute the proportion of occupied cells in a uniform grid over this projection. A higher coverage indicates better exploration, meaning that the agent visits a broader portion of the state space. We evaluate this metric for plain PPO, PPODRND, which serves as the
12

---

Published in Transactions on Machine Learning Research (11/2025)
Table 5: Exploration analysis. Percentage coverage (mean=se) scores are averaged over 15 repetitions. The highest mean values are highlighted in bold and underlined if they fall within one standard error of the best score.
PFO
PPOpaND
EPPOmeen
EPPOcos
EPPOLnd
part parallyaa-ccheetal-aaa---om
7.51 + 0.70
8.12 + 0.62
8.90 + 0.74
9.27 + 0.52
9.34 + 0.68
3.18 + 1.21
5.26 + 1.45
12.46 + 0.34
12.61 + 0.30
12.72 + 0.29
strongest baseline for directed exploration, and the EPPO variants on the paralysis ant environment with the parallel strategy and the paralysis cheetah environment with the back-one strategy. Table 5 reports the percentage coverage results. The EPPO variants consistently achieve higher coverage than both plain PPO and PPODRND, demonstrating their enhanced ability to explore the state space. non-stationary environments, agents must gather data from diverse and high-reward regions of the state distribution and contimually adapt to the newly visited states through plasticity preservation. Notably, PPODRND increases the coverage of plain PPO but does not improve plasticity preservation (see Figure 3), and its adaptation performance in terms of AULC and Final Return remains inferior to that of EPPO variants. The strength of evidential value learning lies in its ability to jointly achieve plasticity preservation and directed exploration within a single probabilistic learning framework. We provide further details in Section B.1.4.
Discussion. Our experimental findings are as follows:
(1) Evidential value learning helps preserve plasticity. EPPO variants yield higher effective and stable ranks while creating fewer dormant units compared to plain PPO, thereby better preserving the plasticity of the critic networks. This capacity to retain plasticity enables EPPO variants to contimue adapting to changing environment dynamics.
(ii) Directed exploration boosts performance. EPPO variants with directed exploration (EPPOcor and EPPO outperform the baselines across all metrics. They also surpass EPPO which uses the mean value function for policy improvement. These results highlight the unique contribution of directed exploration to performance. Notably, the addition of directed exploration also improves the performance of plain PPO, as demonstrated by PPODRND.
(ii) Plasticity alone or directed exploration alone is insufficient to overcome non-stationarity, Basclines focusing solely on preserving plasticily (PFO, CB) or solely on directed exploration (PPODRND) yield partial improvements but fail to adapt contiunially. This indicates that both properties are necessary for sustained performance.
(w)) Evidential value learning with directed exploration accelerates convergence and improves training stability while preserving plasticily. EPPO variants achieve superior task adaptation compared to the baselines, as demonstrated by the final return scores and learning curves. They also converge more rapidly and improve training stability, as supported by higher AULC scores. By modeling value uncertainty, evidential value learning maintains plasticity throughout training.
(v) Equipping an agent with a mechanism to quantify the uncertainty of the value function enables it to preserve plasticity and explore effectively in the face of non-stationary dynamics. Our best-performing algorithms, which incorporate uncertainty quantification into the value function, allow agents to maintain plasticity and conduct directed exploration. This facilitates rapid and continual adaptation non-stationary environments, supporting our key hypothesis.
Compute time. We perform our experiments using two computers equipped with GeForce RTX 4090 GPUs, an Intel(R) Core(TM) 17-14700K CPU running at 5.6 GHz, and 96 GB of memory. Our experiments are conducted on these two machines with four parallel sceds. We mcasure approximately the total wall-clock time for the computation of 15 seeds across all environments at 74.8 hours for PPO, 75 hours for PFO, 78.8 hours for CB, 78.1 hours for PPOPRND, 75.3 hours for EPPOmean: 75.4 hours for EPPO cory and 75.6 hours
13

---

Published in Transactions on Machine Learning Research (11/2025)
for EPPOin- The total execution time for all experiments reported in this work is approximately 533 hours, equivalent to 22.2 days on two GPU-supported workstations.
5 Limitations and broader impact
We observe EPPO to be sensitive to the choice of some hyperparameters, such as the regularization coefficient & and the confidence radius k. While this is a common weakness most deep reinforcement learning algorithms, the effect of the resulting brittleness may be larger non-stationary environments. We expect that choosing the confidence radius based on a generalization bound, as practiced commonly in bandit research (Li et al., 2010; Srinivas et al., 2010; Kaufmann et al., 2012; Lattimore & Szepesvári, 2020) and increasing the Bayesian modeling hierarchy will make EPPO more robust to hyperparameters. As on-policy policy- gradient algorithm, EPPO shares similar thcoretical properties with other PPO variants. The effect of the evidential learning extension on non-asymplolic convergence is a challenging problem; hence, it requires special investigation. Although our study demonstrates that evidential value learning improves the control of non-stationary systems, we did not investigate whether the quantified uncertaintics are calibratod and how strong the correlation between their calibration and performance. We leave this interesting problem to separate study. Our results limited to rigid-body locomotors of single physics engine, despite covering comprehensive variations of challenging scenarios at non-stationarity levels exceeding those of prior studies. We do not expect extending our results to more tasks to bring any additional insights. We view testing our approach on physical robotic systems as the natural next step.
Continuous control of a non-stationary environment the core problem of building an agentic system on a physical platform. Non-stationarity the essential element of developing co-adaptive environments where robots and humans learn via bilateral feedback. Such co-adaptation is crucial to ensure human-centric growth of the capabilitios of agentic systems of the future. Our work contributes to the responsible AI initiative by facilitating the application of the powerful PPO algorithm to co-adaptive system development.
Acknowledgments
AA, GB, and MH thank the Carlsberg Foundation for supporting their research under grant number CF21- 0250. We also Ihank all reviewers and the area chair for improving the quality of our work through their thorough reviews and active discussion during the rebuttal phase.
References
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plasticity continual deep reinforcement learning. In Proceedings of The 2nd Conference on Lifelong Learning Agents, 2023.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. ar preprint 3XXi::3303.0877, 2023.
Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, 1lya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In International Conference Learning Representations, 2018.
Alexander Amini, Wilko Schwarting, Ava Solcimany, and Danicla. Rus. Deep evidential regression. In Advances in Neural Information Processing Systems, 2020.
Peter Auer, Nicolo Cesa- DiancBianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 2002.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. 1rxiv:6077 1607006450 2016.
arxiv preprint
14

---

Published in Transactions on Machine Learning Research (11/2025)
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint rrXit:2204.05622 2022.
Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica, 2021.
Marc G Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. MIT Press, 2023.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning, 2009.
Glen Berseth, Zhiwei Zhang, Grace Zhang, Chelsoa Finn, and Sergey Levine. CoMPS: Continual meta policy search. arviv preprint Arxi::2112. 04467, 2021.
Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed bandd problem with non-stationary rewards. In Advances in Neural Information Processing Systems, 2014.
Zhenshan Bing, David Lerch, Kai Huang, and Alois Knoll. Meta-reinforcement learning in non-stationary and dynamic environments. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 2023.
Christopher Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 2017.
Wesley Chung, Lynn Cherif, Doina Precup, and David Meger. Parseval regularization for continual reinforce- ment learning. In Advances in Newral Information Processing Systems, 2024.
Mare Deisenroth and Carl E Rasmussen. PILCO: model-based and data-efficient approach to policy search. In International Conference on Machine Learning, 2011.
Shibhansh Dohare, Richard S Sutton, and Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. ar preprint orXiv:2108.0632 2021.
Shibhansh Dohare, Qingfeng Lan, and A. Rupam Mahmood. Overcoming policy collapse in deep reinforcement learning. In Sizteenth European Workshop on Reinforcement Learning, 2023.
Shibhansh Dohare, J Fernando Garrande Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 2024.
Bradley Efron. Large-scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction. Cambridge University Press, 2012.
Junyu Gao, Mengyuan Chen, Liangyu Xiang, and Changsheng Xu. A comprchensive survey on evidential deep learning and its applications. arXiv preprinl arXiv:2409.044 2024.
Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. Bayesian Data Analysis. Chapman and Hall/CRC, 2013.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pincau, Aviv Tamar, et al. Bayesian reinforcement learning: A survey. Foundalions and Trendot in Machine Learning, 2015.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Interna- tional Conference Machine Learning, 2002.
Melih Kandemir, Abdullah Akgül, Manuel Haussmann, and Gozde Unal. Evidential Turing processes. In International Conference on Learning Representations, 2022.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Policy consolidation for continual reinforcement learning. In International Conference. on Machine Learning, 2019.
Emilie Kaufmann, Olivier Cappe, and Aurelien Garivier. On Bayesian upper confidence bounds for bandit problems. In International Conference on Artificial Intelligence and Stafistics, 2012.
Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning a CVaR policy. Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 2022.
Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Bariinkk,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regencrative regularization. In Proceedings of The 3rd Conference on Lifelong Learning Agents, 2025.
Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with Hare and Tortoise networks. In International Conference on Machine Learning, 2024.
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, 2010.
Guilherme Cano Lopes, Murillo Ferreira, Alexandre da Silva Simões, and Esther Luna Colombini. Intelligent control of a quadrotor with proximal policy optimization reinforcement learning. In 2018 Latin American Robotic Symposium, 2018 Brazilian Symposium on Robotics (SBR) and 2018 Workshop on Robotics in Education (WRE), 2018.
Carlos E. Luis, Alessandro G. Bottero, Julia Vinogradska, Felix Berkenkamp, and Jan Peters. Value- distributional mosel-based reinforcement learning. Journal of Machine Learning Research, 2024.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement. learning. In International Conference Learning Representations, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Under- standing plasticity neural networks. In International Conference on Machine Learning, 2023.
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Discntangling the causes of plasticity loss in noural networks. Proceedings of The 3rd Conference on Lifelong Learning Agents, 2025.
Walter Mayor, Johan Obando-Ceron, Aaron Courville, and Pablo Samuel Castro. The impact of on-policy parallelized data collection on deep reinforcement learning networks. In International Conference on Machine Learning, 2025.
Reginald McLean, Evangelos Chatzaroulas, J K Terry, Isaac Woungang, Nariman Farsad, and Pablo Samuel Castro. Multi-task reinforcement learning enables parameter scaling, Reinforcement Learning Journal, 2025.
Nis Meinert, Jakob Gawlikowski, and Alexander Lavin. The unreasonable effectiveness of deep evidential regression. In Proceedings of the AAAT Conference on Artificial Intelligence, 2023.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Luckeciano Carvalho Melo and Marcos Ricardo Omena Albuquerque Máximo. Learning humanoid robot running skills through proximal policy optimization. In 2019 Lalin American robolics symposium RS), 2019 Brozilian symposium on robotics (SBR) and 2019 workshop on robotics in education (WRE), 2019.
Mayank Mittal, Nikita Rudin, Victor Klemm, Arthur Allshire, and Marco Hutter. Symmetry considerations for learning task symmetric robot policies. In 2024 IEEE International Conference on Robolics ond Automation (ICRA), 2024.
Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcchre. No representation, no trust: Connecting representation, collapse, and trust issues in PPO. In Advances in Neural Information Processing Systems, 2024.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted Boltzmann machines. In International Conference on Machine Learning. 2010.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, 2022.
Dongpin Oh and Bonggun Shin. Improving evidential deep learning via multi-task learning. Proceedings of the AAAI Conference on Arlificial Intelligence, 36(7), 2022.
Ian Osband, Benjamin Van Roy, Daniel J Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research, 2019.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference. IEEE, 2007.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 04671, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continu- ous control using generalized advantage estimation. International Conference on Learning Representations, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint arXiv:1707.06347, 2017.
Clemens Schwarke, Victor Klemm, Matthijs van der Boon, Marko Bjelonic, and Marco Hutter. Curiosity- driven learning of joint locomotion and manipulation tasks. Tn Proceedings of The 7th Conference on Robot Learning, 2023.
Clemens Schwarke, Mayank Mittal, Nikita Rudin, David Hoeller, and Marco Hutter. RSL-RL: A learning library for robotics research. Xr preprint arXiw:2509.1079 2025.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncortainty. In Advances in Neural Information Processing Systems, 2018.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phenomenon deep reinforcement learning. In International Conference on Machine Learning, 2023.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Sceger. Gaussian process optimization the bandit setting: No regret and experimental design. In International Conference on Machine Learning, 2010.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Christian Alexander Steinparz, Thomas Schmied, Fabian Paischer, Martus-constantin Dinu, Vihang Prakash Patil, Angela Bitto-nemling, Hamid Eghbal-zadeh, and Sepp Hochreiter. Reactive exploration to cope with non-stationarity in lifelong reinforcement learning. In Proceedings of The 1st Conference on Lifelong Learning Agents, 2022.
Scbastian Thrun. Lifelong learning algorithms. In Learning to learn. Springer, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physies engine for model-based control. In IEEE/RSA International Conference on Intelligent Robots and Systems, 2012.
Tristan Tomilin, Meng Fang, Yudi Zhang, and Mykola Pechenizkiy. Coom: game benchmark for continual
reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xiv prepriul or arXin:2307.09288 2023.
René Traoré, Hugo Gaselles-Dupre' Timothée Lesort, To Sun, Guanghang Cai, Natalia Díaz- MizzRRorrgueez and David Filliat. DisCoRL: Continual reinforcement learning via policy distillation. arXiv preprint DrXiiu1907.0030 2019.
Dennis Thomas Ulmer, Christian Hardmeier, and Jes Frellsen. Prior and posterior networks: A survey evidential deep learning methods for uncertainty estimation. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL ftttp:://openreveee met/forum? (iorrmm?dd=xss8k97755
Dingrong Wang, Deep Shankar Pandey, Krishna Prasad Neupane, Zhiwei Yu, Ervine Zheng, Zhi Zheng, and Qi Yu. Deep temporal sets with evidential reinforced attentions for unique behavioral pattern discovery. In International Conference on Machine Learning, 2023.
Dingrong Wang, Krishna Prasad Neupane, Ervine Zheng, and Qi Yu. Evidential conservative Q-learning for dynamic recommendations, 2024.
Yuhui Wang, Hao Hc, Xiaoyang Tan, and Yaozhong Gan. Trust region-guided proximal policy optimization. In Advances in Neural Information Processing Systems, 2019.
Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Lukasz Kuciñskii and Piotr Miloć. Continual world: robotic benchmark for continual reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
Hongzheng Yang, Cheng Chen, Yueyao Chen, Markus Scheppach, Hon Chi Yip, and Qi Dou. Uncertainty estimation for safety-critical scene segmentation via fine-grained reward maximization. In Advonces in Newral Information Processing Systems, 2024a.
Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li. Exploration and anti-exploration with distributional random network distillation. In International Conference on Machine Learning, 2024b.
Yuzhe Yang, Guo Zhang, Zhi Xu, and Dina Katabi. Harnessing structures for valuo-based planning and reinforcement learning. In International Conference on Learning Representations, 2020.
Junwei Zhang, Zhenghao Zhang, Shuai Han, and Shuai Lü. Proximal policy optimization via. enhanced exploration efficiency. Information Sciences, 2022.
Peng Zhao, Lijun Zhang, Yuan Jiang, and Zhi-Hua Zhou. A simple approach for non-stationary linear bandits.
In International Conference on Machine Learning, 2020.
Xujiang Zhao, Shu Hu, Jin-Hee Cho, and Fong Chen. Uncertainty-based docision making using deep reinforcement learning. In 2019 22th International Conference on Information Fusion (FUSION), 2019.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, al. Secrets of RLHF in large language models part i: PPO. arXiv preprint ar arXxv23077.0996, 2023.
18

---

Published in Transactions on Machine Learning Research (11/2025)
APPENDIX
A Derivations
A.1 Derivations for evidential deep learning
We follow the derivations from Amini et al. (2020), adapting them to our notation whenever necessary,
Normal inverse-gamma (WIG) distribution We use the notation
(W, ( ~ NIG (m, ,W,
= NNulk, oi InoSam(o (c,
3o
a+1
(-25-
23 v(w p2)
exp
2(To
02
202
where w E R and v, 0, > 0. The mean, mode, and variance are given by
B
B
E [n] w, E [
var [M] u(a -0-11 1)
for a > 1.
The second and third terms correspond to aleatoric and epistemic uncertainty, respectively.
Model evidence and type II maximum likelihood loss distribution. We marginalize out μ and o:
We derive the model evidence of an NIG
plymm)
ply (alu.02)pp ppp (m)))(((,
()
P pulu, o²) p (M, 0² m²) du do²
0
0
(vlu, p 0² ,i, du do²
0==0
(1-1²²²
2To2 exp
20
0
8 VV
a+1
2B + v(w u2)
du do
exp
T(a)) 2
20
+ ky-wa
100
3 3-
I+'
do²
cxp
=0
V 2n
1/2
202
28 + m(y-w)²
800
3 0 3 20
exp
1+v
2a do
O 2 /1 11 1/v[(a)
202
T(1/2+ a) I(a)
(v (23(++ v))a (v(y-))++28(1+ (3((1+))) -(++a)
+
where I(.) is the Gamma function. Therefore, the evidence distribution p(y) m) is Student-t distribution, i.e.,
p(ylm) = St (z/lu, 3(1 + v) 20x
va
19

---

Published in Transactions on Machine Learning Research (11/2025)
which is evaluated at y with location parameter w, scale parameter B(1 D)/va, and degrees of freedom 2a. We can compute the negative log-likelihood (NT.I.) loss as:
(m) = log p(y m)
3(1 +v)
2a )
- log St
w
va
1
(,) a log (92) + a 1 log ((V- w) v +2) + log
[(a) [ 2)
log
2
where Ω = (1++)))
A.2 Derivations for the generalized advantage estimator
Given the definition of the k-step estimator as A!ta = VVe + 7 Vit + Ik-1 r'rett. we have that
var 100 = var [Vi] + 2 var [V++a).
We adapt our estimator's variance approximation for EPPO to
ACAE
(1 2)²
1
201
var
(1)
1=1
= (1 A)²
[V]
2
21 2(1
var [Ve+*]
1=1
11 - X)2
1
var [Vi] +
Σ 1m1
21 var [V++)).
1 X2
i.e. the form we have in (5).
B Further details on experiments
B.1 Experiment Details
In this section, outline the details and design choices for experiments and non-stationary environments. We use the Ant and HalfCheetah environments with the 'v5' versions of MuJoCo (Todorov et al., 2012), as these tasks do not reward the agent for maintaining stability.
B.1.1 Slippery environments
Our experimental design inspired by Dohare et al. (2021; 2024). We construct a non-stalionary environment. by varying the floor's friction coefficient. Searching for feasible friction values, we the minimum 0.5 and the maximum at 4.0. Outside of this range, solving the tasks either becomes infeasible yiclds low rewards due to excessive action costs, limited movement, the agent simply falling.
To introduce variation across tasks while ensuring differences between tasks, we incrementally change the friction by 0.25, resulting in 15 distinct tasks. We implement two strategics for these changes:
decreasing: Friction starts at its maximum value and gradually decreases.
increasing Friction starts at its minimum value and gradually increases.
20

---

Published in Transactions on Machine Learning Research (11/2025)
These setups ensures that the agents experience non-stationarity in both increasing and decreasing friction scenarios. We implement these changes by modifying the publicly available environment XML files 5 to adjust the floor friction coefficients.
B.1.2 Paralysis environments
We introduce a novel set of non-stationarity experiments by dynamically modifying the torque capabilities of leg joints the Ant and Hal Thalfhhettah environments, inspired by Al-Shedivat define six torque modification schemes for Ant and four for HalfCheetah. Each scheme targets selected joints, progressively reducing their torque capacity until they become completely paralyzed, after which their functionality is gradually restored to the fully operational state. This process results in sequence of nine tasks, where each joint's torque capacity changes in increments of 25%, following the pattern: [100, 75, 50, 25, 0, 25, 50, 75, 100]. Note that while the policy can still output full torques, the applied torque is scaled according to the specified coefficients.
(2018). Specifically, we
Paralysis on ant. The Ant environment consists of four legs and eight joints. We design distinct experiments by paralyzing different joints, ensuring that control tasks remain unique across experiments. For instance, we paralyze the right back leg, we do not conduct a separate experiment on the left back leg, as the locomotion is symmetric and would result an equivalent control task. We create the following experiments:
back-one: Paralyzing a single back leg. The affected joints are 6 and 7.
front-one: Paralyzing single front leg. The affected joints are 2 and 3.
back-two: Paralyzing both back legs. The affected joints are 0, 0, 6, and
front-two: Paralyzing both front legs. The affected joints 2,3, 4, and 5.
cross: Paralyzing diagonally opposite legs (right back and left front). The affected joints are 0, 1,
and 3.
parallel: Paralyzing the left-side legs (one back and front). The affected joints are 2, 3, and
7.
Paralysis on halfCheetah. The HalfCheetah environment consists of two legs and four joints. To prevent the agent from resorting to crawling, we modify only one joint per leg. We create the following experiments:
back-one: Paralyzing single joint in the back leg. The affected joint is 2.
front-one Paralyzing a single joint in the front leg. The affected joint is 5.
cross-vl: Paralyzing diagonally opposite joints in the back and front legs. The affected joints are and 4.
cross v2: Paralyzing a different pair of diagonally opposite joints in the back and front legs. The affected joints are and 5.
B.1.3
Plasticity preservation analysis
We calculate the metrics as follows:
Shstps://gite wtt.:/oco/////aaee http://aaaaaa-oonnaaitooo/////// Foundation/
n wtw.m//a//////aaaaaa/mmamaamaa ht/
eetah.xml
21

---

Published in Transactions on Machine Learning Research (11/2025)
Effective rank is calculated using the feature matrix & R from the penultimate layer, with
singular values 0 for 1, 1.2,...:q: where max(n, m). We define pk = ok where || = Σx [ The effective rank of & is given by:
||||||
effective rank( () a exp Pp...P......Ppp
where the entropy H (p1, Pa...... ..) - Ph log Pk.
Stable rank is also computed using the feature matrix + from the penultimate layer, with:
Σ
* of > 111
stable rank($) - min
where 5 is a threshold. We set 5 = 0.01, meaning that the selected rank captures at least 99% of the total variance.
Dormant wnit percentage measures the portion of neurons that remain consistently inactive across a batch of inputs. We compute activations immodiately after applying the nonlincarity and consider a neuron dormant if output remains below a small threshold (0.01) for all samples in the batch.
We mcasure these metrics at the end of each task in the evaluation environment and report their averages over the entire training process. The results for each environment. are presented in Figures 5 to 7. We also provide the evolution of these metrics along with the maximum absolute gradient of the critic throughout training in Figures 8 to 9.
Significance test. Figure 4 presents p-values for the relative performance of the methods, see Section B.2 for more details. Only the lower triangular part of each matrix shown, where each entry corresponds to the p-value from comparing the row and column methods. The results show that our EPPO variants preserve plasticity significantly better than plain PPO at 0.05 significance level. Note that each pairwise test is evaluated and reported independently, i..., Bonferroni correction was applied.
B.1.4 Exploration analysis
Following Mayor et al. (2025), we applied UMAP with default parameters to project the states inlo a 2D space. This projection necessary due to the large number and high dimensionality of the states. We then applied min-max normalization with small epsilon (1c-5) to scale the projected data to the range [0, 1).
Let the resulting projection be denoted as D = {(x1, vk) i = 1, ..... N}, where 24, yr E [0. We discretize the 2D space into a uniform grid of G x G cells and assign each point (æt, y1) to its corresponding grid cell:
idx² min (L., GJ, 1), idx min (Lu G-
The set of occupied cells defined as
O (idx idx.) i 1, LJ....N..
and the percentage coverage is computed as
Percentage Coverage = x 100
where |0] the number of unique occupied cells, and @ is the total number of grid cells.
We used grid size of = 100, corresponding 0.01 resolution per grid cell. Since Mayor el al. (2025) does not report the specific choices of UMAP parameters, scaling method, or grid size, selected these values to ensure fairness and reproducibility. The same 15 random seeds are used for this experiment as in the other experiments.
We relied on the following implementation: https://8ocs. httpe////ooc...piis.s.eee /apaa//aapp/aa///// http://mm////tttt 1/stablttlleeee Bumap.
22

---

Published in Transactions on Machine Learning Research (11/2025)
Table 6: p-values for effective rank in slippery.
Table 7: p-values for effective rank in paralysis.
PPC BPaO
BPPC
EPPO
PPOsuu CoPas
BP20
PPO
FEO
0.000
0.000
0.000
PFO
0.000
0.000
PPO
0.127
0.994
PPO
0.747
8Pa0
0.000
0.680
1,000
0.019
BPPO
01003]
0.621
1.0001
0.000
EPPO
0.010
0.577
1.000
0.070
0.825
2220car
0.001
0.809
1,000
0.000
0.751
EPPOand
0.001
0.683
1.000
0.011
0.457
0.135
B220
0.003
0.789
1,000
0.000
0.736
0.486
Table 8: p-values for stable rank in s slippery.
Table 9: p- values for stable rank in paralysies
PPC
PFO
PPGnasn
EPPO
EFPO
PPO
PPO
BP20
PPO
PPO
CB
0.000
0.003
CB
010001
0.000
P20μmr
0.000
0.044
0.874
0.002
EPPO
01.001
0.016
0.958
0.000
EF
EPPO
0.000
0.004
0.702
0.000
0.248
0.051
Rp90
0.000
0.124
1.000
0.000
0.924
0.606
Table 10: p-values for dormant unit percentage in Table 11: p-values for dormant unit percentage in sl slippery.
paralysis.
PPC
PFO
CB
DPCuss
EDDO
EDPC
EPPOLI
PPO
DFC
CB
PPO
EPPO
EPPO..
ED7O1sd
PPO
2P0
PPO
0.087
PFO
(.000
0.001
0.004
CB
(1000)
0.000
0.278
0.517
0.998
01663
1.000
1,000
SP20
0.003
0.071
0.921
0.010
EPPO
0.000
0.001
1,000
0.000
EFPO
0.003
0.069
0.973
0.010
0.,888
2220car
01.00]
0.013
1,000
0.000
0.962
EPPO
0.000
0.006
0.826
0.000
0.288
0.044
2230
0.000
0.044
1,000
0.000
0.951
0.494
Figure Each lower-triangular entry shows the p-value of a one-sided paired t-test testing whether the method in the row significantly outperformed the method in the column (soe Section B.2). Smaller p-values indicate stronger evidence in favor of the row method's superior performance.
23

---

Published in Transactions on Machine Learning Research (11/2025)
B.2 Statistical significance
Throughout our experiments we evaluate the significance of performance improvements via statistical tests. Our approach performs a one-sided paired t-test between the model with the highest average performance over all seeds against each of the other models. We compare the null hypothesis of equal mcans compared to the alternative of better performance by the other model. We consider the null hypothesis to be rejected for p-values smaller than 0.05. Runs are paired over random seeds and we assume approximately normally distributed performance values, fulfilling the t-test assumptions. As this procedure only compares baselines against the perforrrromi method instead of all pairwise combinations and relies on a conservative one-sided testing approach, we do not employ further multiple testing corrections.
B.3 Hyperparameters
In this section, we provide all the necessary details to reproduce EPPO. We evaluate EPPO with 15 repetitions using the following seeds: 2,, 4, 7,8,, 9, 10, 11, 12, 13, 14, 15). Our implementation will be made public upon acceptance. We list the hyperparameters for the experimental pipeline in Table 12.
B.3.1 Training
Architecture and optimization details. We train EPPO for 500 000 steps per task, performing updates to the policy and critic 10 times every 2048 step with a batch size of 256. The learning rate is set to 0.0003 for both the actor and critic, optimized using Adam (Kingma & Ba, 2015). The actor and critic networks each consist of a 2-layer feedforward neural network with 256 hidden units. Unlike other basclines, our critic network outputs four values instead of one to predict the evidential priors. We apply Layer Normalization (Ba et al., 2016) and ReLU activations (Nair Hinton, 2010) for both networks. The policy follows a diagonal normal distribution. Following common practice in the literature, we set the discount factor y = 0.99, the GAE parameter to À 0.95, and the clipping rate to = 0.2. Gradient norms are clipped at 0.5, and GAE advantage estimales are normalized within each batch.
Evaluation details. We evaluate the models the beginning and final steps of each task, as well every 20 000 steps, using 10 evaluation episodes. The evaluation environment seeds are set the training seed plus 100. For metric calculation, use the mean return across the evaluation episodes.
EPPO details. We set the regularization coefficient (E) to 0.01 to scale it down, selecting this value heuristically based on its contribution the total loss. To prevent overfitting and allow flexibility in learning, we use uninformative, flat priors for the hyperprior distributions. Specifically, we choose normal distribution N(Ww 0, 100?) for w, though positively skewed distribution may further improve performance. For v, a, and 3, we use gamma distribution Gam(5, 1) to ensure positivily. Additionally, we shift the hyperprior distribution of a by +1 to ensure finite mean.
PFO details. We followed the original implementation and set the feature regularization coefficient to 10.0. No additional hyperparameters were tuned, and the rest of the implementation was kept identical to our PPO setup.
CB details. We set the decay rate to 0.0001 and apply l regularization with a coefficient. of 0.0001, following the original paper. We performed grid search over maturity thresholds in [1000, 10 000 and replacement rates in [0.0001, 0.00001, 0.000001] using one representative task from each experimental setting. For slippery environments under the increasing setting, we selected maturity threshold of 1000 with replacement rates of 010001 for Ant and 0.00001 for HalfCheetah. For paralysis environments under the back-one setting, we used a maturity threshold of 1000 with replacement rate 0,0001 for Aut, and a maturity threshold of 10000 with replacement rate 0.00001 for HalfChectah. We adopted the generate-and-tent process from the official repository?, while keeping the rest. of the implementation identical to our PPO setup. We
7 hhepps: /ttp:////ttttttttt con/ shi bhansh/ coanshh/oss-coo loss-of-s--. cftsss---ppil -a--e-s-ppasses par-ppassiii wwwttttttyyytt city htt.t///treee/e3335eeee
3dd42e de
24

---

Published in Transactions on Machine Learning Research (11/2025)
Table 12: Hyperparameters used in the experimental pipeline.
Policy learning
Seeds
[1, 2,.,.... 15]
Number of steps per task
500000
Learning rate for actor and critic 0.0003
Horizon
2048
Number of epochs
10
Minibatch size
256
Clip rate
0.2
GAE parameter 0.95
Hidden dimensions of actor and critic Activation functions of actor and critic
[256, 256]
ReLU
Normalization layers of actor and critic Optimizer for actor and critic Discount factor 0.99
Layer Norm
Adam
Maximum gradient norm 0.5
Evaluation-related
Evaluation frequency (steps)
20 000 and end the tasks
Evaluation episodes
10
EPPO-related
Regularization coefficient (5) 0.01
Hyperprior distribution of w
N (w|0, 1002)
Hyperprior distribution of
Gam(v/3 1)
Hyperprior distribution of a
Gam (a 1) + 1
Hyperprior distribution of /
Gam (315,
Grid Search-related
Seeds
[1001, 1002, 1003]
Radius parameter к for EPPOcor
[0.01, 0.1,0.55
Radius parameter к for EPPOind
[0.01, 0.05, 0.1]
Maturity threshold for CB
[1000, 1000
Replacement rate for CB
[0.0001, 0,0010,,00001,00000 0.000001
PFO-related
Feature regularization coefficient
10.0
CB-related
Regularization rate (f2) 0.0001
Decay rate
0.0001
PPOperd-related
Hidden dimensions of bonus ensemble
[256, 256, 256, 32]
Activation functions of bonus ensemble
ReLU
Normalization layers of bonus ensemble
None
Learning rate for bonus ensemble
0.0003
Optimizer for bonus ensemble
Adam
Number of ensemble elements
10
Bonus scaling factor
0.9
The +1 ensures a finite mean for
25

---

Published in Transactions on Machine Learning Research (11/2025)
Table 13: Radius parameters (x) of EPPO.
Confidence radius parameter (k)
Experiment
Environment
Strategy
EPPCcor
EPPOind
decreasing
0.05
0.1
Ant
increasing
0.1
0.25
s1 slippery
decreasing
0.05
0.1
Hal HalfCheetah
increasing
0.1
0.1
back-one
0.05
0.01
front-one
0.1
0.1
back-two
0.01
0.25
Ant
front-two
0.1
0.01
cross
0.01
0.1
Faralysis
parallel
0.05
0.01
back-one
0.05
0.01
front-one
0.1
0.1
HalfCheetah
cross-vi
0.05
0.25
cross-v2
0.05
0.1
observed that the performance of CB is highly sensitive to the choice of maturity threshold and replacement rate but also to random seeds.
PPODRND details. We use bonus ensemble with 10 neural networks, each composed of 4-layer feedforward architecture with hidden dimensions [256, 256, 256, 32). Each network takes a concatenated state-action pair as input. We apply ReLU activations after each layer and do not include normalization layers. We optimize the ensemble using Adam (Kingma & Ba, 2015) with a learning rate of 0.0003. We scale the output of the ensemble by a bonus factor 0.9 to guide exploration during training. We adopt the architecture and hyperparameters from the original implementation by Yang al. (2024b).
Grid search details for of EPPO. We introduce confidence radius parameter (x) that controls the level of optimism incorporated into exploration. To determine appropriate value, perform grid search over E [0.01, 0.05, 0.1 for EPPOnd and E [0.01,0.1, 0.25] for EPPOcor. selecting these ranges based on their influence on the advantage estimate. We train models using three seeds (1001, 1002, 1003) and exclude them from the main results. After evaluating the AULC metric, we select the optimal x values and use them for EPPO's final evaluation. Table 13 presents the k values selected for training.
Practical implementation of EPPO. We provide pseudocode in Algorithm 1 illustrating how implement EPPO variants by overlaying color-coded modifications top of standard PPO implementation, where each color corresponds to specific EPPO variant. All lines are shared across EPPO variants and PPO unless otherwise indicated in the comment section.
B.4 Result visualizations
The learning curves across environment steps are illustrated in Figures 10 to 12. In these figures, the thick curve (dashed, dotted, dash-dotted, or solid) represents the mean returns across ten evaluation episodes and 15 random seeds, with the shaded area indicating one standard error from the mean. The legend provides the mean and standard error for the AULC and final return scores, listed in this order. The vertical black dotted lines mark the task changes.
26

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm Evidential Proximal Policy Oplimization variants (EPPOmean means EPPOcoz> EPPOina) PPO
Initial policy parameters 8, value function parameters Φ, clipping threshold e, minibatch size M, number of update epochs K, trajectory horizon T, discount factor 7, GAE parameter X, learning rates Ar, λv, radius for EPPOcor and EPPOind, regularization cocfficient &
Input:
2: for each epoch do
3: Roll out policy in the environment and fill the buffer D with (5t, oz. Ft, S141, de)
4: V + va(s)) and Ve+1 < (o($++1)
b Value estimates for PPO
5: we, M, ae, 3e, + -V)) and W441, 0441 + Vo($+11)
Di For EPPOs
V t V4+1 + Wh+1
Mean value estimates RPPOS
var + 3.
ar 1+ 1
1t ) var [V++1+ +
8x+1 (1 (1+ )
Variance value estimates of EPPOs
a1+1
8:
Compute deltas: 6, <- >V++1 - de) - Ve
9:
Initialize accumulator A +- 0 and A as empty list of size T for mean
Initialize accumulator var [A] < and var A as empty list size T for variance EPPOcoss RPPO1ad for T - 1 to 0 by -1 do
10:
11:
12:
A +- 6, + 7>A(1 de)
13:
A <- A
14:
var
[A] f- (G1)² (var [V++]] + (1 - de) de)var var [A])
EPPOcoe> EFFOLnd
15:
var
A var [V]]
1 X
2
var [A]
Equation (4) EPPOcor
A
[A]] + 1- var [V]
A
16:
var
+
var [A]
D Equation (5) EPPO:nd
1+À
A
17:
end for
18:
- A var A for all samples
UCB with Equation (3) EPPO EFFO
19:
Compute returns: R₂ Å, V2 and normalize advantages
20:
for k == to K do Shuflle D and split into minibatches of size M for each minibatch do
21:
22:
Compute importance ratio: re(0) [e(ac(s()
23:
:Taaa (ar(st)
24:
Compute clipped objective: (0) = 37
1 Σ min
+)),)
25:
Update policy with gradient clipping: 0 + A VoLeipp
26:
Compute value loss: LvF(Ø) -vF(d)= Vo(ss) Rs
PPO
27:
Estimate we, Ut, at, B2 t Vo(st)
b EPPOs
28:
Compute model-fit loss: where Pr = 213(1 + v()
Equation (2) EPPOS
(NLL(H)) - Σ log
log (Ri) + (or + 1 2) log ((ts wi)² 52 log
I (ax)
[(a++
1 Compute regularization: Greg (Ø) - E log p(og)( log (ogp(()))) -log(la)))))))))>>>>>))))))))- + logg(u))
29:
PP PPPOS
30:
Compute value loss: Lvr(Ø) (d) & (Lreg (0)
31:
Update value function with gradient clipping: r - dy VoLvr(()
32:
end for
33:
end for
34:
end for
27

---

Published in Transactions on Machine Learning Research (11/2025)
slippery-ant- -nweeareaiideiiinnggnnnn
Mean
Median
Outliers Stable Rank
Effective Rank
Dormant Units (%)
PPD
PFD
CB
PPOORND
EPPCnr
10
20
30
140
160
180
200
10
20
al i
Mean
Median o Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPD
PFD
CB
PPOORND
Qqai
EPPC2ed
10
20
30
40
50
100
150
10
20
30
40
50
al aalipppryyaae aappery-at-tie wwpaaattttiitee iincreasing
Mean
Median
Outliers Stable Rank
Effective Rank
Dormant Units (%)
PPD
PPD
8
CB
PPOORND gaan
EPPCird
10
20
30
40
100
150
200
10
20
30
40
s1 Auippper-cooe peepppeyeeeeaaaa--aep wweetah-increasing. increasingg
Mean
Median o Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPD PRD
PPOpRND
EPP0
EPPC
10
20
30
40
50
100
150
10
20
30
40
Figure 5: Plasticity preservation analysis for the slippery experiment.
28

---

Published in Transactions on Machine Learning Research (11/2025)
aaralysis-ant-bak...
Mean
Median
o Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPO
PFD
CB
O
PPODRND
EPP Ugenn
EPPU
10
20
30
125
150
175
200
10
20
paralysis-ant www.llyeiiaattfrott_oommm one
Mean
Median o Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPD
PFD
PPODRND EFPO
EppU
EPPC
10
30
40
140
160
180
200
220
10
paralysis-
wewralyyss-ann-bbbe baat-back_tee
Mean
Median
o Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPD
PRD CB
PPOORND
EPPO
EPPCjrd
30
125
150
175
200
10
20
Mean Median o Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPD
0
PRD
CB
PPODAND
EPP
EPPC
EPPCned
10
20
30
140
160
180
200
5
10
15
20
25
paralysis-ant-crree
Mean
Median
o Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPD
PFD
RPPO
0
EPPCird
10
20
30
125
150
200
10
20
paralysis-ard ant -a---anttaaaaaaaa pett-ppaallllll
Mean
Median
Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPO PFD- CB- PPDORND EPPC
EPPO
10
20
30
140
160
180
200
220
10
20
Figure 6: Plasticity preservation analysis for the paralysis experiment on Ant environment.
29

---

Published in Transactions on Machine Learning Research (11/2025)
wwwpssssscchtttaabbckkk.oo o
Mean
Median
Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPD
PFD
CB
PPOARND
EPPOqun
EPPOned
10
20
30
50
100
150
10
20
30
40
-hhreleaai-- foochh-frott_ge _one
Mean
Median
Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPD
PFD
CB
00-
PPOORND
EPPO Qqan
EPPC2ed
10
20
30
50
100
150
10
20
30
40
parallaie-chheeta-crrr ccocce-ccsss__ye v1
Mean
Median o Outliers Stable Rank
Effective Rank
Dormant Units (%)
PPD
PPD
CB
PPOORND EPP0
EPPC,rd
10
20
30
50
100
150
10
20
30
serrallsis-cheeaahhoe seaaah-crosssvee v2
Mean
Median o Outliers
Effective Rank
Stable Rank
Dormant Units (%)
PPD PRD CB
PPOpaND ggenn EPPC
10
20
30
50
100
150
10
20
30
Figure 7: Plasticity preservation analysis for the paralysis experiment on HalfCheetah environment.
30

---

Published in Transactions on Machine Learning Research (11/2025)
paralysis-ant-parallee
125
PPO 21.964 1:012
PPOpRND 23.198 1.059
EPPOmmeg 26.7%/ 0.864
Effective Rank
100-
EPPO 25.897 807 = 1.242
aP 3: 0.902
50
25-
225
Stable Rank
200-
175-
PPO 169.832 + 4.022
150-
PPCCRNO 174. 2,550
EPPOmein 183. 730 2.655
125
EPPO 182.09) + 3.456
EPPO 187.863 2.380
14.746 -1 0.828
PPOnmin 14.070 0.574
Dormant Units (%)
20-
EPPO 10.775 0,657
EPPO 11,230 11,23011: 0,792
EPPOR4 0.609
15-
10-
1250
PPO 116.385 + 15.756
PPODANO 184.561 27.6
Critic Max Gradient
EPPOmean ().23 +: 0.012
EPPO (1.449 = 0.017
EPPOnd 0.445 = 0.013
750
300
250
Iterations
x10
Figure 8: Plasticity preservation metrics and maximum absolute gradient norms throughout training for the paralysis experiment on Ant ent-parallel environment.
31

---

Published in Transactions on Machine Learning Research (11/2025)
ppralysis-chheta---lc---mm
100
PPO 17.024
PPOpRND 16.253 1.658
EPPO 16.386 = 1,717
Eflective Rank
EPPO 16.498 =: 1.713
aP 1,835
40
PPOORMD 126. - 5.778
200
EPPO 144.805 - 6.249
EPPO 14.640 5.525
EPPO 142.190 5.662
150
Stable
100
50
PPO 1,408
PPOpaNi 18.968 + 1.048
Dormant Units (%)
EPPO,..a 15.570 +: 1.4 1.414
EPPO 16:19G 1.293
EPPOna 16,426 1.301
20
10
1500
PPO S0.246 +: 10.845
PPOpaNn 94.295 + 11.677
Critic Max Gradient
EPPO...n 0.606 + 0.051
EPPO... 0.630 0.003
1000
EPPOnd 0.595 0.029
500
Iterations
x10
Figure 9: Plasticity preservation metrics and maximum absolute gradient norms throughout training for the paralysis experiment on HalfCheetah back back-one one environment.
32

---

Published in Transactions on Machine Learning Research (11/2025)
s anteeer--nt-deecreaae
3000
Episode Retur
PPQ 2330 233 (2357 229)
PFO 2523 + 109 3:13 + 100)
2000
CB 2185 63 (2303 = 71)
PPOpRND 2475 139 (2583 + 142)
1000
EPPOmsin 2504 + 126 (26M) 181)
EPPO X61 + 128 (2114 + 128)
EPPOnd 6114 + 138 (2741
145)
Timesteps
106
4000
Episode Return
PPG 249R + 201 (2483 212)
3000
CB 1519 183 (1:27 199)
2000
PPODRNO 2633
(2672 191)
EPP0 2432 2 (2522 331)
1000
EPPO
(2621
274)
EPPOn. 1666 1 213 (2970 + 231)
Timestops
1
elippery ant increasirennngg
Episode Return
3000
PPO 2237
d 231 (2841 250)
PFD 2485
50 (2
991
CB 2285
111 (2308 +. 116
2000
PPOpann 2317 23 (2 068
EPPO 2875
(9002
1000
EPPO... 2041 + SU (3071
EPPO 2779 1 S7 (341 1
Timesteps
x10
siiipery-cheeta--nnnreesse
Eeisode Return
4000
PPD 2530 1 297 (372) 310)
PFO 1s 130
[1006
16
e9 1835
(192)
1
1000
PPOppwD 2021
- 246
(21 250
EPPO 2522
(2978 228
EPPO 3:15
+ 240 (877 248)
EPPO 3374 + (3509 227
3
Timesteps
x10
Figure 10: Learning curves for the slippery experiment.
33

---

Published in Transactions on Machine Learning Research (11/2025)
paralysis-ant-
4000
Episode Return
PPO 2003 312 226 + 325
PP0 2258
138 (25
114)
2208 1 (2478 + 126)
2000
"PUnANo 2502 121 (2781 4
EPPOm 2455 + 78 (2700 + 83)
EPPO
EPPUnd 2724 + 174 (2977 109)
0.0
1.0
1.5
2.5
3.0
4.0
4.5
Timesteps paralysis-ant anttrrotttttt--oo-oooooooo
x10*
4000
Episade Retur
PPO 2054 200 12253 284)
PFU 270-2008 57 (233
87
08 2133 1-1:4 (111- (213)
185)
XXIU
PPOORNO 2547
22802 125)
EPPOmea 2407 + 92 (2649 + 96)
2749
3000 107
EPPUnd 2713 = 121 (2956 135)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
Timesteps paralysis -eta-aatt--- taattbbacktwo
110
Episode Return
3000
PPO 1972 1928
174 (2T88 + 205)
2136
(24
62)
CB 1818 - 86 (2103 1 77
2000
PPODAND 2226 + 92 (2512
EPPOmenn 2203 + 7 (2533 + 96)
1000
EPPDc 2189 + (2400 + 86)
EPPD., 2088 + 2327 112)
0.0
0.5
1.0
1.5
2.9
2.5
3.0
3.5
4.0
4.5
Timesteps
x10
paralysis-and -nttrrii--ttt-ottt--ee
Episode Return
3000
PPO 1975 +: 124 2252 189)
PF0 56 (2219 - G1)
C9 2129 + (2310 + 42)
2000
PPOμAND 2144 + 100 (245 115)
EPP0men 2259
(2536 + 98)
1000
EPPO... 2244 + 93 (2433 97)
ЕРPО
0.0
0.5
1.0
15
2.9
2.5
3.0
3.5
4.0
4.5
Timesteps
paralysis-on -at---ntt-eeee -ross
Episode Return
3000
PPG 1808 185 (214 221)
PFC 216l 77 (3107
79)
Ca
(2013
2XXX0
PPOoRD 2012 00 22310 104)
EPPO
2245 105)
1000
EPPO,, 2197 + (2500 75)
2281
(2570
0.0
0.5
1.0
155
2.0
2.5
3.0
3.5
40
45
Timesteps paralysis-ant ant patapalllel
x10
Episode Return
PPU (2807
1991
PPO 2238 + 86
(280]
+
05)
CB-1918 120
12
2000
PPO 2338
132
2647 - 145)
EPPO 2350
101
(2619
114)
EPPO 2348
127
(2553
144)
EPPO 2538
159
1583
163)
0.5
2.0
3.0
3.5
Timesteps
x10
Figure 11: Learning curves for the paralysis experiment on Ant environment.
34

---

Published in Transactions on Machine Learning Research (11/2025)
paralyeis-cheetah -heke back-one
Episode Return
4000
PRO 2411
224 (2504
2611
PFO 2181 282 (2215
3.3
CB 2258 142 (2914
1.21
2000
PRO 2131 + 294 (2004 317
EPPDnpon 3199 472 (3320
287)
EPPOc> 3502 173 (3636 1 178)
EPPO 3515 131
(3718 + 133)
0.5
4.0
4.5
Timesteps
x10
paralysis- cheetah front-one
Episode Retum
PPO 201e 299 (2115 325)
PFO 2485 271 (257
295
CB 2355
185
(2184
136)
2000
PFODAND 2542 310 (225 1.362)
EPPO 3354 227 (3540 + 235)
EPPOc*
(3724 232)
EPPO,N 3695 + 241
(3892 + 248)
0.0
0.5
1.0
3.5
4.0
4.5
Timesteps
x10
paralysis chrstahccrooss---
cross-vi
Episode Return
4000
PPO 2411 235 (2403 211)
PFO 231 287 (23-9
310)
CB 22-2225--2 + 118
(2133
131)
2000
PRODAND 2487
1.245 22648 266
EPPO 3012 +: 238 (3159 254)
EPPO 3205 +: 224 (3420
251
EPPO 3120 + (3341
220)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
Timesteps
x10"
saralyeis-cheetah-crom
4000
de Return
PPO 2477 220 (2550 235)
PFO 1903 :: 245 (1953 260)
(2434
PPO 2371 +25 (251 251)
EPPO 3039 1: (32663511 (3217
EPPO.,, 3250 = 195 (3150 4 208)
EPPO 3283 212
(3468 228
0.0
0.5
1.0
1.5
200
2.5
3.5
4.0
4.5
Timesteps
x10"
Figure 12: Learning curves for the paralysis experiment on HalfCheetah environment.
35


Reviews:
[{"review_id": "TgZUuI6zs4", "reviewer": "Reviewer_5uv8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a distributional formulation to the GAE module in PPO, which is parameterized explicitly with some Gaussian family and uses the distribution-aware objective to compute a mean-value risk measure to guide the policy. The idea is interesting, and the problem set is relevant in robotics. But I have a few serious concerns:\n1. The authors claim \"Rapid adaptation to each new situation is desired instead of remembering all previous situations.\" From the description of this work, learning in non-stationary dynamics seems to be the same as continual learning, which also adapts to new environments. I am not sure why this work necessarily needs to NOT remember all previous situations, which I believe is even more difficult. Isn't remembering all scenarios the ultimate solution for either setup? \n\n2. The formulation in (3) is precisely the distribution RL with risk measure being the mean-variance risk measure. To this end, the authors need to discuss risk-sensitive control. Furthermore, in this case, the usual Dist RL parameterization has much more freedom (e.g., approximating the distribution function). What is the advantage of this method against dist RL using, say, directly parameterize distribution functions? I am sure one can also regularize with some prior as well.\n\n3. Looking at the results in the paper and appendix, I think at the beginning stage PPO is already not as good as the proposed method, which suggests that the distributional formulation has additional effect, in which I require the authors to compare with distributional RL and risk-sensitive control methods as well. \n\n4. The setup of the experiment and the evaluation metric seems too restricted and results are not very convincing. There are a multitude of robotics work (in locomotion, which is the same task as in this work) that deal with either slippery surface (friction coefficient down to 0.1) or even rough terrain which can all be solved by PPO with state-of-the-art performance. With curriculum training, the tasks are essentially the same as in the experiment section. So I am perplexed by the presented result, which does not make sense to me for both the AULC and final return metric. It seems to me that this might be some implementation issue. Again, it is natural that a sophisticated design can bring higher performance. Then the question is, what benefits does the proposed method really have if other method with more slightly different technique can solve the same problem (if not even better)?", "claims_and_evidence_yn": "No", "claims_explanation": "Please see the comments above. Overall the results are acceptable but I do have a few concerns. The number in effective rank and stable rank does not vary too much. The difference is very subtle. For dormant units, all the mean range is within 0.1%-0.3%. Variation this small really significantly impacts the network performance? I’d like to see a more direct and effective result. \n\nAdditionally, the paralysis-ant case, none of the method shows smooth learning behavior. Why?\n\nFor the formulation, using MLE instead of Maximum A Posteriori (MAP) in bayesian learning frameworks does not make sense to me. The derivation is a nice touch but the choice of the formulation seems overly complicated and the reasons for this formulation look adjacent to me.", "audience_yn": "Yes", "audience_explanation": "This paper has a few elements that are interesting, e.g., regarding changing environment dynamics, using distributional critic and risk sensitive control. The application field in robotics is also highly relevant these days.", "requested_changes": "Please check the comments above. I would like to see\n1. More involved discussion of distributional RL, risk sensitive control, and their relationship with the proposed work. \n2. Stronger base line that robotics learning people actually use to solve the same problem, e.g., RSL_RL from https://github.com/leggedrobotics/rsl_rl. \n3. More direct results showcasing the critic is smoothly changing with changing dynamics.", "broader_impact_concerns": "No ethical concerns as far as I can tell.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d0H42AoIFV", "reviewer": "Reviewer_iEJd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the design of deep reinforcement learning algorithms under environment nonstationarity. The authors propose Evidential Proximal Policy Optimization (EPPO), an algorithm built on top of PPO that integrates evidential deep learning into the critic. They provide the option of implementing the algorithm in combination with directed exploration, a technique for increasing the exploration of high-uncertainty state and action regions. The authors conduct a range of experiments in (modified) MuJoCo environments and show that empirically the proposed algorithm achieves higher scores measured by area under learning curve and final return.\n\nOn the positive side, the paper clearly motivates, introduces, and evaluates the proposed method. It is generally well written --  the existing works & their shortcomings, the technical development of this work, environment setup, discussion of experimental results are mostly clear. The experiments, though only conducted on two environments, are sufficiently comprehensive and consistently demonstrate the better performance of the proposed algorithm over the baselines.\n\nAt the same time, I find the paper lacking in several aspects.\n\n1) The proposed algorithm is a hybrid of PPO, evidential learning, and directed exploration, making the paper an \"A+B\" type of work with limited technical contribution. While works of this nature can be important/impactful/interesting to the community, I personally believe that the bar for such combinations should be higher and would hope to see more depth in terms of experiments in larger-scale problems, mathematical analysis, and/or more thorough ablation studies (especially regarding the claim on the importance of plasticity).\n\n2) While the presentation is mostly clear, I find Sections 2.2 and 3.1 hard to follow, having no background knowledge on evidential deep learning. I particularly have problems understanding what innovations are needed to make evidential learning applicable to the estimation of value functions, and would appreciate the authors' clarification.\n\n3) While the performance from the proposed method is consistently better, the results are often not statistically significant, with the differences between EPPO variants and baselines falling within overlapping error bars.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors posit that two properties of an algorithm play important roles in dealing with nonstationarity. The first is plasticity of the critic and the second is directed exploration. This motivated the algorithm proposed in the work, and the contribution of the two properties to algorithm performance in nonstationary RL environments are comprehensively studied through experiments.", "audience_yn": "Yes", "audience_explanation": "While evidential learning and directed exploration have been studied, their integration with PPO and application in the context of nonstationary RL is new. It is good to see that these techniques lead to better solution of nonstationary RL problems and I believe researchers in the community will find this work interesting.", "requested_changes": "I have the following questions/comments that I hope the authors can clarify, besides those raised in the first text box.\n\n1) The formulation introduced in Section 2.1 considers a time-homogeneous reward function and a time-dependent transition kernel. Usually such assumptions on reward/transition kernel fixed over time are made for the purpose of mathematical analysis. However, as this paper does not contain theoretical results, I wonder why the authors need to impose the time homogeneity of reward?\n\n2) Some references seem cited in the introduction section with no clear purpose. For example, in \"On-policy algorithms, such as Proximal Policy Optimization (PPO) (Schulman et al., 2017), are particularly well-suited for non-stationary environments (Sutton et al., 2007) because they rely solely on data from the most recent policy, ensuring policy improvement through sufficiently small updates (Kakade & Langford, 2002).\" --  how does the work of Sutton et al. and Kakade & Langford relate to the discussion here?\n\n3) I do not think the experimental results conclusively show that higher critic plasticity leads to better performance in nonstationary environments. The benefit from directed exploration is more convincing, as the authors can carry out experiments with and without this component and compare. However, plasticity is not directly controllable. Across the different algorithms (and variants), plasticity is different, but so are many other factors. This makes it inaccurate to attribute the observed performance differences specifically to plasticity. At best we can only say there is some correlation. Furthermore, the separation in effective rank, stable rank, and dormant units across is not always large enough and well within the standard error range.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kSks7o9vHk", "reviewer": "Reviewer_ToWy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Please summarize the contributions of the paper in your own words. Please also list any key strengths and/or weaknesses, but please be mindful that this is NOT a substitute for the next two text boxes.\n Application of evidential based deep learning to reinforcement learning for uncertainty aware modeling of the value function\nLeverage evidential value learning to construct a probabilistic extension of the generalized advantage estimator \nExperimental designs tailored to this non-stationary setting \n\nStrengths: paper is well written\nExperimental design is well thought out\n\nWeaknesses\nI see 3 major problems with this work\n1. I am not convinced that this problem is any different than continual learning as described in [1]. The authors have attempted to make a distinction (using discrete tasks), but then use the same environments and problem set-up as [1] which is a continual learning problem. Thus, I am not sure about the framing of the non-stationary problem. If we change the framing of the problem, there are numerous other methods that the authors need to compare their method too. For example, the continual backpropagation algorithm should be compared to the solution of maintaining plasticity & doing better exploration. \n\n2. There is a lack of baselines for comparison. Loss of plasticity, by now, is a well researched problem with many different mitigation strategies. We need to know that the method prescribed in this paper is a useful method compared to other strategies, but we cannot make that decision without other baselines. Similarly, exploration strategies do exist in the literature and we cannot make any decisions about the effectiveness of the proposed method without also comparing to other exploration strategies\n\n3. Statistical testing: I appreciate that the authors showed both box plots & t-tests for the plasticity results but the authors need to include statistical testing of all results, in addition to some clarity on the method that they used to do these tests. For example, I would assume that the data generated by the experiments are not normally distributed and therefore a t-test cannot be used. Did the authors test for normality first? In addition to this, did the authors correct for performing multiple comparisons?\n\nCitations\n\n1. Dohare, S., Hernandez-Garcia, J.F., Lan, Q. et al. Loss of plasticity in deep continual learning. Nature 632, 768–774 (2024). https://doi.org/10.1038/s41586-024-07711-7", "claims_and_evidence_yn": "No", "claims_explanation": "1. This problem does not seem to be that different than the problem studied in [1]. The authors of this work use the same environments as [1] but claim that their problem is different. They also claim that continual learning 'assumes a sequential generation of tasks' but this isn't a widely accepted formalism. [1] presents the continual learning problem not as a generation of tasks, but as 'non-stationary' reinforcement learning where the friction of the environment changes which seems to be the exact problem this manuscript attempts to investigate. \n\n2. Loss of plasticity is a known phenomenon and has had several mitigation strategies proposed to overcome it. In a similar vein, there are various exploration strategies that have been proposed to improve exploration. Yet, the submitted work does not compare the proposed method to previous works from either line of work. \n\n3. The statistical results need some clarity on the methods applied & consistency of their application across sets of results. For example the comparisons between algorithms use a ranking, but the plasticity metrics compare using p-values. The authors should explain each of these processes.", "audience_yn": "Yes", "audience_explanation": "Being able to continually learn/adapt to non-stationarity is an important research direction that the community needs to address. This paper could provide some insights into how to deal with these issues, if certain things are addressed.", "requested_changes": "1. Critical: If the authors believe that this is a different problem than that studied in [1], then they need to provide a more thorough rationale for how their problem setting differs from continual learning problems. \n\n2. Critical: building on (1), the authors do not include appropriate baselines for (a) plasticity loss mitigation, or (b) exploration strategies. There is little to no discussion of either of these lines of work in the submitted manuscript, so these areas should be included in the background section.   \n\n3. Critical: the authors need to fully explain their methods of statistical testing. This includes statistical testing of all results, in addition to some clarity on the method(s) that they used to do these tests.\n\n4. Critical: It seems that the hyperparameters of included baseline methods were used from previous works, but then the hyperparameters of the proposed method were specifically tuned. The hyperparameters of each method need to be tuned [2] such that fair comparisons can be made between methods. \n\nCitations\n\n2.  Empirical Design in Reinforcement Learning. Andrew Patterson, Samuel Neumann, Martha White, Adam White; 25(318):1−63, 2024.", "broader_impact_concerns": "I don't think this is required.", "additional_comments": "I don’t think [1] proposed dormant unit percentage, I think it was [3].\n\nCitations\n\n3. Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. 2023. The dormant neuron phenomenon in deep reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning (ICML'23), Vol. 202. JMLR.org, Article 1332, 32145–32168.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)