Reinforcement learning with non-ergodic reward increments: robustness via ergodicity transformations
Authors: Dominik Baumann
Erfaun Noorani
James Price
Ole Peters
Colm Connaughton
Thomas B. Schön
Venue: Accepted by TMLR ()

Abstract: Envisioned application areas for reinforcement learning (RL) include autonomous driving, precision agriculture, and finance, which all require RL agents to make decisions in the real world. A significant challenge hindering the adoption of RL methods in these domains is the non-robustness of conventional algorithms. In particular, the focus of RL is typically on the expected value of the return. The expected value is the average over the statistical ensemble of infinitely many trajectories, which can be uninformative about the performance of the average individual. For instance, when we have a heavy-tailed return distribution, the ensemble average can be dominated by rare extreme events. Consequently, optimizing the expected value can lead to policies that yield exceptionally high returns with a probability that approaches zero but almost surely result in catastrophic outcomes in single long trajectories. In this paper, we develop an algorithm that lets RL agents optimize the long-term performance of individual trajectories. The algorithm enables the agents to learn robust policies, which we show in an instructive example with a heavy-tailed return distribution and standard RL benchmarks. The key element of the algorithm is a transformation that we learn from data. This transformation turns the time series of collected returns into one for whose increments expected value and the average over a long trajectory coincide. Optimizing these increments results in robust policies.

Published in Transactions on Machine Learning Research (01/2025)
Reinforcement learning with non-ergodic reward increments: robustness via ergodicity transformations
Dominik Baumann"
dominik. S
Cyher-physical Systems Group
Aalto University Espoo, Finland
Erfaun Noorani
enooroni@umd edu
Department of Electricol and Computer Engineering University of Maryland College Pork, MD, USA
James Price
james.price.288 weweee22wwarrik.....ooo
Deportment of Mothematics University of Warwick Warwick, United Kingdom
Ole Peters
p.peters@lml.cmm
London Mathematical Laboratory London, United Kingdom
Colm Connaughton
L.ommaughton&@lll.rrr...
London Mathematicul Laboratory London, United Kingdom
Thomas B. Schön
wu.se
Department of Information Technology Uppsala University Uppsala, Sweden
Reviewed on OpenReview: https: // openreview. net/ forum? id= eakh
Abstract
Envisioned application areas for reinforcement learning (RL) include autonomous driving, precision agriculture, and finance, which require RL. agents to make decisions the real world. A signíficant challenge hindering the adoption of RI. methods these domains the non-robustness of conventional algorithms. In particular, the focus of RL is typically on the expected value of the return. The expected value is the average over the statistical ensemble of infinitely many trajectories, which can be uninformative about the performance of the average individual. For instance, when we have heavy-tailed return distribution, the ensemble average can be dominated by rare extreme events. Consequently. optimizing the expected value can lead to policies that yield exceptionally high returns with a probability that
AAsso with the Department of Information Technology, Uppsala University, Uppsala, Sweden.
Also with the Santa Pe Institute, Santa NM, USA.
I A with the Department of Mathematics, University Warwick, Warwick, United Kingdom.

---

Published in Transactions on Machine Learning Research (01/2025)
approaches zero but almost surely result in catastrophic outcomes in single long trajectories. In this paper, we develop algorithm that lets RT. agents optimize the long-term performance of individual trajectories. The algorithm enables the agents to learn robust policies, which we show in an instructive example with a heavy-tailed return distribution and standard RL benchmarks. The key element of the algorithm is a transformation that we learn from data. This transformation turns the time scries of collected returns into one for whose increments expected value and the average over a long trajectory coincide. Optimizing these increments results in robust policies.
1 Introduction
Reinforcement learning (RI) has experienced remarkable progress in recent years, particularly within virtual environments (Mnih et al., 2015; Silver et al., 2017; Duan et al., 2016; Vinyals et al., 2019). However, the seamless transition of RL methods to real-world, e.g., robotics, applications lags behind, primarily due to the non-robust nature of conventional RL approaches (Amodei et al., 2016; Leike et al., 2017; Russell et al., 2015). In addressing this issue, rescarchers have explored spectrum of mothods from risk-scnsitive RL (Prashanth et al., 2022) to robust (worst-case) RL (Pinto et. al., 2017). In this paper, we take step back and look at the optimization objective in RL and how it may, by design, result in non-robust policies. Traditional RL literature, including inftuential references and introductory textbooks such the ones by Sutton & Barto (2018); Bertsekas (2019); Powell (2021), typically frames the RL problem as maximizing the expected return, i.e, the expected value of the sum of rewards collected throughout a trajectory. Intuitively, at each time step, an agent shall choose an action that maximizes the return it can expect when choosing this action and following the optimal policy from then onward. While this indeed seems intuitive, may not result in the desired behavior of individual agents. The expected value is the average over infinitely many rollouts of a policy. If consider an environment with heavy-tailed return distribution, the highest expected return may be achieved by extremely risky policy that receives very high returns in a few cases but fails in all others. Suppose an autonomous car learns a driving policy through RL. At deployment time, when we have a passenger in the car, it does not matter to the passenger whether the policy of the autonomous car receives a high return when averaging over multiple trajectorics- a high ensemble-average return could also result from half of the journeys reaching the destination very fast and half crashing and never reaching it. The return single instance of a long journey would be negligible if crash occurred somewhere along the way and this is the return that would matter to the individual. Thus, the time average would be the better choice for optimization objective in such scenarios.
Nevertheless, existing RL algorithms have demonstrated remarkable performance by optimizing expected returns. Optimizing the time average might require developing entirely new RL algorithms. An alternative is to find a suitable transformation. In particular, we seek to find a transformation such that for the increments of transformed returns, the expected value and time average coincide. We will discuss that the increments of such transformed returns will then follow a Brownian motion, i.e., the return grows linearly. Then, we can apply existing RL algorithms and optimize the long-term behavior of individual agents.
The field of risk-sensitive RL (Prashanth et al., 2022) follows similar approach. In most of risk-scnsitive RL, e.g,, algorithms using an entropic risk measure, the agents try to optimize the expected value of transformed returns. By learning with transformed returns, the agents can achieve higher performance with lower variance. However, those approaches typically work with fixed transformations. Inspired by Peters & Adamou (2018), we analyze for which dynamics a popular transformation from risk-sensitive RL optimizes the long- fong-term return. Further, we propose an algorithm for learning suitable transformation when the reward function is unknown, which is the typical setting in RL.
Contributions. In this paper, we make the following contributions:
We relate the shortcomings of the expected return as an optimization criterion the ergodicity of rewards and illustrate and assess the impact of non-ergodie rewards on RL policies through an intuitive example. This showcases the implications of optimizing for the expected value in such
2

---

Published in Transactions on Machine Learning Research (01/2025)
settings- which we commonly encounter in RL problems and makes case for the need for "ergodicity transformation." Note that this notion of ergodicity is not identical to the notion of ergodic Markov decision processes (MDPs) typically studied in RL, and we formally define our notion of ergodicity in section 2.1 and relate it more explicitly to the notion of ergodic MDPs in section 6.1.
We propose a transformation that can convert trajectory of returns into trajectory for whose increments time average and expected value coincide. This enables off-the-shelf RL algorithms to optimize their long-term return instead of the conventional expected value, resulting in more robust policics without developing novel RL algorithms.
We demonstrate the performance of this transformation in an intuitive example and proof-of- concept on standard RL benchmarks. In particular, show that our transformation indeed yields more robust policies.
2 Problem setting
We consider a standard RL setting in which an agent with states E S C R in the state space: S and actions E A S Rd in the action space A shall learn a policy æ: S + A. Its performance is measured by an unknown reward function r: A + R. The agent's goal is to maximize the accumulated rewards r(tx) it receives during a trajectory, i.e., the refurn R(T) at 4 = T,
T
R(T) = Σ r(T)), TE=0
(1)
where r(t&) ;= r(s(tx), a(tx)). For this, the agent interacts with its environment by selecting actions, receiving rewards, and utilizing this feedback to learn an optimal policy. The RL problem is inherently stochastic, it involves learning from finite samples, often within stochastic environments and with potentially stochastic policies. In standard RL, we, therefore, typically aim at maximizing the expected value of equation (cf. the "reward hypothesis" stated by Sutton & Barto (2018, p. 53))
T
Eg
Σ
r(TH)
(2)
=
Nonetheless, this conventional approach may encounter challenges when the return distribution is heavy-tailed. To illustrate this point, we consider an instructive example introduced by Peters (2019).
2.1 Illustrative example
Imagine agent starting with initial reward of r(to) = 100 is offered the following game. We toss (fair) coin. If it comes heads, the agent wins 50 % of its current return. If it comes up tails, the agent loses 40 %. Mathematically, this translates to
r(lx) 0.5R(tn-1) =f -0.4R((=-1)
otherwise,
where 7) is a Bernoulli random variable with equal probability for both outcomes.
When analyzing the game dynamics, find that the agent receives expected reward r(tx) equal to % of its current return. Consequently, the expected return for any trajectory length T appears favorable, growing exponentially with T:
E E[R(T) = 100 1.05.
(3)
However, when we simulate the game for ten agents and 1000 time steps, we find that all of them end up having a return of almost zero (see figure 1a). The reason that the coin toss game is non-ergodic. If the dynamics of stochastic process non-ergodic, the average infinitely many samples may be arbitrarily
3

---

Published in Transactions on Machine Learning Research (01/2025)
225
150
1020
100
90
R(tx)
10
60
10-20
200
400
600
800
1,000
-1.05
-1.05
la
100
105
110.25
(a) Simulation of the coin toss experiment. We simu- late the game for 1000 time steps and 10 agents. The dashed red horizontal line marks the initial reward of 100, and the dashed blue ascending line the expected value. After 1000 time steps, all agents end up with lower return than they started with (note the loga- rithmic scaling of the y-axis).
(b) Possible paths for three iterations of the coin-toss example. While the average across paths keeps growing.
the number of paths that lead to
successful outcome shrinks.
Figure Simulation and sample paths of the coin-toss example.
different from the average over a single but infinitely long trajectory. We illustrate this phenomenon figure 1b for three itcrations of the game. If we look at the average across iterations, we see that it steadily keeps increasing as predicted by equation 3. However, only one of all possible paths ends up with a return higher than the initial one. In the limiting case, if simulate infinitely many trajectories of the game, each of finite duration T, we obtain small set of agents that end up exponentially "rich" so that averaging over all of them, i.c., taking the expected value, yiclds 100 1.05". However, if we increase the duration, T - co, the set of agents ending up exponentially rich shrinks exponentially to measure zero. That is, if we only simulate one agent for T + 0o and average over time, we receive a time average limy 1 surely. Hulme et al. (2023) provide a more detailed analysis of the statistical properties of the coin-toss game in their appendix. In appendix A.1, we further show for a fewer number of iterations and more independent runs that there are indeed sample paths that end up with higher than the initial return.
T En== r(Tx) = 0 almost
In RL, ergodicity is typically discussed with respect to the underlying MDP, i.e, we ask whether not the MDP is ergodic. In this paper, we are concerned with what happens if rewards are non-ergodic. We discuss the relation between those two notions of ergodicity in more detail in section 6.1. To define the notion that we are concerned with proporly and connect it explicitly to RL, let us abstract from the coin-toss example and consider an arbitrary discrete-time stochastic process X. We can now generale multiple realizations of this process, in the example, by playing the game multiple times. Let (t) denote the value of realization at time step tg. The process X is ergodic if, for any time step t and realization i,
1 N
lim N
(() lim N
Σ (TH) 1==1
(4)
T- T
almost surely. The left hand side is E[X(tx)), the expected value of X at time f.- The right-hand side is the time average of realization i. For an ergodic process, these averages are equal. In the RL setting, we are interested in whether or not the rewards r(42) are ergadie, i.e., whether or not.
T
R(T) T T
B(r(4))
lim T-
r(T)) lim T==1
Σ
(5)
almost surely. For ergodic rewards, maximizing the expected value at each step corresponds maximizing the long-term growth rate of the return for any given realization. However, the coin-toss example demonstrates,
4

---

Published in Transactions on Machine Learning Research (01/2025)
when rewards are non-ergodic, optimizing the expected value may yield policies with negative long-term growth rate.
2.2 Solving the ergodicity problem
Redefining the optimization objective of RL algorithms may require a complete redesign. Alternatively, we can take existing algorithms and modify the returns to make their increments ergodic. For equation 5 to hold, we need the additive increments of the relurn to be stationary and independent. As discussed by Peters & Adamou (2018), such processes are Lévy processes. The only Lévy process with continuous sample paths is a Brownian motion with drift (Breiman, 1968, Ch. 12). Thus, following Peters & Adamou (2018) we seek to find a transformation h(R) whose increments Ah follow a Brownian motion with drift. In our discrete-time setting, this translates
(M(R((++1) =
((((t++1)) (((t:)))))(((((--))))
h(((tt))) = + oo(t))
(6)
with drift μ. volatility o, and v(tx) random variable with finite variance.
In the following, we assess the performance of standard RL algorithms in the coin toss game, with and without a transformation h. We then propose an algorithm for learning a transformation h with ergodic increments and relate our findings resk-sensitive RL.
3 RL with non-ergodic dynamics
For the coin toss example, we have already seen empirically that the dynamics are non-ergodic. Optimizing the expected value then yields a "policy" which the agent decides to play the game, leading to ruin in the long run almost surely. While standard RL algorithms aim to optimize the expected value, they need approximate it from finitely many samples. Thus, in this section, we evaluate whether a standard RL algorithm indeed proposes a detrimental policy and discuss how we can transform the returns to prevent this. In the version presented in the previous section, the coin toss gaine offers the agent a binary decision: either play or not. Here, we make the game slightly more challenging by letting the agent decide how much of its current return ("wealth") it invests at each time step. Thus, we have a continuous variable F E [0, 1] and the reward dynamics are
rlx) = 0.5FR(tk-1) =f=1, -0.4FR((=-1) otherwise.
We use the popular proximal policy optimization (PPO) algorithm (Schulman et al., 2017), leveraging the implementation provided by Raffin al. (2021) without changing any hyperparamelers to learn policy. During training, PPO receives the reward r(tx) in every iteration. Having trained policy for 1 x 10 episodes, we exccute it 100 times for 1000 time steps and show the first ten trajectorics in figure 2a. We see that all ten agents end up with return lower than the initial reward of 100. While this could still be caused by a bad choice of agents, it is confirmed by computing statistics over all 100 trajectories. When computing the median of the return after 1000 time steps, we obtain 2.5 x 10-4, i.., the average agent ends up with a return close to zero. The mean over all agent yields 115. That is, a small subset of agents obtains a high return. This confirms the discussion from the previous section. Even only approximates the expected value, PPO does learn a policy that leads to ruin for most agents.
One possibility for coping with non-ergodic dynamics is finding suitable transformation. For the coin relatively straightforward and the outcomes are fully known, we
toss game, where the dynamics analytically identify an appropriate transformation: the logarithm. While Hulme et al. (2023) provide the technical explanation for why the logarithm is an appropriate transform for the coin-loss game, we here give an intuitive explanation. The dynamics of the coin-toss game are exponential, as can, for instance, be seen from equation 3. Through the logarithm, we basically "lincarize" the return dynamics, thus achieving dynamics of the form of equation 6. We subsequently train the PPO algorithm once more with the logarithmic transformation. Specifically, redefine the rewards F(lx) == log( R(t+)) log R((+-1)). As before,
5

---

Published in Transactions on Machine Learning Research (01/2025)
10
10a
10
R(tx.
109
10
10a
10-9
101
200
400
600
800
1,000
0
200
400
600
800
1,000
1
4x
(a) Without transformation.
(b) With transformation.
Figure 2: Learning bet strategies for the adapted coin toss game. Without transformation, most agents end losing, while they end winning with transformation.
100 experiments for 1000 time steps each and show the first ten trajectories in figure 2b. We see that all agents end up with a significantly higher return than the initial reward. A statistical analysis confirms this observation, yielding a median return 5645 and a mean of 15 883. Both values substantially surpass those obtained by the agents trained with untransformed returns.
This evaluation underscores that standard RT. algorithms may inadvertently learn policies leading to unfavor- able outcomes for most agents when dealing with non-ergodic dynamics. Furthermore, it demonstrates that appropriate transformation can mitigate this. Besides, we see that even the average return is lower for the standard PPO agent, even though this should be what the agent maximizes. The reason for this is that the probability of ending up with a very high reward with risky policy non-zero for finite episode lengths but very small. Thus, during training, in some of the x 10 episodes, the agent will experience some these very high returns, leading it assigning high values those risky policies. When testing on 100 policies, the probability of encountering such high return rollout is again very low. We confirm this by evaluating the policy learned with untransformed returns x 106 times and computing the statistics. Then, receive mean return of approximately 48 010, i.e., higher than the mean for the policy learned with transformed rewards.
Remark 1. The quantitative results clearly differ between runs, as the environment and training process are stochastic. Nevertheless, the qualitative results are consistent: the training with transformed returns results in beller performance. With transformed returns, the agents somelimes get trupped in local oplima with 0, which still results significantly higher returns for the average agent.
We can support these empirical findings by analyzing the optimal choice of F under transformed and untransformed returns. We have already scen in equation 3 that without transformation, the expected return grows exponentially with time.
Proposition 1. When solving the optimization problem in equation under the dynamics in equation the optimal F E [0,1] is 1.
Proof. When adding the parameter F, the dynamics in equation 3 change to
E[R(T) = RI) 0.05F)"
Thus, F should be as large as possible to maximize B[R(T)).
Similarly, we can analyze the optimal choice of F with transformed returns.
Lemma 1. When solving the optimization problem in equation for R(T) Jog(R(T)) under the dynamics in equation 7, the oplimal F PE [0, F 0.25.

---

Published in Transactions on Machine Learning Research (01/2025)
Proof. For the expected value of logarithmic returns, find (sce also the appendix from Hulme et al. (2023))
T
[[log(R(T)] = log(R(0)) + (log(1 + 0.5F) + log - 0.4F))
2
T
log( + log(1 0.2 0.1
We maximize this by setting the derivative with respect to F to 0:
T -04F + 0.1 0,
2 --002F++0.11 +1
from which we can find F 0.25.
This result is also known as the Kelly criterion (Kelly, 1956).
In conclusion, when maximizing expected untransformed returns, we have an expected return of 100 1.05". However, the average agent ends up with a return of almost surely as goes to infinity. For logarithmic returns, expected and time average growth rate coincide, as shown by Hulme et al. (2023). Then, the expected return at T is
R(T) = R(O) exp T
T log(1 0.2 0.25² + 0.1 .0.25) = 1.006"
Thus, in expectation, the focus on expected return yields an with a higher growth rate (1.05 > 1.006). However, the long run, optimizing logarithmic return yields a growth rate larger than one, while optimixing without transformation yields return of almost surely.
4 Learning an ergodicity transformation
In scenarios like the coin toss game, due to the perfect information of future returns, it is possible to derive suitable transformation analytically- for a more detailed discussion, we refer the reader to Peters & Adamou (2018). However, the true power of reinforcement learning (RL) lies in its ability to handle complex environments for which we lack accurate analytical expressions. Therefore, it is desirable to learn transformations directly from data.
this section, we develop an algorithm for learning such a transformation from data. particular, we show how for a timo-scries of returns we can learn a transformation that transforms them into returns with increments that (spproximately) follow equation 6. For the coin-toss game, where different, policies amount to different choices of F, learning such transformation once would be sufficient the policy does not change the problem structure choosing the logarithm always yields erogide increments, no matter the choice of F. In the general RL setup, this might be different. We discuss how to embed the transformation into an RL hm that can handle also such cases in section 7.
The central characteristic of the transformation captured by equation 6: we want the increments to ergodic and, in particular, to be independent and identically distributed (f.i.d.). with constant variance. However, determining this i.i.d. property with high degree of accuracy, especially from real-world data, be challenging. Instead, approximate the behavior of the transform to that of varianco-stabilliinng transform.
Definition 1 (Bartlett (1947)). A variance stabilizing transform is defined as
h(x) = [ Vo(u) du,
with variance function v(u) describing the variance of random voriable as a function of its mean.
A variance stabilizing transform aims transform given time series into one with constant variance, independent of the mean (Bartlett, 1947). This is generalization of our desired i.i.d. property if the

---

Published in Transactions on Machine Learning Research (01/202)
10a
109
R(t
10a
10a
0
200
400
600
800
1,000
4
Figure 3: Learning bet strategies for the adapted coin toss game with learned transformation. Similar to the logarithm, also with the learned transformation, the majority of the agents ends up winning.
transformation h(R(tt)) has ii.dd increments, then the increments also have constant variance, independent of the mcan. In particular, it reflects what we define in equation 6, where the mean is determined by the drift 14, and we have defined that the random variable v(4)) must have finite variance. The i.i.d properly implied by equation 6 might then not be achieved. However, we show later in this section as well as in section 7 that also with this approximation the transformation is effective. Thus, our objective bocomes finding a variance stabilizing transform following definition In our case, as we want to stabilize the variance of the increments, we adapt the original definition of the variance function v(u) in definition to
u(u)= = Var[R(te+1) - R(tx) R(tx)= = u].
This variance function represents the variance of the following increment as a function of the current transformed return.
The approach for estimating v(u) from data is inspired by the additivity and variance stabilization method for regression (Tibshirani, 1988). Estimating v(u) first involves plotting R(tx) against log((R(t++1) R(fx) - (aff))-Aj'' with f the empirical mean of the increments. In our setting, the mean of the increments of the original untransformed process may not be constant throughout a trajectory. Hence, assuming a constant f results in small values having an over-estimated variance and large values having an under-estimated variance. The straighttorward way fix this would be to estimate p(u) function of u; however, this introduces further cstimation problem. Instead, we estimate the second-momen function and use this as proxy for the variance function,
(²( E[(R(te+1) - -[(x)))² R(tx) u].
In appendix A.2, we show that 1²(2) v(u), which is satisfactory for our needs as if the process R(tx) has i..dd increments, then will the process R(tx) for any R.
To estimate the function lo)(a²(u)) we plot R(tx) against sents taking the expected value. We use the locally estimated scatter-plot smoothing (LOESS) method (Cleve- land, 1979). The reason behind estimating log(H² (u)) is that this guarantees (²(u) always to be positive, which is vital as the variance stabilizing transform requires us to take the square root. This approach follows the rcasoning by Tibshirani (1988).
R()))))). Then, fitting curve repre-
Having derived this transformation, apply it to the coin toss game. We first collect return trajectory with =. Based on this trajectory, learn ergodicity transformation following the steps described in this section. Then, we again train PPO agent but feed it the increments of transformed returns as previously with the logarithmic transformation. As before, we execute the learned policy 100 times for 1000 time steps each and show rollouts for the first ten agents in figure 3. Also with this transformation, most agents end up lcarning winning strategies. The statistics confirm this: across all 100 agents, we have a modian return of around 17 517 and an average return of around 956 884, Thus, we conclude that we can learn a suitable transformation from data, enabling PPO to learn policy that benefits individual agents in the long run.
8

---

Published in Transactions on Machine Learning Research (01/2025)
We provide Python implementation of the transformation and the coin toss example in the supplementary material.
5 Risk-sensitive RL
The ergodicity transformation serves as means for RL agents to optimize the long-term performance of individual returns, enabling the learning of robust policics, as demonstrated in figure 3. Another approach to improving the robustness of RL algorithms is through risk-sensitive RL. While risk-sensitive RL is not motivated by ergodicity, it also proposes transforming returns. Inspired by Peters Adamou (2018), we can analyze these transformations and determine under which dynamics they yield transformed returns with ergodic increments. This analysis allows us to gain insights into which type of transformation may offer robust performance in which settings.
Here, we focus on the exponential transformation,
(i) 3 exp(
where 3 E IR \\ {0} is a hyperparameter with B < 0 the "risk-averse", and 3 > 0 "risk-seeking" case. If this transformation were an ergodicity transformation, then its increments (Ts()(())) - h N - 1)) would follow equation 6. If we now assume that the dynamics the return R(tx) belong to the class of Itō processes, i.e., a general class of stochastic processes, we can derive concrete equation describing the return dynamics. This derivation becomes relatively technical, and we defer it to the appendix (appendix A.3). Here, we only present the result and discuss its implications. We can derive the return dynamics
+ W. + 3
R₂
+
(8)
The obtained return dynamics are logarithmic in time. Logarithmic returns (or regrets) are common in the RL literature (He et al., 2021; Velegkas et al., 2022; Yang et al., 2021). Consider a scenario where a robot arm must reach a set point, and the reward defined as the negative distance to that set point. Initially, rapid progress can be made by moving quickly the roughly correct direction. As the robot gets closer, the movement becomes more fine-grained and slower, resulting in slower progress. By using an exponential transformation, we counteract this phenomenon, ensuring that all time steps contribute equally to the return.
We next apply the exponential transformation for various choices of B to the coin-toss game and test both the "risk-averse" and the "risk-seeking" setting. For the risk-seeking setting (3 > 0), we quickly run into numerical problems. The coin-toss problem has itself exponential dynamics, and thus, returns can get large. Exponentiating those again lets us reach the limits of machine precision. For the risk-averse setting (3 < 0), we consistently learn constant policies with F = While this still better than the policies standard PPO learned, it cannot compete with the results from figure
This outcome is not surprising. From an ergodicity perspective, the exponential transformation is only suitable if the dynamics are logarithmic. The dynamics of the coin-toss game are exponential, which is precisely the inverse behavior. Thus, we would not expect the transformation to yield good policies, as is confirmed by our experiments.
6 Ergodicity in RL and related work
In this section, we first discuss how our definition of ergodicity relates to the more standard one typically considcred in RL. We then discuss the significance of non-crgodicity in RL, relate our contributions to the RL literature, and briefly discuss connections to economics.
6.1 Ergodic rewards vs. ergodic MDPs
When (rrodiiryyyyccty discussed in the RL literature, this notion typically related the underlying MDP; see, for instance, Chapter 10 by Sutton & Barto (2018) or Chapter 8 by Puterman (2014). If an MDP

---

Published in Transactions on Machine Learning Research (01/2025)
is ergodic, we converge to a stationary state distribution which the initial state has no inffuence. Based on this definition, there has been work within the RI. community that provides guarantees while explicitly assuming ergodicity (Pesquerel & Maillard, 2022; Ok et al., 2018; Agarwal et al., 2022) or by guaranteeing to avoid any states within an "absorbing" barrier, i.c., only exploring an ergodic sub-MDP (Turchetta et al., 2016; Heim el al., 2020). For Q-learning, Majeed & Hutter (2018) has shown convergence even for non-ergodic and non-MDP processos. Nevertheless, none of these works, as a consoquence of non-ergodicity, question the use of the expectation operator in the objective function.
In general, the concepts of ergodic MDPs and ergodic rewards are related, although not identical. For instance, consider trivial MDP where we stay in the initial state with probability 1 and, in every state, receive a reward of +1 in every time step. Clearly, the MDP is non-ergodic, as the initial state is not forgotten, but the expected value and time average of rewards are identical. Conversely, consider a two-state MDP with a transition probability of 50% % between the two states. Clearly, the state distribution is stationary, and the initial state has no influence. Consider further that we reccive a reward of +1 in state 1 and a reward of -1 in state 2. This is an unbiased random walk. The expected reward for every step 4. is 0. However, the return for T + oo is random variable with unbounded variance. Thus, equation 5 does not hold almost surely.
Generally, an ergodic MDP does result in a stationary state distribution. The reward function is, typically, a deterministic function from the state (and potentially action) space to the real numbers. Thus, also the reward distribution is stationary. While an ergodic stochastic process is stationary, the converse does not necessarily hold. The conditions under which stationary processes are ergodic have, for instance, been discussed by Pafzen (1958).
6.2 Significance of non-ergodic rewards
The coin-toss game is excellent example to illustrate the problem of maximizing the expected value of non-ergodic rewards. When maximizing non-ergodic rewards, we may end up with policy that receives an arbitrarily high return with a probability that tends to zero in time but leads to failure almost surely. Also in less extreme cases, the expected value prefers risky policies if their return in case of success outweighs the failure cases. This results in learning non-robust policies, behavior frequently observed in standard RL algorithms (Amodei et al., 2016; Leike et al., 2017; Russell et al., 2015).
Non-crgodicity is not unique to the coin-toss game. Peters & Klein (2013) have shown that geometric Brownian motion (GBM) is a non-ergodic stochastic process. GBM is commonly used to model economic processes, a domain where RL algorithms are increasingly applied (Charpentier et al., 2021; Zheng et al., 2022). Thus, especially in oconomics, ergodicity should not simply be assumed. Nevertheless, the example of GBM is also informative for other applications. Generally, RL is most interesting when the environment dynamics are too complex to model, i.e., we usually deal with nonlinear dynamics. If already a linear stochastic process such as GBM is non-ergodic, we cannot assume ergodicity for the general dynamics we typically consider in RL. This is confirmed by the amount of recent rescarch RL with heavy-tailed rewards (Zhuang & Sui, 2021: Huang et al., 2024; Zhu et al., 2024), suggesting that it is an important problem.
is often motivated using the example of Russian roulette (Ornstein, 1973). When multiple people play Russian roulette for one round cach, and their average outcome is considered, the probability of death is one in six. However, if a single person plays the game infinitely many times, that person will eventually die with probability onc. In the context of RL, this is akin to the presence of absorbing barriers or safety thresholds that an agent must not cross. Particularly in RL applications where the consequences of failure can be catastrophic, such as in autonomous driving (Brunke et al., 2022), these safety thresholds become vital.
Another way of mrrgodicity-breakinn
How big the impact of non-ergodic rewards is and, hence, what improvement we can expect from the transformation we propose in section 4 depends on "how non-ergodic the rewards are. The physics community has introduced different measures for how much a process deviates from being ergodic (Scott et al., 2009; Mathew & Mezić, 2011). The larger the difference between the time average and expected value, the higher the impact of optimizing transformed returns.
10

---

Published in Transactions on Machine Learning Research (01/2025)
6.3 Related work
In this paper, we propose to transform returns to deal with non-crgodic rewards. In the previous section, we have shown how a popular transformation from risk-sensitive RL (Mihatsch & Neuncier, 2002; Shen et al., 2014; Fei et al., 2021; Noorani & Baras, 2021; Noorani et al., 2022; Prashanth et al., 2022) can be motivated from an ergodicity perspective. 2008; Abdolmalcki et al., 2018; Peng et al., 2019) also proposes to use transformations, but the transformations are lypically justified using intuitive arguments instead of from an ergodicity perspective. Interestingly, most, existing work also uses an exponential transformation, which is the cornerstone of risk-sensitive control. Thus, the analysis we have done for risk-sensitive RL also applies to reward-weighted regression. In the risk-sensitive RL literature, we also find works different transformations than the exponential one. For instance, Prashanth & Ghavamzadeh (2013) estimate the variance of returns and incorporate it into a transformation, Tamar et al. (2015) provide a more general analysis of different risk measures. Investigating also such approaches from an ergodicity perspective would be an interesting study for future work.
wwward-wiigheed regression (Peters & Schaal, 2007; 2008; Wierstra et al.,
Another approach that optimizes transformed returns is Bayesian optimization for iterative learning (BOIL) (Nguyen et al., 2020). BOIL is developed for hyperparameter optimization. While this setting differs from the one we consider, we show in appendix A.4.1 that the transformation used in BOIL can be replaced with ours, leading to similar or better results.
Through the ergodicity transformation, we seek to optimize the long-term performance of RL agents. Improving the long-term performance of RL agents in continuous tasks is also the goal of average reward RT. The idea of optimizing the average reward criterion originated in dynamic programming (Howard, 1960; Blackwell, 1962; Veinott, 1966), and has already in the early days of RL been taken up to develop various algorithms, see, for instance, the survey by Mahadevan (1996). Also in recent years, the average reward criterion has been used for novel RL algorithms (Zhang & Ross, 2021; Wei et al., 2020; 2022). In average reward RL, we still take the expected value of the reward function and let time go to infinity. Were the reward function ergodic, it would not matter whether we first take the expected value or first let time go to infinity. However, for a non-ergodic function, it does. In average reward RL, we first take the expected value. For the coin-toss game, that would yicld an optimization criterion that grows exponentially while the set of agents that obtain return larger than zero shrinks to set of measure zero as time goes to infinity. Thus, average reward RL may fall into the same trap as conventional RL when dealing with non-ergodic reward functions.
A further rescarch direction in RL to which our approach can be related is reward shaping (Ng et al., 1999; Tang et al., 2017; Zheng et al., 2018; Memarian et al., 2021). In reward shaping, we typically try to adapt the existing reward function, for instance, to deal with sparse rewards or to encourage exploration. Usually, the new reward function is then a summation of the original reward and a new element. This new element introduced by reward-shaping techniques can be designed based on prior knowledge about the environment or learned from data. In our case, instead of adding new element to an existing reward function, we transform the entire return. Thus, our approach fundamentally differs from existing reward-shaping techniques. However, the result of the transformation is that the increments of transformed returns follow equation 6. Thus, we have linear, stochastic return dynamics. Such dynamics should be easier to optimize through gradient- based RL algorithms than the arbitrary dynamics that we typically assume. Therefore, reward-shaping may serve as an additional motivation for our approach.
6.4 Connections to economics
The transformations used in risk-sensitive RL are often motivated by economies, in particular, by util- ity (Bernoulli, 1954) and prospect (Kahneman & Tversky, 1997) theory. These theories rely on psychological arguments to argue that some humans are more "risk-averse" than others. Peters & Adamou (2018) have shown how acknowledging non-ergodieity and that humans are more likely to optimize the long-term return than an average over an ensemble of infinitely many trajectories can recover widespread transformations used in utility theory. Empirical rescarch (Moder et al., 2021; Vanhoyweghen et al., 2022: Skjold et al., 2024) has further shown that this treatment can better predict actual human behavior. The ergodicity perspective does not rely on psychology as explanation; instead, it explains psychological observations. It is, in this sense,
11

---

Published in Transactions on Machine Learning Research (01/2025)
more fundamental and, result, more general, namely applicable to cases where psychology cannot be invoked, particularly to inanimate optimizers such as machines devoid of a psyche.
7 Proof-of-concept
The coin-toss game, while illustrative, represents simplified scenario. As already mentioned section 4, in coin-toss game, learning a single transformation is sufficient as the policy parameter F' has no influence on the problem structure. In RL, this not necessarily the case. Thus, we here embod the transformation into an RL algorithm. In particular, we here develop an ergodic REINFROCE algorithm an example for Monte Carlo-basod algorithms. That is, REINFORCE always collects return trajectory and then uses this trajectory to update its weights. In our setting, this is advantageous as it allows us to learn transformation using the collected trajectory. We summarize ergodic REINFORCE in algorithm 1.
Algorithm Pseudocode of the ergodic Monte Cased RL algorithm. for in num episodes do R(to); ...., R(rk) + (ol)
log(uE, (4x)) +
R(t
h(th)) t fox exp(12, (1s)) dx
4:
F+ K E (N(B(1,)) - 1((((xx1)))
Mi+1 f- updato (Fllccy()
We evaluate ergodic REINFORCE on two classical RL benchmarks: the cart-pole system and the reacher, using the implementations provided by Brockman et al. (2016). In both cases, seek to evaluate whether not the ergodicity transformation increases robustness. James et al. (1994) have established that robustness to heavy-tail distributions is related to robustness against model uncertaintics. Thus, we evaluate robustness by changing model parameters ween training and testing. Specifically, we change the length of the pole for the cart-pole system from 0.5 to and the length of one of the links for the reacher from 1 to 1.5. Besides, the cart-pole system has a non-crgodic reward structure as there exists an absorbing state: the episode ends if the pole drops.
We compare the ergodic REINFORCE algorithm that is trained shown in algorithm 1 with the standard REINFORCE algorithm trained with untransformed rewards. In figure 4, we show the mean and standard deviation of the return over five runs. For both algorithms, we plot the untransformed returns. Further details on hyperparameter choices are provided in appendix A.5.
Cart-pole. In the cart-pole environment, the objective is to maintain the pole in an upright position for long as possible. To evaluate the long-term performance of the ergodicity transformation, we train the algorithm using episode lengths of 100 time steps but test it with episodes lasting 200 time steps. Thus, see in figure 4a, the return during testing is higher than during training. We can also see that for ergodic REINFORCE, the agent is closer to the optimal reward of 200 during testing. The standard REINFORCE algorithm performs slightly worse. Thus, we can see that leveraging the ergodicity transformation improves the long-term performance compared to the standard algorithm.
Reacher. In the reacher environment, we aim to track set point with the end of the last link. Thus, extending the episode length does not make sense in this setting. However, this is unnccessary to demonstrate the advanlage of using the ergodicity transformation. In figure 4b, see that, while both algorit successfully improve their return during training, ergodic REINFORCE even more than standard REINFORCE, ergodic REINFORCE can generalize to the new link length and mass during testing. Standard REINFORCE ends up with close to minimal reward during testing.
8 Conclusions and limitations
This paper discussed the impact of ergodicity on the choice of the optimization criterion in RL. If the rewards non-ergodic, focusing the expected return yields non-robust policies that currently find with
12

---

Published in Transactions on Machine Learning Research (01/2025)
-100
200
ergodic
standard
Tritning Phose
Fostling Plase
150
R(T)
-150
100
Training Plaae
Testing Plase
-200
0.2
0.4
0.6
0.8
2,000
4,000
6,000
Number of episodes -102
Number of episodes
(a) Cart-pole.
(b) Reacher.
Figure 4: Ergodic vs. standard REINFORCE on common benchmarks. For the cart-pole, we see slight improvements when using the ergodicity transformation, while for the reacher, only ergodic REINFORCE learns a successful policy.
conventional RL algorithms. An alternative to changing the objective and, with this, having to come up with entirely new RL algorithms is trying to find an ergodicity transformation. We presented a method for learning an ergodicity transformation that converts a time series of returns into a time series with ergodic increments. Then, optimizing the expected value of those ergodic increments equivalent to maximizing the long-term growth rate of the return. We further showed how the ergodicity perspective provides theoretical foundation for transformations used in risk-sensitive RL. We demonstrated the effectiveness of the proposed transformation on standard RL benchmark environments.
This paper is the first step toward acknowledging non-ergodicity of reward functions and focusing on the long-term return and, with that, robustness in RL. This opens various directions for future research. Firstly, addressing the challenge of transforming returns in RL algorithms that update weights incrementally rather than relying on episodic data remains an open question. Secondly, our transformation currently focuses solely the current. return, but returns may also depend the current state of the system, suggesting the possibility of state-dependent transformations. Then, also investigating the computational complexity and trading off potentially more robust performance with the additional complexity through the transformation is a crucial aspect. Thirdly, extending this research to multi-agent RL could be promising, building on insights by Fant et al. (2023) and Peters & Adamou (2022) regarding the impact non-ergodicity on the emergence of cooperation in biological multi-agent systems. Finally, investigating the connection between optimizing time-average growth rates instead of expected values and discount factors, as explored by Adamou al. (2021), could make the discount factor hyperparameter in RL dispensable.
Acknowledgements
This research partially supported by the project NewLEADS New Directions in Learning Dynamical Systems (contract number: 621-2016-06079), funded by the Swedish Research Council and by Kjell och Märta Beijer Foundation, and by the Engineering and Physical Sciences Research Council through the Mathematios of Systems 11 Centre for Doctoral Training at the University Warwick (reference EP/S022244/1).
References
Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Hoess, and Martin Riedmiller, Relative entropy regularized policy iteration. arXiv preprint 2018.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Alexander Adamou, Yonatan Berman, Diomides Mavroyiannis, and Ole Peters. Microfoundations of discount- ing. Decision Analysis, 18(4):257-277 2021.
Mridul Agarwal, Qinbo Bai, and Vaneet Aggarwal. Regret guarantees for model- reinforcement learning with long-term average constraints. In Uncertainty Artificial Intelligence, pp. 22-31, 2022.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safcty. arXiv preprint arXiv:1606.06360 2016.
Maurice $ Bartlett. The use of transformatious. Biometrics, 3(1):39 52, 1947.
Danicl Bernoulli. Exposition of new theory on the measurement of risk. Econometrica, 22(():23-33 1954.
Dimitri Bertsckas. Reinforcement Learning and Optimal Control. Athena Scientific, 2019.
David Blackwell. Discrete dynamic programming. The Annals of Mathematical Statistics, pp. 719 726, 1962.
Leo Breiman. Probability. Addison-Wesloy Publishing Company, 1968.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint Xrx6 01540, 2016.
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning- based control to safe reinforcement learning. Annual Review of Control, Roboties, and Autonomous Systems, 5:411-444, 2022.
Arthur Charpentier, Romuald Elie, and Carl Remlinger. Reinforcement learning in economics and finance.
Computational Economics, pp. 38, 2021.
William Cleveland. Robust locally weighted regression and smoothing scatterplots. Journal of the American Stalistical Associalion, 74(368):829 836, 1979.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Picter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1339,1338, 2016.
Lorenzo Fant, Onofrio Mazzarisi, Emanuele Panizon, and Jacopo Grilli. Stable cooperation emerges in stochastic multiplicative growth. Physical Review E, 108(1):L012401, 2023.
Yingjic Fei, Zhuoran Yang, and Zhaoran Wang. Risk-sensitive reinforcement learning with function ap- proximation: A debiasing approach. In International Conference on Machine Learning, pp- 3198 3207, 2021.
Jiafan He, Dongruo Zhou, and Quanquan Gu. Logarithmic regret for reinforcement learning with linear
function approximation. In International Conference on Machine Learning, pp. 4171 4180, 2021.
Steve Heim, Alexander von Rohr, Sebastian Trimpe, and Alexander Badri-Spröwitz. A learnable safety measure. In Conference on Robot Learning, pp. 627-639, 2020.
Ronald A Howard. Dynamic Programmingg and Markov Processes. John Wiley, 1960.
Jiayi Huang, Han Zhong, Liwei Wang, and Lin Yang. Tackling hcavy-tailed rewards in reinforcement lcarning with function approximation: Minimax optimal and instance-dependent regret. bounds. Advances Neural Information Processing Systems, 2024.
Oliver Hulme, Arne Vanhoyweghen, Colm Connaughton, Ole Peters, Simon Steinkamp, Alexander Adamou, Dominik Baumann, Vincent Ginis, Bert Verbruggen, James Price, and Benjamin Skjold. Reply to ''The limitations of growth-optimal approaches to decision making under uncertainly". Econ Journal Watch, 20 (2):335 348, 2023.
Kiyosi Itó. Stochastic integral. Procerdings of the Imperial Academy, 20(8):519 1--51,9--2,,,,1 1944.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Matthew R. James, John S Baras, and Robert J Elliott. Risk-sensitive control and dynamic games for partially observed discrete-time nonlinear systems. TEEE Transactions on Aulomalic Control, 39(4):780- 792, 1994.
Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455, 1998.
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. (2):263 292, 1997.
Econometrica, 47
John L Kelly. A new interpretation of information rate. The Bell System Technical Journal, 35(4):917-926, 1956.
Peter E. Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations. Springer Berlin, Heidelberg, 1992.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Orlega, Tom Everitt, Andrew Lefrancy, Laurent. Orseau, and Shane Legg. Al safety gridworlds. arXie preprint arXiv:1711.09883, 2017.
Sridhar Mahadevan. Average reward reinforcement learning: Foundations, algorithms, and empirical results. Machine Learning, 22:159-1955 1996.
Sultan Javed Majeed and Mareus Hutter. On Q-learning convergence for non-Markov decision processes. International Joint Conference on Artificial Intelligence, pp. 2546 2552, 2018.
George Mathew and Igor Mezić. Metrics for ergodicity and design of ergodic dynamics for multi-agent systems. Physica D: Nonlineur Phenomena, 240(4-5):432 442, 2011.
David Meder, Finn Rabe, Tobias Morville, Kristoffer Madsen, Magnus T Koudahl, Ray J Dolan, Hartwig R Siebner, and Oliver J Hulme. prrodicity-breakinn reveals time optimal decision making in humans. PLoS Computational Biology, 17(9):e1009217, 2021.
Farzan Memarian, Wonjoon Goo, Rudolf Lioutikov, Scott Niekum, and Ufuk Topeu. Self-supervised online reward shaping in sparse-reward environments. In IEEE/RSJ International Conference on Intelligent Robols and Systems, pp. 2369 2375, 2021.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement. learning, Machine Learning, 49(2):267 290, 2002.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukeuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pp. 1928 1928-1937, 2016.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, pp. 278-287, 1999.
Vu Nguyen, Sebastian Schulze, and Michael Osborne. Bayesian optimization for iterative learning. Advances in Neural Information Processing Systems, pp. 9361 9371, 2020.
Erfaun Noorani and John S Baras. Risk-sensitive reinforce: A Monte Carlo policy gradient algorithm for exponential performance critcria. In IEEE Conference on Decision and Control, pp. 1522-1527, 2021.
Erfaun Noorani, Christos Mavridis, and John Baras. Risk-scnsitive reinforeement learning with exponential criteria. arXiv preprint arXiu:2212.090100 2022.
Jungseul Ok, Alexandre Proutiere, and Damianos Tranos. Exploration in structured reinforcement learning. Advances in Neural Information Processing Systems, 2018.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Donald S Ornstein. An application of ergodic theory to probability theory. The Annals of Probability, 1(1): 43-58, 58, 1973.
Emanuel Pafzen. Conditions that a stochastic process be ergodic. The Annals of Mathematioal Statistics, 29 (1):299 301, 1958.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-wcightod regression: Simple and scalable off-policy reinforcement learning. av preprinl urXi::9910.0017,, 2019.
Fabien Pesquerel and Odalric-Ambrym Maillard. IMED-RL: Regret optimal learning of ergodic Markov decision processes. Advances in Neural Information Processing Systems, pp. 26363-26374 2022.
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In International Conference on Machine Learning, pp. 745 750, 2007.
Jan Peters and Stefan Schaal. Learning control in operational space. The International Journal of Robotics Research, 27 2008.
Ole Peters. The ergodicity problem in economics. Nalure Physics, 15(12):1216-12) 1221 2019.
Ole Peters and Alexander Adamou. The time interpretation of expected utility theory. ar preprinl ar 1iX:::801.006600 2018.
Ole Peters and Alexander Adamou. The ergodicity solution of the cooperation puzzle. Philosophical Transactions of the Royal Sociely A: Mathematicnl, Physical and Engineering Sciences, 380(2227) 222200422, 2022.
Ole Peters and William Klein. Ergodicity breaking in geometric Brownian motion. Physical Review Letters, 110(10):100603, 2013.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pp. 2817-2826, 2017.
Warren B Powell. Reinforcement Learning and Stochastic Optimization. John Wiley & Sons, 2021.
L. A. Prashanth and Mohammad Ghavanzadeh. Actor-critic algorithms for risk-sensitive MDPs. Advances in Neural Information Processing Systems, 2013.
LA Prashanth, Michael C Fu, et al. Risk-sensitive reinforcement learning via policy gradient search. Foundations and Trends& in Machine Learning, 15(5):5377 0633, 2022.
Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-basee baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 12 2021.
Stuart Russell, Daniel Dewey, and Max Tegmark. Rescarch priorities for robust and beneficial artificial intelligence. AT Magazine, 16(4):005-114, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint 1rXi::1707.003 2017.
Sherry Scott, Thomas C Redd, Leonid Kuznetsov, Igor Mezić, and Christopher KRT Jones. Capturing deviation from ergodicity at different scales. Physica D: Nonlinear Phenomena, 238(16):1668 1679, 2009.
Yun Shen, Michael Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement learning. Neural Compulation, 26(7):1296 (298)):99811355 2014.
16

---

Published in Transactions on Machine Learning Research (01/2025)
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354 359, 2017.
Benjamin Skjold, Simon Richard Steinkamp, Colm Connaughton, Oliver Hulme, and Ole Peters. Ergodicity transformations predict human decision-making under risk. OSF Preprints c96yd, Center for Open Science, 2024.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradient for coherent risk measures. In Advances Neural Information Processing Systems, 2015.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filiip DeTurck, and Picter Abbecl. # exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
Robert Tibshirani. Estimating transformations for regression via additivity and variance stabilization. Journal of the American Statistical Association, (5(402):394 405, 1988.
Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite Markov decision processes with Gaussian processes. Advances in Newral Information Processing Systems, 2016.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. AAAI Conference on Artificial Intelligence, pp. 2094-2100, 2016.
Arne Vanhoyweghen, Brecht Verbeken, Cathy Macharis, and Vincent Ginis. The influence of ergodicity risk affinity of timed and non-timed respondents. Scientific Reports, 12(1):1-9, 2022.
Arthur F Veinott. On finding optimal policies discrete dynamic programming with no discounting. The Annals of Mathematical Statistics, 37(5):1284-1294, 1966.
Grigoris Velegkas, Zhuoran Yang, and Amin Karbasi. Reinforcement learning with logarithmic regret and
policy switches. Advances in Neural Information Processing Systems, pp. 36040 36053, 2022.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnocki, Michaël Mathicu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 375(7782):3500-333 3544 2019.
Ziyu Wang and Nando de Freitas. Theoretical analysis of Bayesian optimisation with unknown Gaussian process
arXiv preprint arXiu:1466. 7758, 2014.
Chen-Yi Yu Wei, Mehdi Jafarnia Jahromi, Haipeng Luo, Hitoshi Sharma, and Rahul Jain. Model-freo reinforce- ment learning in infinile-horizon average-reward Markov decision processes. In International Conference on Machine Learning, pp. 10170-1000 101800 2020.
Honghao Wei, Xin Liu, and Lei Ying. A provably-efficient model-free algorithm for infinite-horizon average- reward constrained markov decision processes. In AAAI Conference on Artificial Intelligence, pp. 3868-3876, 2022.
Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Episodic reinforcement learning by logistic weighted regression. In International Conference on Artificial Neural Networks, pp. 407-416, 2008.
Kunhe Yang, Lin Yang, and Simon Du. Q-learning with logarithmic regret. In International Conference on Artificial Intelligence and Statistics, pp. 1576-1584, 2021.
Yiming Zhang and Keith W Ross. On-policy deep reinforcement learning for the averago-reward criterion. International Conference on Machine Learning, pp. 12535-1222 12545, 2021.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C Parkes, and Richard Socher. The AI economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science Advances, 8(18): eabk2607, 2022.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, 2018.
Jin Zhu, Runzhe Wan, Zhengling Qi, Shikai Luo, and Chengehun Shi. Robust offline reinforcement learning with heavy-tailed rewards. In International Conference on Artificial Intelligence and Statistics, pp. 541-549, 2024.
Vincent Zhuang and Yanan Sui. No-regret reinforcement learning with heavy-Lailed rewards. In International Conference on Artificial Intelligence and Statistics, pp. 3385-3393, 2021.
18

---

Published in Transactions on Machine Learning Research (01/2025)
10
R(ta)
10
10-5
0
20
40
60
80
100
tk
Figure 5: Coin-toss with lewer iterations. With fewer ilerations and more trajectories, we see that a few do end up with a higher than the initial return.
A Appendix
A.1 Coin-toss example with fewer iterations
In figure la, have seen that while the expected return is increasing, all individual trajectories end up with lower than the initial return. The reason is, as shown in figure 1b, that the number of sample paths that end up "winning" decreases as the trajectory length increases. To verify that there are actually sample paths that end up with a higher return, we show in figure another simulation of the coin-toss game. We here simulate the game for only 100 iterations to increase the probability of positive sample paths and simulate 100 independent games. We see that there are indeed sample paths where the final return is higher than 100. Nevertheless, those are minority, and in most cases, we still end up with lower return.
A.2 Proportionality of variance and second moment functions
The variance-stabilizing transform h(z) is unique up to linear transformations. That is, the function ah(x)) + b for Rt, b will also produce time series with the desired properties. Thus, we only need to estimate the variance function up to scalar multiplier. In the following, we approximate 1² (w) = [[[(((++1) R(tx))² R(tx) = u| using a Taylor series expansion and show that v(u) x ( In particular, we have
(2) = [[(([(e+1)) R(ta))) R(tx)) = u]
[[[((t++1) ) R((t)) = w]
2E{[R(te+1) (R(t)) = u][[(tt)) R((t)) = u]
+ E[(((t)) R((tt) u]
=E[(R((++1) 2 [R(tt) = w.] 22uR[[((2+1) R((+)) u] u²
We now perform second-order Taylor expansion with the function h- on the random variable h(R(tx+1)) to find E[R(t++1) R(tx) = u].
[[ R(t+) ud
= =E|4- (M((((()(((( | uu]
[h)u)) (((((+)))))(+((( h li(tx) = u]
h(u))
+ 2 (M==))((((())(((((+++))) =(u))² R(tx) = u)
-m (m- m2 (H(((()))
19

---

Published in Transactions on Machine Learning Research (01/2025)
In the final step, h(u) is function that transforms the original time series into time series with independent increments, we can assume that, for all n E N,
B((((((t=1)) -(())))))))))(()))))))))))))"")) R(tt) = u] mm R.
That is, the moments of the transformed increments are stationary over the state space. We can then use the inverse-function rule to calculate the derivatives as
1
(h (h-())) W/th (((h)))(u))) = h'
-h((h-(h(())) -h((u) (((4))) = 1'hh- h((h-(((u)))² hl()3
Hence, we have
mi m2h"(u)
E[R(tx+1) R(tx) ul 2ut
h'(u)
2u)
We use a similar method to find E[R(t )2 R(t+) = u/. However, this time, we perform the Taylor expansion with the squared-inverse function h (xx) = (h- (T){))))²
E R R(tx) = u|
= h 1 (h((((t++1)) R(tx) = ul
2n mm (h (h(h))) + 11 (h (h-2)"(h(w))-
We can use the chain rule to calculate the derivatives the squared- inverse function,
2u (h ()-))(((w)) 2(h (((h(w)))~>h(00)) h'())
and
(h (h(())))( = (u) 2
(4-))())))
N((u33
h'(u)22
Hence, we have
E[R(t | R(tx) u) ~ u² 2um
mzuh"(u)
+ m2 h'(w)²
12(w) h((u)3
Substituting into equation 9 gives us
(2) = E[R(te+1)* 12 R(tx) = u]
22 (t(++1) R(tx) = u] + 2²
122 2um1
myuh"
+ 12(u)²2
h'(u)
1((133
2u + M(u)
mi
m2h"(u)
u²
2h'(1)3
m2
(u)22
Finally, use the fundamental theorem of calculus definition to get
v(u) = 12(2²² = (²(() x u(u) (approximately).
20

---

Published in Transactions on Machine Learning Research (01/2025)
A.3 Derivation of the risk-sensitive reward function equation
For the sake of clarity, we perform our analysis in continuous time. We assume that the return follows an arbitrary Itō process
dR = ((R dt + g(R) dW(t),
(10)
where [(R and ( are arbilrary functions of and ((1) is a Wiener process. This captures a large class of stochastic processes, as both f and g can be nonlinear and even stochastic. Assume now that the risk-sensitive transformation hys extracts an ergodic observable from equation 10. Then, its increments follow Brownian motion, i.., the contimuous-time version of equation 6:
dh,s d + dW (1).
(11)
As know hys: now seek to find and g for which equation 11 holds. Following Itô's lemina (Itô, 1944), can wrile dR
(ŌR [R
aR This
1 0²
F2 ah
aR Thes
dR
dt
dW(t).
(12)
Öt
2
In( 3 3
As we can invert (R) such that R(hiss) it into equation 12 and obtain
and since the inverse is twice differentiable, can insert
dR =
1
1
02
dt + dt())
Bl1ss
2
Bhis
1
o²
di
(13)
B2 (xp(3R)
2 B3 exp(23R)
82 exp(3R) (t)
This equation provides valuable insights into the role of B. Specifically, it highlights that the volatility term (the coefficient of dl (t)) is always positive, regardless of the sign of B. However, the drift term (the coefficient of dt) depends on the sign of B. For B < 0, the drift term is positive, while for > 0, starts negative when 3 is small and then turns positive as increases.
From an ergodicity perspective, the risk-averse variant with 3 < 0 is suitable when equation 13 exhibits positive drift, while the risk-secking variant with B > 0 is more appropriate when equation 13 has negative drift. This aligns with intuitive reasoning: when the drift is negalive, there limited gain from caution, and one might choose to go all in and hope for luck. This is also the case when the drift is too small to outweigh the volatility.
The differential dynamics in equation 13 have a closed-form solution. We start the derivations by simplify- ing equation 13. We introduce k k(R)= = 1 exp(arj and c. = 3 which results in
1 + ((R)))) dt + (())dW()).
dR
From this, we can see that the resulting stochastic differentiable equation belongs to the class of reducible SDEs and has a known, general solution (Kloeden & Platen, 1992, pp. 123 124):
R R= ( Cx
W. (1++We+100
where ((r) f ds k(a) = =f" 8 exp( Now, we need to find an expression for ((R):
((R) = / ds
n 82
3
21

---

Published in Transactions on Machine Learning Research (01/2025)
This expression is invertible,
(
= In
o
+
In R|.
8
B
B
Thus, finally obtain equation 8:
1 + W, + +((0) = =
L + W. + 2)
R2 = -
Cv
W, + g
in
+ lm
3
A.4 Hyperparameter optimization
Besides the experiments presented in the main body, we also compared our learned transformation in a hyper- parameter optimization task with the BOIL (Bayesian optimization for iterative learning) algorithm (Nguyen et al., 2020). Before presenting the results, we briefly introduce BOIL.
A.4.1 Bayesian optimization for iterative learning
Boil aims to train a machine learning algorithm given a d-dimensional hyperparameter X Rd for T iterations. This process produces training evaluations R(- xr, T)) with T E [Tiin: Tmax]. These evaluations could generally be returns of an episode in RL training accuracies in deep learning. Here, focus episode returns in reinforcement learning. Given the raw training curve R(- 2, T), BOIL assumes an underlying, smoothed black-bon function f and then aims to find x² = arg max f(x, maax). This black-box function is modeled as a Gaussian process (GP), and the next set of hyperparameters is selected using a variation (Wang& & de Freitas, 2014) of the expected improvement (Jones et al., 1998) algorithm.
Existing Bayesian optimization approaches for hyperparameter optimization typically define the objective function an average loss over the final learning episodes. This ignores how stable the performance is and might be misleading due to the noise and fluctuations of observed episode returns, especially during early stages of training. Therefore, in BOIT, the authors propose compressing the whole learning curve into a numeric score via preference function. In particular, they use the Sigmoid function (specifically, the Logistic function) to compute this "utility score" as
y = (R, mo: 90) = RR( 1,,,T) E(- mo.9o)
R(u | xI,T)
(14)
ma))
where a dot product, and the Logistic function E(- ma. 90) is paramoterized by a growth parameter 9o defining the slope and the middle point of the curve mg. The choice of the Sigmoid function is mainly motivated by intuitive arguments. Since early weights are small, less credit is given to fluctuations at the initial stages, making it less likely for the surrogate function to be biased toward randomly well-performing seltings. As weights monotonically increase, lyperparameters with improving performance are preferred. As weights saturate, stable, high-performing configurations are preferred over short "performance spikes" which often characterize unstable training. The score assigns higher values to the same performance if it is being maintained over more episodes.
The intuition provided by Nguyen et al. (2020) is that the optimal parameters mo, 9o will lead a better fit of the GP, resulting in better prediction and optimization performance. The authors then parameterize the GP log marginal likelihood in terms of my and gu and optimize both parameters using multi-start gradient descent.
A.4.2 Comparison
We tried to apply BOIL to the coin toss game i.c., we tried to optimize hyperparameters for an RL algorithm the coin toss game using BOIT. Unfortunately, there ran into numerical problems caused by the large
22

---

Published in Transactions on Machine Learning Research (01/2025)
200
Average Reward
150
-20
40
100
60
50
Ergodic
80
Ergodic BOIL
BOIL
0
100
0
100
200
300
400
500
100
200
300
400
500
Episode
Episode
(a) Cart-pole.
(b) Reacher.
Figure 6: Comparison of BOIL and our transformation for hyperparamcter optimization of deep RL algorithms. Our non-poramebric transformalion performs at least en por with stale-of-the- luperpprommeeer oplimization algorithms.
values the return can have in some runs. Therefore, we compare BOIT, to our learned transformation on the same benchmarks we used in section as they were also used by Nguyen et al. (2020). However, instead of learning policies, optimize hyperparameters of deep RL algorithms that try to learn those policies. This is slightly different from the setting we designed our transformation for, and in that sense, it also challenges its generality. The used deep RL algorithms are the double deep Q-networks (DDQN) (Van Hasselt et al., 2016) algorithm for the cart-pole and the advantage actor-critic (A2C) algorithm (Mnih et al., 2016) for the reacher. both cases, we tune the learning rate(s) and the discount factor. We adopt the code from Nguyen et al. (2020), only adding the ergodicity transformation but without changing any parameter settings.
We show the mean and standard deviation of the average return general, non- parametric transformation proposed in this paper achieves comparable performance the tuned Sigmoid from Nguyen et al. (2020) on the cart-pole system and can outperform it on the reacher. This shows that while BOIL relies on intuitive arguments to develop a parametric transformation, we can achieve at least an en-par performance with a non-parametric transformation motivated from basic principles. Further, Nguyen et al. (2020) showed significant benefits of BOIL over existing hyperparameter optimization methods based on Bayesian optimization. Thus, coming up with reward transformations, in general, can significantly enhance learning. While the transformation in BOIL is designed for specific setting, transformation has more universal character and is applicable in more diverse settings.
five training runs in figure 6. The
A.5 Hyperparameter choices
The hyperparameter choices for the experiments in section are provided in table 1.
23

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Hyperparameters for the experiments section 7.
Cart-pole
Reacher
Discount rate
0.99
0.99
Training episodes
1000
500
Test episodes
100
100
Training episode length
100
200
Test episode length Epochs
200
200
10
10
Nodes in the actor neural network
16
64
Learning rate
0.0007
0.001
24


Reviews:
[{"review_id": "OLrzJS3ikU", "reviewer": "Reviewer_cFTQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The considered problem of heavy tailed rewards distribution from the lens ergodicity is interesting. It is good to motivate such a discussion in the community.\n\n2. The paper presents a variance stabilizing transform as a mechanism to fix the problem of non-ergodic rewards. Further, the paper shows with simulations that the proposed approach works.\n\n\nWeaknesses:\n1. I could not find why the setup is non-ergodic (in rewards). To convince myself, I verified the result with simulations, I realized that the multiplicative scaling of rewards R(t) = f(R(t-1)) was leading to non-ergodicity here. Especially, after taking the logs of the rewards, it was much clearer that the expected value of the cumulative sum log of the reward function is negative, but the cumulative sum function is positive. Adding some details will help readers better understand the simulation and the problems with non-ergodic rewards.\n\n2. How is equation 6 derived for the discrete time case? The paper first need to provide the transformation setup of Peters & Adamou (2018) to motivate the Equation 6.\n\n3. In an RL setup, R(t_k) is dependent on the policy. However, no such policy is mentioned in the expectation computation of $\\mu^2(u)$. How/Why does the proposed transformation work for any mis-specified policy based expectation? This is the biggest challenge I think this work faces. See example of variance measured for a particular policy here: \n\nLa, Prashanth, and Mohammad Ghavamzadeh. \"Actor-critic algorithms for risk-sensitive MDPs.\" Advances in neural information processing systems 26 (2013).\n\n4. What are the challenges in studying the gap between the expected untransformed rewards and expected transformed rewards is not clear? Also is their a gap on using transformation for any arbitrary policy?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers a reinforcement learning setup with non-ergodic rewards. This setup is closely related to Risk Aware RL where algorithms and their analysis are tuned to guarantee performance of any RL agent instead of average performance of large number of RL agents. This setup is particularly important with heavy tailed reward distribution where optimizing over average cases may be detrimental of the larger number of agents. It can find use cases in RL based LLM alignments to ensure average behavior from head tokens does not dictate behavior of long tail tokens.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition to the weaknesses above:\n\n1. In the paragraph below Eq. 8: \"Logarithmic returns (or regrets) are common in the RL literature.\". Please cite sources for this.\n\n2. It would help to lay out a proper step by step algorithm for \"Ergodic\"-RL. You can use a meta algorithm which estimates $\\mu^2(u)$, transforms the rewards, and then applies off the shelf RL algorithms (PPO/REINFORCE), etc.\n\n3. After multiple re-reads, I finally understood the objective is to construct a transformation such that the increments grow linearly and hence simply taking $R'(t+1) = R(t+1) - R(t)$ does not work. Please specify this discussion appropriately in the abstract and introduction to strengthen the paper and also simulate the toy examples with this to further strengthen the result.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8Y3ZPh9OTS", "reviewer": "Reviewer_2U9z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\\\nThe paper is in general well-written with a clear motivation. The authors nicely explained the issue of non-ergodic rewards with plain words and a toy example, backed with sound theory in the appendix. I believe the paper is of interest to the audience in safe/robust RL and ergodic RL and I learned from it.\n\n**Weakness**:\\\nOne apparent weakness is the lack of convincing experimental results. As the proposed solution is simple (logarithmic/learned variance), it should not be difficult to test them on more sophisticated algorithms and environments that come with the stable-baselines library. If compute resources are of concern, then I believe the authors should compare with the baseline case in coin toss with $\\tilde{r}(t_k) = R(t_k) - R(t_{k-1})$. This baseline gives the agent a sense of reward difference and we should see it improves learning but still underperforms the proposed log-difference if the theory is correct.  \n\nMoreover, it is quite unclear to me how Cart-pole and Reacher in Figure 4 relate to the central theme of the paper, as their rewards do not have \"the loss\" part as the coin toss example does. The authors did not explain why the original REINFORCE fail on reacher either. In my opinion, to strengthen this part, it may be wiser to try a modified reward function that explicitly reflects such non-robustness; or just remove Figure 4 totally and replace with theoretical insights/discussion from the appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigated the context where optimizing expectation of returns could result in unsafe policies attaining high return in rare cases but fail most of the time. One of the potential causes could be non-ergodic rewards and authors investigated it using the coin toss example. Through eqs 4 and 5 the authors explained the ergodic rewards maximizing the expectation of which is equivalent to maximizing growth rate.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the above my requested changes.", "broader_impact_concerns": "not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "czh6s27lUh", "reviewer": "Reviewer_LhKg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The focus on long-term performance over ensemble averages is highly relevant for deploying RL in high-stakes, real-world domains.\n2. The identification and formalization of the non-ergodicity problem in RL are novel contributions.\n3. The coin toss game example effectively illustrates the impact of non-ergodicity and the advantages of the proposed transformation.\n\n**Weaknesses**\n\n1. It's unclear how prevalent the heavy-tailed return distributions issue is in RL problems. A broader discussion on environments that naturally lead to long-tailed reward distributions would strengthen the paper. Additionally, exploring the relation to reward hacking—where agents exploit specific reward structures—could provide valuable context.\n2. The potential connection to causal inference concepts, such as false correlations, also worth discussing. Agents might mistakenly associate certain actions with high rewards due to randomness rather than causality. Discussing this aspect could enhance the paper's impact.\n3. The paper suggests that risk-sensitive RL methods typically rely on fixed transformations. If the proposed method generalizes existing approaches, the claim that risk-sensitive RL is limited to fixed transformations may not be entirely accurate. Clarification on how this approach extends or differs from traditional risk-sensitive methods would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the challenge of non-ergodic reward processes in reinforcement learning (RL), highlighting that optimizing expected returns across many trajectories can lead to poor performance in individual runs. This issue is particularly critical in real-world applications like autonomous driving and financial trading, where robustness is essential.\n\nTo tackle this problem, the authors propose a novel method that transforms non-ergodic returns into ergodic ones using learned transformations. This approach allows standard RL algorithms to optimize long-term performance without modifying their core structures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The example in Section 3 is intuitive and effective, but there appears to be some redundancy with the previous section's setup. Streamlining these sections could improve the paper's flow.\n2. In Section 4, the authors approximate the transformation to behave like a variance-stabilizing transform. However, the implications of this approximation are not fully discussed. Elaborating on how this impacts the robustness of the learned policy would enhance understanding.\n3. While the authors mention that ergodic Markov Decision Processes (MDPs) and ergodic rewards are related but defer formal analysis to future work, including a brief discussion on the relationship between these concepts in the current paper would be advantageous.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a resubmission of TMLR 2783. The manuscript has been improved overall, especially including a more comprehensive discussion on how the work relates to previous work in risk-sensitive RL and how the concept of reward ergodicity relates to the most common MDP ergodicity.

The reviewers, beside noting some area for improvements over the clarity of the manuscript, generally agree that this paper is a valuable contribution to TMLR and advocate for accepting it.

My recommendation is to accept the paper upon a few additional changes to be integrated in the final version:
- The title could be modified to give a clearer account of the paper's contribution. This is ultimately the authors choice, but something on the lines of "Reinforcement learning with non-ergodic rewards: Robustness via ergodicity transformation" could be considered;
- The contribution of the paper is in general very tied to the coin-toss problem. The problem per se is interesting, but the paper could clarify early on (e.g., in the abstract) that this is the main object of the analysis. Moreover, it is not as established as, e.g., LQR, so a description of which applications may benefit from solving this problem would be useful;
- Others:
    * Figure 1 only shows "bad trajectories", which does not "visually" validate the reported expected return;
    * Risk-sensitive RL section looks underdeveloped. For the exponential utility transformation, it does not clarify which $\beta$ values have been tested. Is there a value of $\beta$ for which $F \in (0, 1)$? Other approaches in the risk-sensitive RL literature seems to potentially solve the coin-toss problem, such as the mean-variance mentioned in the paper and CVaR optimization (e.g., https://proceedings.neurips.cc/paper/2015/hash/024d7f84fff11dd7e8d9c510137a2381-Abstract.html)
Decision: Accept (poster)