Risk‑Seeking Reinforcement Learning via Multi‑Timescale EVaR Optimization
Authors: Deep Kumar Ganguly
Ajin George Joseph
Sarthak Girotra
Sirish Sekhar
Venue: Accepted by TMLR ()

Abstract: Tail-aware objectives shape agents' behavior when navigating uncertainty and can depart from risk-neutral scenarios. Risk measures such as Value at Risk (VaR) and Conditional Value at Risk (CVaR) have shown promising results in reinforcement learning. In this paper, we study the incorporation of a relatively new coherent risk measure, Entropic Value at Risk (EVaR), as a high-return, risk-seeking objective that the agent seeks to maximize. We propose a multi-timescale stochastic approximation algorithm to seek the optimal parameterized EVaR policy. Our algorithm enables effective exploration of high-return tails and robust gradient approximation to optimize the EVaR objective.  We analyze the asymptotic behavior of our proposed algorithm and rigorously evaluate it across various discrete and continuous benchmark environments. The results highlight that the EVaR policy achieves higher cumulative returns and corroborate that EVaR is indeed a competitive risk-seeking objective for RL.

Published in Transactions on Machine Learning Research (10/2025)
Risk-Seeking Reinforcement Learning via Multi-Times EVaR Optimization
Deep Ganguly
cs22x500@hitpp.oo..
Deportment of Computer Science and Engineering Indian Institate of Technology Tirupati
Sarthak Girotra
cs206037
Department of Computer Science and Engineering Indian Institute of Technology Tirupati
Sirish Sekhar
cs20b043@iltp..oom
Department of Computer Science and Engineering Indian Institute of Technology Tirupati
Ajin George Joseph
ajin@ittp.oc in
Deportment of Computer Science and Engineering Indian Institute of Technology Tirupati
Reviewed on OpenReview: https: // openreview. net/ form?? id= AnDEgNDsis
Abstract
Tail-aware objectives shape agents' behavior when navigating uncertainty and can depart from risk-neutral scenarios. Risk measures such as Value at Risk (VaR) and Conditional Value Risk (CVaR) have shown promising results in reinforcement learning. In this paper, study the incorporation of relatively new coherent risk measure, Entropic Value at Risk (EVaR), as high-return, risk-seeking objective that the agent seeks to maximize. We propose a multi-timescale stochastic approximation algorithm to scek the optimal parameterized EVaR policy. Our algorithm enables effective exploration of high-return tails and robust gradient approximation, to optimize the EVaR objective. We analyze the asymptotic behavior of our proposed algorithm and rigorously evaluate it across various discrete and continuous benchmark environments. The results highlight that the EVaR policy achieves higher cumulative returns and corroborate that EVaR indeed competitive risk-seeking objective for RL.
1 Introduction
Sequential decision making modelled as a Markov decision process (MDP) Puterman 2014 is the underlying formulation of Reinforcement Learning (RL) (Sutton Barto 1998.. The primary objective in RL is typically to maximize the expected cumulative return, which may be discounted, undiscounted, or averaged (Puterman, 2014), depending on whether the horizon is finite or infinite. Optimizing for expected return has demonstrated remarkable success in structured and less volatile applications such Atari games, board games, scientific
experiments [Chen et al.) 2022). and other regularized, simulated environments (Shao et al. 2019 Sethy ct al. 2015 Silver ct al. 2016 due to stable and predictable environment dynamics. This approach often struggles es in environments characterized by high uncertainty and variability in returns, such as safety-critical
systems (Zhang et al. 2020 Martin H. de Lope 2009), finance (Filos 2019 Cao et al.] 2021), navigation (Wu al. (2024). industrial automation, healthcare (Wang al.] (023)), and robotics (Kober et al. 2013 Gu et al.) (2023), where robustness and adaptability are essential. In such high-stakes scenarios, accounting for risk becomes crucial.

---

Published in Transactions on Machine Learning Research (10/2025)
To handle these challenges, reinforcement learning must go beyond simply maximizing the expected return. Instead, it should incorporate risk-awareness by modifying the objective to account for return variability, giving rise to risk-sensitive RL (Garcıa & Fernández 2015 Howard & Matheson 1972bp. The standard expected cumulative reward criterion does not inherently avoid rarc but severe negative outcomes, nor does it consider the impact of large positive rewards. Consider two policies TT and T2, where P., (Reward = =-10)) 0.5 and F=, (Reward- + 1100) 0.5. Also, P7₃ (Reward = -1) 1.0. Although Ti has a higher expected return, it exhibits extreme volatility, leading to highly variable outcomes which can be categorized as risky. In contrast, T2 provides stable and predictable outcome. The expected cumulative reward criterion would favor Ti despite its high risk, highlighting the need for alternative evaluation metrics that account for risk. Classical RL objectives based on expected cumulative reward can be classified as risk-neutral, meaning they do not explicitly consider the uncertainties associated with actions. In contrast, risk-sensitive decision-making can be divided into two approaches: risk-averse RL the agent prioritizes stability, favoring policies with low reward variability, which crucial safety-critical applications, and risk-seeking RL the agent pursues higher mean returns, even at the cost of greater potential losses, resembling human decision-making palterns in portfolio management (Gollier) 2001) and super-human racing AI (Wurman et al. 2022 Kaufmann et al.) 2023), explained by cumulative prospect theory [Tversky & Kahneman 1992). By integrating risk-sensitive objectives into RL, agents can achieve balance between maximizing expected rewards and managing uncertainty. This leads to more robust and adaptive behaviors in dynamic and high-risk environments.
Risk Measures: In this paper, we assume the existence of a probability space J,P),, where Q represents the sample space, F o-field over Ω and P signifies a probability measure over F. In this paper, we consider random variables defined over this probability space. Several risk measures are commonly used in decision- making, each with different assumptions and applications. These include the Markowitz Mcan-Variance risk measure (Markowitz & Todd 2000),, which assumes that returns follow normal distribution and balances expected return against variance. Another approach is the Wang transform function (Wang 1996), which distorts the cumulative distribution function to model risk aversion or risk-socking behavior. More widely used in risk-sensitive optimization are Value at Risk (VaR) (Rockalellar et al.) 2000) and Conditional Value at Risk (CV&R) (Rockafellar et al. 2000; Rockafellar & Uryasev 2002). CVaR, and VaR, of random variable X at confidence level E [0, 1] are defined as follows"
CVaRa (X) = E X
X VaR (X)]: where VaR. (X) = sup B E R P B) > a).
(1)
(2)
While VaR, identifies a gain threshold, CVaR, provides a more comprehensive assessment by evaluating the expected gains in the best-case scenarios. This makes it particularly useful in risk-sceking decision-making, financial risk management, and safety-critical applications. Acceptance of the risk measures depends on the stability of their estimation procedures and the simplicity of optimization. The incorporation of these risk measures in policy optimization can be achieved either by modifying the objective function where cumulative rewards are transformed non-linearly using a risk measure, commonly, CVaR, (Chow et al. 2015 Tamar
el al. 2015 Kashima 2007 Keramati et al. 2020 Singh et al. 2020) or by considering the risk measure a constraint in the optimization setting, also commonly CVaR,, (Prashanth, 2014 Chow & Ghavamzadehi 2014 Zhang et al. 2024 Ahmadi et al. 2021 CVaR, is more widely accepted due to its coherent nature.
Eutropic value at risk (EVaR) AAhrndi-Javid 2012) is a fairly new risk measure based on the exponential moment of gains, derived from the Chernoff bound, that provides a convex and coherent lower bound on gains, making it particularly useful for managing tail risk. It is defined as follows: For a random return X E I. Entropic Value-at-Risk (EVaR right lail) is defined with a confidence parameter a E 10, as
(1 log
B e
EVaRa[X] [X] = inf
(3)
350
For the validity of the above definition, we assume that the moment-ggneratinn function Mx(8) = ex
exists for all / > 0. EVaR minimizes the worst-case bound on the right tail by optimizing over exponential
Throughout, we adopt right-tail (gain) definitions of VaR/CVaR/EVaR; maximizing them is risl-seeeking in the sense of preferentially targeting upper-tail returns.
2

---

Published in Transactions on Machine Learning Research (10/2025)
moment bounds. It acts dual to the Legendro-Fenchel transform of the cumulant function log E[e ensuring a convex upper bound on extreme right-tail values. From large deviation theory, the cumulant function has a dual formulation: log [[ = supper [Ee[X]- KL(E|IP)) Hence, we obtain the dual representation of E as follows:
EVaR. [X]
inf sup 8=0 &
(B&[X] (L(E|IP) log
sup inf & 0==
Ee[x]
[L(E)PP) log a
3 3 sup Eg with (L(E)IP) & log(1/a). Ç&P
From the above characterization, easily follows that, B(X) & EVaR, & esssup(X), where esssup = inf(æ R: PXX x) = 1} is the essential sup of X. Note that CVaR, XdlF where the probability measure Povaa A) IP(((X > (X)) o A)), for Borel set A. Furthermore, (L((PcvaR. PP) flog d S t log dPP log J. Also, Pecvaa << P. This leads to the well-known ordering of risk measures:
(X), & CVaR. (X) < EVaR. (X).
This ordering indicates that VaR provides the least conservative risk assessment, while EVaR offers the most robust and conservative measure, with CVaR serving as an intermediate risk measure.
For continuous r.v.S., VaR, estimate follows the asymptotic normality A (VaRo; a(1-a) (V (Serlling 2009);
where / is the PDF. This implies that as one approaches extreme tails (a small), the variance the estimator becomes prohibitively large, making both VaR,, and CVaR, difficult estimate accurately. Indeed, VaR uses hard indicator 1(X > 8), while CVaR employs the same indicator to gate samples; both therefore discard [[1- - 1)1% of sample points. Consequently, their stochastic gradients exhibit order-of ceier-f--manituudee higher variance. EVaR is a coherent risk measure as it satisfies subadditivity, positive homogeneity, monotonicity, and translation invariance. Its convex formulation ensures both robustness and tractability, making EVaR more suitable for optimal decision-making in risk-seeking seltings. EVaR, avoids explicit tail estimation by leveraging the moment-gent gennerttngn function for its computation. It reweights the entire distribution with smooth exponential factors, yielding low-variance gradient estimates and a learning curve that remains both monotone and smooth.
Problem Statement: In this paper, our objective is to seek optimal decision-makine making under uncertainty, which is modeled as a Markov Decision Process (MDP). An MDP is defined by the tuple A, R, P,r), where S and are the finite state and action spaces, respectively. R: A x S - R is the reward function, where R(s, a, s') represents the reward received for each state transition from s 4 s' taking action a. The transition probabilities are P S x A A where A is the probability simplex in IR and for particular state-action pair, P(-|ss a) is the transition probability, Po(.) is the initial state distribution, and y E [0, 1) is the discount. factor. For each state 8, the set A(s) gives all available actions. A stationary policy x-1ss) is a probability distribution over actions that depends on the current state s. Here we consider parancterized stochastic policies, which are parameterized by a p-dimensional vector 0, which means that the policy space can be written as Il = {Te(-(s), E S 0 e θ c RP). In this paper, we consider the following risk-seeking control problem:
8* arg max JEVaR 6E
EVaR., [[[T]
1
Eraat [R()
arg max inf log
(4)
350 3
where R(T)) 1-'-1 R(ss,, ar, St+1), with So ~ Po, a (s1 PP((|s) ai), and T E N. Note that (B) ()) exists for V0, since R(-) is bounded.
The entropic value-at-risk is inherently connected to exponential tilting, a technique that modifies the probability distribution to emphasize higher-risk, high-reward outcomes. To illustrate this connection,
3

---

Published in Transactions on Machine Learning Research (10/2025)
CVaRa
EVaRg
3
40
Risk Measure Value
30
1
20
10
-1
-2
0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
(Confidence Level)
a (Confidence Level)
Figure 1: Comparison of VaR, CVaR, and EVaR computed for for heavy-tailed (Pareto) and light-tailed (Normal) distributions, showing EVaR's strouger tail-risk sensitivity. The canonical ordering Va & CVaR, & EVaR, holds, with heavier tails accentuating the EVaR gap poo-prrviiin a sharper tail-aware nature that prioritizes trajectories with sparse, high returns.
consider the exponentially tilted probability measure Qa, defined
dQ
gr
(5)
dPP na
E L [(()))
This transformation effectively reweights the original probability distribution Pry, increasing the probability of trajectories with higher cumulative rewards R(T). As a result, the modified measure Q5 gives greater importance to risk-socking outcomes, ensuring that extreme rewards are more hcavily considered. Using this tilted distribution, we obtain the following relationship between the KL divergence and the EVaR objective:
KL (P.) = BEgs R
log Ey [M(T) and
1
JEvaR
= inf
Eqa [
|L(Cs||P
log a
000
B
dQ,
Br
sup Qe.M
[
KL (
& log
where Mg = Qo >3>> 0,
dIP xo
E T
[eBR(T)]
This formulation highlights that EVaR selects the extreme expectation constrained by KL-divergence bound (Kl-ball of radius log (1/a))). In other words, it finds the most extreme risk-soeking expectation while ensuring the alternative probability distribution remains within reasonable divergence from the original measure, thereby maintaining robustness in decision-making under uncertainty. The EVaR objective does not merely maximize expected returns but incorporates a risk-seeking adjustment that prioritizes high-reward yet riskier trajectories. Further, the KL-divergence constraint prevents excessive deviation from the original probability distribution, ensuring balanced trade-off between exploration and risk seeking.
set of tilted distributions, cach
The field of risk-sensitive control and reinforcement learning has been well studied, beginning with the seminal work of (Howard & Matheson 1972ah which introduced the application of an exponential utility function lo rewards. A substantial body of research has concent on risk-sensitive control in continuous- time, Snite-horizon settings, particularly for problems with known transition kernels
Related Literature:
FFleming McEncancy 1995: Whittle 1990 Coraluppi & Marcus 1999 Koenig Simmons 1994). Early
developments extended optimal control techniques to reinforcement learning (Littman & Szepesvári 1996
Borkar 2001 2002 2010
Mihatsch & Neuneier 2002
Heger 1994). The use of exponential ergodic
4

---

Published in Transactions on Machine Learning Research (10/2025)
0.30
Policy A (Gaussian)
B (Low Tilt)
Policy B (Exponential)
150
1.1 (Moderate Tilt)
0.25
Policy A (Tilted, 8
Cumulative Reward
125
1.5 (High Tilt)
Policy B (Tited, 8-0.6
Density
0.20
100
0.15
75
50
0.05
25
0.00
-2 0
20
40
60
80
100
Return
Training Episodes
(a)
Figure 2: (a) Plot shows exponentially tilted distributions for two RL policies with moderate risk sensitivity. Policy A remains stable, while Policy B shifts siguificantly, emphasizing its long tail. Tilting reweights probability mass toward high-reward outcomes, enhancing risk-seeking learning. (b) Low / encourages exploration with stable learning, moderate & balances exploration and risk sensitivity, while high f prioritizes high-reward strategies but introduces higher variability.
performance in discrete time control (Di Masi & Stettner 1999 2007 was further generalized to address risk-sensitive average cost criteria (Cavazos- Cadena & Hernandez Hernandez, 2011 In reinforcement learning, risk-sensitive methods have been adapted for model-free settings with unknown transition dynamics, employing
techniques such as relative entropy stochastic scarch and Q-lcarning (Borkar & Mcyn 2002 Osogami 2012
Recent advances in risk-sensitive reinforcement learning integrate conditional value-at-riss (CVaR) objectives for robust policy optimization (Chow et al.) 2015 Chow Ghavamzadeh] 2014), quantile temporal-differeen
learning to capture return distributions more accurately (Rowland et al.j (004)) leverage sample-basod dynamic programming techniques that augment Bayes-adaptive MDPs with CVaR constraints derive
risk-averse policies [Rigter et al. 2021 , and incorporate coherent risk measures alongside non-linear function approximation Lam et al. 2022). EVaR policy optimization in RL is relatively new and less explored area.
(Ni & Lai 2022 proposed a trajectory-basee policy gradient method to optimize EVaR-induced risk-sensitive
crileria, Dixit et al. 2021 developed nested EVaR-constrained models, and (Hau et al., 2023) introduced a
dynamic programming approach for EVaR objectives by formulating EVaR-based Bellman equations under
known transition dynamics.
Our Contribution: In this paper, we provide an online multi-time scale stochastic approximation algorithm to estimate the EVaR of the reward distribution and also seek the optimal EVaR policy in the context of model-free risk-seeking reinforcement learning setting.
2 Proposed Method
In this section, we recast the control objective (4) into two tightly coupled sub-problems. Prediction demands, for any fixed policy parameter 0, online, sample-effccient estimate of the entropic value-at-risk (ev and optimization then needs dependable ascent direction built from those estimates to steer 0 towards an EVaR- optimal policy, We tackle both challenges simultaneously within a multi-timescale stochasti-apprroiinatii framework: a fast inner loop continually calibrates the EVaR estimate, while a slower outer loop performs simultaneous wewptutseptuuurburetee bation- based gradient ascent, which seeks the risk-seeking solution.
2.1 EVaR Estimation
In this section, we propose an online, multi-timescale approach to estimate EVaR. From Eq. 4 we have
log Ey ET-,o [ - log a)
JEVaR (f) = inf
(6)
020
5

---

Published in Transactions on Machine Learning Research (10/2025)
H(B) for truncated Pareto reward (shape=3, scale=1)
G(x) for truncated Pareto reward (shape=3, scale=1)
6 0.01
0.01
0.05
10
0.1
G(x) = H(1.)
300
200
0.00
0.50
1,00 1.50
1.25
1.75
2.00
1.75
B
X
(a) ((8) for different confidence levels
(b) G(x) (1/2x) 8s. x for different confidence levels
Figure 3: Dependence of the EVaR curves on the confidence level for truncated Pareto reward distribution (shape= 3, scale= 1). (a) The function H(3) diverges more steeply as B - 0 for lower a, indicating stronger lail-risk sensitivity. (b) The convex transformation G(æ) = ((1xx) attains its minimizer al larger æ as a decreases, reflecting a more conservative risk-seeking nature.
Let H(3) = 8- (log [(r] log logg), where 3 > 0. We establish the following result for H.
Proposition 1. The function G(3) = (b) is convex in over B 0.
Proof. Given, H (1) (log E B A31 + (1- (1- Define normalized weights:
log az)
To prove convexity, for any À E [0, and B1, 12 0, let
x
(1
A)
v
=+1=1.
B
3
By Hölder's inequality:
R(+2
R(+)
Ri+)
IV
71
E
P2
Taking logarithms:
R(r) D
m(+)
R(c)
log
E
< 4
log E
e 01
+ 1 log E
P2
Multiply by and subtract log
1 AH
1
1
1 (
<
(1
2)H
3
B1
32
Thus, H (1) is convex in B.
The original function H(3) exhibits mixed curvature due to the 8-scaling in the denominator and the 3 reparamctorizes the function, flipping the curvature to enforce convexity. This property is critical because it simplifies the optimization landscape, making it much easier and more reliable to find the unique global optimum.
exponential term. However, the transformation / - 1,
Corollary 1.
min H(3) min G(z),
850
Morcover, the minimizers satisfy 8* 1/1*.

---

Published in Transactions on Machine Learning Research (10/2025)
Proof. Define the change of variable B. Then G(x) H Let 3 be global minimizer of H(3)). Since, 3 minimizes H, we have for any x > 0,
H(3) and x= = 1/8". Then G(x*) = H 20 H(3*) < H( G(z). Hence, G(x*) = ((*)) & G(x), vr> O. Thus x is global minimizer of G, and
min G(x) & G(x*) = H(3)) min H()). =>0
R>0
Conversely, suppose x= minimizes G(z). Define 3 1/x". By an identical argument,
min (33) & 113²) G(x*) = min G(z). 8=0
2>0
Putting these two inequalities together,
min H(3) = min G(z).
8=0
2>0
Furthermore, the minimizers match by 3* 1/x"
To establish the convergence and stability of our approach, we require certain rogularity conditions on the risk-seeking objective and the policy-induces reward distribution. These conditions ensure that the EVaR optimization problem remains well-posed and that the gradient estimates concentrate sufficiently around their expected values.
Assumption 1. The variable 1/3 is restricted to compact set 1 [mmine mmax] with 0 < Emin max 8..
Assumption 2. Varg. (R(r)) [R. (E (Fe)) (R()) a0
There erists a > 0 such that for all x E L,
dQu
c//
where the exponentially tilted probability measure Qz is defined as
dPP
E En [
This assumption ensures that the reward distribution under the exponentially tilted mcasure Qr always retains a minimum level of variability. Specifically, it guarantees that the variance of rewards does not collapse to zero for any risk sensitivity parameter æ. This is critical because the policy search requires sufficient. variability in rewards to effectively explore the policy space and avoid degenerate solutions. Lemma 1. Under Assumptions 1 and 2:
The variance Varg. (R(T)) continuous x over I Tmin Tmax
30 > O such that Varq, (R(T)) > o, E
Proof. Let [xx) = E, [T The tilted expectation Eq, [R(T]] and Eq, [R(T)]] are given by:
[R(T] = E, Ex[R(r)e R(r)
Eq, [[[]]]]]] E, [R() R(r)
Z(x)
Z(x)
(ar I), the terms R(T)e R(r)/a and e are bounded. By the Dominated Convergence Theorem (DCT), E, R(()) E, ((T)))) and Z(x) continuous in
By ([R(F) < R and Assumption I 1-y
Rie
Since Z(z) > et ")aana > O, the ralios [z_ and Eg, [ continuous. Thus, Varq, (R(T))
Eq. [ (Eq, [R())])2 is continuous. By Assumption Vare. (R(T)) 2 a >>>00 0 for all é I. Continuity of Varq, (R(T) and compactness imply Varg, (R(T) attains its minimum on Let = minger Varg. (R(T)).
By Assumption 2 σ 0.
Theorem 1. Under Assumptions and 2, the function
G(x) = x (log E R(r)/w log
is m-strongly conver I [mminy ]max with modulus

---

Published in Transactions on Machine Learning Research (10/2025)
Proof. One can easily find the second-order derivative of G(z)) as follows:
G" (z) Eq, (2.[R(()]] (Eq, [R(T)))
Varç. (R(T))
x
By Lemma 1 Varq. (R(T)) > o 0. Since mmax:
G"(x)
m > O.
mmax
Thus, G(x) is strongly convex
We now propose a tines-l framework to estimate Jeva by solving the above optimization problem via finding the roots of the derivative G'. For this purpose, derive closed-form expression for G with respect to as follows:
(' = dr x (log Er [e
d
- log ] Er (e E, [R(T)) e BBT E, (
log
a
The interchange of L.] and d in the above equality is possible through the bounded convergence theorem. In practice, the exact expectations E, [r()// and E, [A(()ee(tr)) are often intractable to compute directly. To address this, we replace these expectations with online average estimates, computed from observed samples over time. Consequently, we arrive at a two-timescale stochastic approximation algorithu for finding x= arg min G(x)). In this framework, one timescale used update the estimate of G', while the other handles the solution update in the direction of the estimate. This separation enables more efficient and stable convergence of the gradient estimation process.
de
d1+1 0z + &c (e (e - -0))
where T141 Ty and the slep-size ò, E (0, 1)
(8)
Wt+1 wi & R(T 1)/s (R(T(+1) s,r)),
The above single timescale stochastic recursions estimate the expectations E=(cB(r)/a and E, [ which amount to estimating G'. Now we can apply these estimates to seek x by calibrating the iterates the negative direction of the derivative estimate as follows:
Tt+1 Tt & log Oz - where & (0, is the step-size. we
Note that the above recursion is maintained at slower time scale rolative to the recursions Eq. This is required because one needs a good estimate of G" to efficiently calibrate x4- This develops a bidirectional coupling where the recursion Eq. can be considered quasistatic w.r.t. recursion Eq. 8 This can be illustrated as follows:
E1+1 = 3t 8, E1 6,
De log
We
(10)
a
By stacking Eqs. (8), and 10 in vector notation, we obtain the following
Oet1
0z
R(Te=))/a Ir ((TT+))/ (R(T(+1) I(wrr) E (log de - we)
Wi+1
=
Wi
+ 5z
F++1
If we let lim &t 0, then from the above equation, one can find that while x, quasi-static, the estimates ir and oa get nearly equilibrated their mean-field limits 0* (æx) and w" (xx) respectively. This quasi-static equilibrium eliminates bias in gradient estimates.
8

---

Published in Transactions on Machine Learning Research (10/2025)
Assumption 3. We assume that the step-size schedules {ōthee and (Erhee real-valued, positive, deter- ministic, pre-delermined sequences, and they salisfy
Σ (8 + 4 = 0C,
- Σ TeN
Σ teN
0o,
lim 1-00 òe
Et 0.
Examples of such step sizes can be = ,, required as they are critical technical requirements for ensuring almost sure convergence. The first condition ensures that they decay fast enough. The second condition ensures that updates happen throughout the entire time continuum. The final condition (time scale separation) ensures that the ratio of the step sizes must approach zero, ensuring the updates for 24 occur on slower timescale compared d, and wp.
lorlogt or 23/3 6 1/t. The above assumptions are
The following theorem characterizes the limiting behavior of the proposed empirical EVaR estimation recursion under the stated regularity assumptions and step-size conditions:
Theorem 2 (Convergence of JEVaR under static 0). Given policy To, under Assumptions 1-3, the coupled stochaslic recursions Eq.
& Eq. ( constitude slochastic Euler discrelization of the ODE system:
i E, Sooto R(T)// D,
w E Exana
R(T))
IT (((())/z)
E, Eyano [ [<((T)/
E Eynna no R R(()e R(T)/2
4 log
I.E (R(T)//z)
(*))))
Furthermore, the iterates satisfy:
(Ur, we, x1) 8.s. (0)),,
with equilibrium (0), (0)) characterized by:
E Erawo ((T))
(*()) E Er~ro (0))) w*(()=
and,
x* x*))**((0)
xx (0) E foR 0}.
Proof. (Sketch) We analyze the coupled stochastic recursions for (De, wr) on the fast timescale and æ, on the slower timescale.
Fast recursions.
The updates admit the stochastic approximation form
0c+1 0. + (br( Mc-1), Wi+1 we + Se (wr) MH+1),
with drifts
h2(0) E R(r)/2 0, h²(w) E R(( w (((T)/
[[
and martingale-differenee noise terms (Mr), (MH) adapted to (F.). Both drifts Lipschitz, and stability of the iterates follows from the
theorem. The limiting ODEs are linear,
و Elee El "(>\/=] 9,
e (())) - w Ele Ele [(T)/
which converge globally the unique equilibria
E Er (R(T) (
0 (I. 8) = [ (TT)/)/
w* (x,0)
E Erang [((())/
9

---

Published in Transactions on Machine Learning Research (10/2025)
Slow recursion. On the slower timescale, the update for xt tracks the ODE
④ = - = =-(10g log 1 (2, (1 wn (a,
By Theorem 1, G(x) is strongly convex and hence has a unique minimizer x* (0). Using G(z) as Lyapunov function yields
d
dt G(az) - (G'(ax)) < s0
with equality only at (0). Thus - x² (0) almost surely. Joint converyence. Timescale separation ensures (0,,w) track (2r. w) 0)) as os evolves. Conse- quently,
(e) - (z** w (x (0), x (0)) almost surely.
The argument is standard multiltimescale stochastic approximation. On the fast timescale, the auxiliary variables 02 and wt rapidly converge to the exponential moment and its tilted expectation, namely (* 0) and w (2, 0). On the slower timescale, evolves according to the gradient flow of the EVaR objective G(x:), which is strongly convex and admits a unique minimizer x* (0). Because the fast variables equilibrate much faster than ia changes, the overall system converges to the point
(0 x (x)))))
(x (0),)) x (0)),
ensuring almost sure convergence of the algorithm.
For the detailed proof, please refer to Appendix
Corollary 2. For a given policy me, as t D, the iterates {x²} satisfy the following
lim G(æ,) = JevaR((). (
(11)
600
Proof. Follows from the continuity of G and Theorem
The asymptotic convergence guarantees established above do not quantify the rate at which the estimators (Dr, (0,,w,,z σt) converge to their equilibrium values. To characterize the finite-time behavior of the estimators and provide explicit error bounds, we establish a non-asymptotic convergence rate here. The following theorem quantifies the mean squared error (MSE) for each estimator.
Theorem Let 8, ; & #, b, 1) and c, 0. Then:
1. Convergence of d.: For e> 3, Then for any fized > O,
B 00 2
Ki ir
with Ki > 0, Vi 1 and (2x 0) EE
e R(r)//>)
2. Convergence of wh: For 2E. Mrll/ Then for any fixed æ > 0,
E [ - (xx K2 for some K₂ > Vt and w (xx, Br-xo [R(()e
x 8 20* (I, 0)
8. Convergence of æs: Let d = (1-b) b and c max (1 2E [eo ). Under the Assumptions
1 and
3 we have
Kg E|1x, x= x=1
where x arg ming G(x), and the constant Kg 0.
10

---

Published in Transactions on Machine Learning Research (10/2025)
Proof. (Sketch) The estimator Dt converges to H* (x. 0) (/( with mean-square error O(t under polynomial step-sizes & e/t" as shown by Lemma 6
Similarly, Lemma / proves that we converges to w* ( Re with the same O(1²)) rate. Now, we proceed with the finite-time analysis as follows. [
(Slow recursion). On the slower Limescale, the update for see is
I++1 = It - (x) + me);
where the gradient error 7 1og(0,/0*) (wi w*) inherits its variance from the fast recursions. By Lemmas 7,
m =
Defining V/2 [ae x* and using strong convexity of G with parameter p 2/xmaxx one obtains the recursion
B[V++1] & (1- 2 + O(
(Rate bound). With & d/tb for be E (3, 1), the negative drift dominates the error terms. A comparison- sequence argument shows
B[V]] = O(t²),
yiclding the claimed rate
B[ - x 2
Kg 7
K3 > 0.
The fast recursions (Lemmas 6-7) guarantee that ú2 and track their population limits with vanishing error O(t-)). The slow recursion for Ie is then essentially stochastic gradient descent on the strongly convex EVaR objective G(z),, perturbed by this decaying error. Because the inner estimates improve sufficiently quickly, the outer descent achieves a finite-time rate O(1-²). Intuitively, the method balances accuracy of inner coonnentiallstatiiiit estimation with progress of the outer descent, ensuring polynomial sample complexity guarantces.
For the detailed proof, please refer to Appendix A.2
The above result captures the precise varinne trade-off dictated by the multi-tinescal. approach. The fast coordinates (1),we must enter an 0(+-)-MSE band rapidly enough that their residual bias is negligible for the outer recursion, whereas the slow coordinate must move with step-size exponent b > so the outer estimate appear quasi stationary. Selecting b r destroys this quasi static regime, and would allow the perturbation to dominate, inflating the outer-loop variance and impeding convergence. Further, the choice of d balances the strong convexity constant and the time-scale separation b, while the choice of depends on the exponentially tilted reward distribution, which scales with reward variability.
2.2 EVaR Optimization
The gradient estimation of the EVaR objective with respect policy parameters employs the simultancous perturbation stochastic approximation method (Spall 1992), computationally efficient technique for high- dimensional optimization. This approach perturbs all parameters simultaneously using randomized direction vector A, e R circumventing the O(p) computational complexity of finite-difference methods. The gradient estimate is constructed as follows:
VoJ ( JEVAR (0 + c..i) - (EVAR c.Ai)
(12)
2c,
where > 0 with lim 0 and A. E H2 with each of the components are A. lid Bernoulli(tI) w.p. 0.5. Also, A, [A A7g A Since the true JEVAR is not available, we estimate it as follows:
JEVaR (0 G.Ai) - JEVaR (0 GAi) G(xt) G(x,)
VoJ EVaR (0)
20,4,
=xt log 0++ a I7
0,¹ log
(13)
a
11

---

Published in Transactions on Machine Learning Research (10/2025)
The algorithm maintains parallel estimators (0T w,"",""" ) and {0r, , , for the perturbed policies o, and 0 - Ar, respectively, to prevent cross contamination of gradient signals. Each estimator executes N₂ inner iterations of the recursions Eq.(S) Eq. 9 per outer policy update, so that 0f and 27 approach stationtay values before the gradient approximation in Eq. is computed.
The random perturbations (with zero mean) can introduce variance gradient estimates, which gets asymptotically averaged over multiple random perturbations during the stochastic gradient recursion. As the number of perturbations increases, the average gradient estimate approaches the true gradient asymptotically, meaning that with enough samples, the variance in the gradient estimate becomes negligible and the estimate becomes incroasingly accurate. Also, this approach is advantagcous as it does not require explicit computation of individual partial derivatives. Instead, it estimates the gradient using only two function evaluations per iteration, making it highly efficient. Unlike finite difference methods that perturb each parameter separately, this method perturbs randomly chosen parameters simultancously, roducing computational complexity while maintaining a robust gradient estimate. Furthermore, it benefits from asymptotic unbiasodness as more iterations accumulate, the stochastic noise from perturbations cancels out, ensuring convergence to the true gradient. Finally, the update rule of the policy parameter using the above gradient estimate is:
G GT
where G. - x Inf (n)))), G xx In )
9t
A,
2c
0++1 - 0, + 91.
(14)
where Or, C. E (0, 1) are learning rate and perturbation parameter respectively. Now this procedure is illustrated in A Algorithm 1
Note that the above update rule can be rewritten as follows:
Oc+1 Os + ar (be + et + Pe + VJEeva(Oe)), where e VJ EVER E VJ J EVaR (0r) Fi)
b. E J. EVaR (EV) Fi]
(15)
and gr = z(0,-) 2(0,)
20, with z(0,t) = G(xx)) JEvaR and
2 x0,) G( JEVAR (oman(0,)) (0,).
The SPSA-based gradient update in our algorithm asymptotically mimics true gradient descent by ensuring three key error terms vanish over time. Bias (b+), arising from finiteddifference gradient approximations scales with the perturbation size c/. Noise (et), stemming from trajectory sampling variability, resembles SGD's minibatch noise. If step sizes de decay sufficiently, these zero-mean fluctuations average out due to the martingale structure, preventing erratic updates. Drift (9t), caused by finite inner-loop EVaR estimation, fades by increasing inner-loop iterations N, ensuring inner approximations align with outer updates. Collectively, these decays- biicaysbbas,, noise, and drift-ensure the update trajectory converges to the true gradient flow of JEvaR. guarantccing eventual convergence to an EVaR-optimal policy.
Assumption 4. For the stepsize sequences, 0i
0, a: 0. 00 Ei= at c0, and Ei of
The following lemma analyzes the behavior of the iterates 0r and we during the and 9g. We define the filtration (Filie, where the o-ficld F, = a (0;, Ai, 0f w x 1
phase of B,
Lemma 2. Let J (1) (0) 23 Jgvan/O0" ae agT crists and max,, 12.65 supø || [VeR 1121s
(0) < e. Then V0 e
by (0:) E
V (v
V J.vaa (.) F (c?)
Here we establish the convergence of the outer policy-update recursion:
Theorem 4. Assume the conditions mentioned in Theorem 3 Further, assume that ayKK < v.. Then the N
ilerates (0.) generated by Algorithm I satisfy the following:
0, E = V = =0 on the event (sup |0.l| < o0) as so
{
12

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm Multi-timescale EVaR optimization
Require: risk level E (0, 1), initial 8g RR. step-sizes {ar, Ct, 81, E.), inner lengths No
1: On - fo: 00 <- 0, w - 0, zo - 1, do - O. wo c 0, 30 <-
2: for t= = ...,, T-....--1... do
3: Draw A, {+1}² IID.
4: 0, - 0: + at A., or - 01 a D.
for 1, ,,...,,,. N₂ do
EVaR estimation for "4
Sample trajectory TeM Tot compute R - ru
of 0 + 5, R Ria/a -7)
7:
wt wi di [R Ri Riaa/ - zt wF)
9:
xt - xi - Er [n
w/t]
10:
end for
11:
for k ,,..., No do
EVaR estimation for "-"
Sample trajectory 22 no.. compute , En=O Tu
12:
of - or + 6:(e Ra/a 1aalii-0 -07)
13:
14:
wi wi + oe R R Th,ala5
27 wi)
oi + xi - Eo Em|11 -1) wil
15:
16:
end for
G x In _
G - Li In( (x)
17:
/1 - G - Gr A
18:
2a
19: Dc+1 0, Ot Dt
20: end for
21:
return 0r
Further, if & is a discrete set, then we have the following.
8, - V and V2 JZvan(( (0) & 0} on the event (sup |0,l < <0} as t - xc
Theorem 4 establishes almost-sure convergence of the outer uplicy-update recursion to the EVaR-critical set; when this set is discrete, the limit is locally EVaR-optimal policy. Equivalently, any accumulation point of the iterates satisfics the first-order stationarity conditions for the EVaR objective. Regarding hyperparameters, in our multi-timescale algorithm, &t: acts as a bias drift superimposed on the true gradient Jevaaa(O.). If Ft does not vanish fast enough for example, if e; decays too quickly or N; grows too slowly then this rosidual bias dominate the small gradient signal and prevent the iterates from converging to stationary point. In Theorem the condition o; (ce (c,N / - ensures that the product a, E[] vanishes, so that Pe only perturbs the trajectory transiently but does not alter the limiting ODE Å J_EVaR(()) and hence does not affect almost-sure convergence to the EVaR-critical set.
The aforementioned result indicates that the distribution parameters O, converge to the local maxima of the objective EVaR,, provided that the iterates 8, remain bounded, which denoted by the condition sup, |0,1| < oc. This condition is necessary because noise can cause the iterates gradually drift outward, potentially leading to divergence. This can be achieved by constraining the iterates remain within a convex compact set, and if they drift beyond its boundary, they can be projected back onto the set. The projected version of the recursion is follows:
0+1 O 0, (2Q,.) xt log 0+t ot)
xt (log
07 a
))}
(16)
13

---

Published in Transactions on Machine Learning Research (10/2025)
Dependency of 0; on Ne. ap. and
2,00
Vary N:
Vary d:
Vary Cr
0.75
0.50
0.25
0.00
10
10
Iteration t
Figure 4: Dependency the policy-paren parey- error [8, - -8 on the three key hyperparameter schedules a 50-state heavy-tailed MDP (Pareto(a 2) rewards, truncated 10, ? 0.95). Curves show the mean over 10 runs for: (i) inner-loop length N; (yellow), which reduces inite-difference bias and accelerates convergence; (ii) step-size a₂ 0-6 (orange), which maintains larger updates early at the cost of greater long-term variability; and (iii) perturbation size c t-0.3 (red), which aggressively shrinks gradient bias O(ct) and yields the fastest, most stable descent.
Jovn Update Trajectories
20
15
10
00
05
-10
Figure 5: minimal MDP with state space S (s1, uniform transition probabilities P(s a) = 0.5, and a quadratic reward R(s, a, s') - 101 yielding an EVaR objective Jevax(0) x 1 Gray arrows depict the true gradient field V.gvaa (0). Starting from 0 1.5] the orange dotted curve shows 20 exact gradient-ascent updates 0,+1 0g 0.1 J (0c), while the red dotted curve shows 20 SPSA-estimated updates using år. This comparison illustrates how Ôr tracks but noisily perturbs the true ascent path toward the EVaR-optimal parameter.
where (v) arg mingee || - -813 and θ is convex and compact. The above recursion can be rearranged as follows:
O4+1 17.0
0: + or E V (O.) V (JEVaR(Or)) Fg - VB(VJevaa(00 (eveaa(())) (0:) + VJEVaR
x(0,t - =(0T)
(17)
2a A.
-110 0, + ac (be be + sot (Jevaa
VJevaa(0) E
V Jeven JEveR(Oe (0.) Fe
(18)
14

---

Published in Transactions on Machine Learning Research (10/2025)
Let
[1 02
0. 00 + ar b1 + fpt + Jevaa (4) + Ex)
and uz
Qt
Then,
Bt+1 = 0: az [JEVaR(P)) (fx) + + by + spr ux].
(19)
Because 0141 minimises ||0- 1611l? over θ, the first order optimality condition gives
< ue, - 0c+1) 2 v0
- E Ne(0++1)
Ut No(0++1),
(20)
where (z) (v: (4, x> & 0, 40 E O is the normal cone at Thus, the correction produced by the projection lies in the negative normal cone at the projected point. Therefore, Eq. (19 can be considered as
B 8. at Jevana(O:) Ne(0(41) + 6++1 + + + by
(21)
Therefore, 0, will asymptotically follow the set- saluvel ODE (enaim 2006
Borkar 2009
E (e) -N)(O).
(22)
dt
Applying Corollary 4, Chapter 5 of Borkar 2009 to the stochastic inclusion Eq. 21
obtain
0₂ asa l" E (0 θ VJ(0) Ne(0) 0} (0 e(V .N(()))
i.e. every sample path converges almost surely to the set E. If the attractor set finite, then the iterates
{0r] converge a.s. to single locally EVaR-optimal policy 8* E.
Projected ascent iterates (Eq. 15) up to 9 descent flow backdrop
X6.
X6,
8
Xb
-1.0 -0.5
0.0
0.5
1.0
1.5
2.0
0
Figure 6: Projected ascent on the toy two state MDP. The return, R(T) = - so the risk seeking 0 Parameters are constrained to the shaded square θ (-0.6, 0.6]². We follow 0,+1 _ 0, a g) to produce the shown iterates 0g 0 02 - 02 (blue). Red x 's mark the updates = 0, + agg: grey arrows project them back θ, and orange wedges depict the outward normal cone No 10 at each boundary hit. The faint vector field in the background is the flow 0 0 of JEVAR.
objective is JEVaR (
Remark. To improve the qualily of the solution, one can inject a decaying Gaussian noise MMaryak & Chin 2008) into the iterates O2 follows:
xtt log 2t)
(og 01 ) qrEt
Gt 2c,
0 0,
(23)
15

---

Published in Transactions on Machine Learning Research (10/2025)
the slep schedule and Et sid (O, The noise term qez introduces rundomness inlo the update process, but this randomness controlled by 4t, which typically decreases over time ensure that the influence of noise diminishes. The fivite-difference gradient estimalor has controlled bias O and the injected noise qıEt adds exploration without affecting asymptotic convergence. The perturbation leaves the limiting ODE unchanged - so the interpolated trajectory remains attracted to E-but it alters the transient behavior of the iterates, supplying occosional random deviations that let the iterate escape any strict, non-global local optimum. When the noise is suitably behaved and certain other conditions are satisfied, the iterates 8, so generated converge to the global mazima of JEVaR:
where qt >
lim [e)] JBVaR(00))
(24)
Please refer to (Moryak Chin 2008 for the conditions required to ensure convergence.
ascont
0.5
6 00
0.5
1.-
-15
-2.
Figure 7: The stochastic ascent path (orange) jumps from the lower left peak to the higher right peak, while the deterministic path (grey) remains trapped. Gaussian kicks qı enable this valley-crossing, and once the iterate enters the right basin, ascent resumes toward the global maximum.
3 Experimental Results
We evaluate our method on both discrete and continuous-control benchmarks. For each environment, we report environment-specfic indicators including mean return, tail-risk metries, dispersion across random seeds, and learning-curve bchaviour. We also conduct selective ablation studies stepsize and perturbation schedules to isolate their effects. Complete implementation details, hyperparameters and reproducibility
artefacts are provided in D and C
3.1 GridWorld
We evaluate our algorithm in discrete e grid environment, where agents must navigate from a predefined start state to a goal state, avoiding obstacles. Each movement incurs a small cost, single misstep into the obstacle yields a high negative reward, while reaching the goal provides significant reward, encouraging efficient path sclection. To promote generalization and robust policy learning, obstacles are randomly placod in each training batch, preventing the agent from overfitting to fixed obstacle configuration. At the beginning of every training batch, the environment samples exactly Six obstacle cells uniformly at random
16

---

Published in Transactions on Machine Learning Research (10/2025)
except for the start, end goal. Here, the per-episode return is
1- Σ
R(()) =
1 goal Irmp
goal reward
where the summation is the -1 step-cost incurred at every time step until termination. Across 5,000 evaluation episodes and 200 distinct obstacle layouts we observe a markedly heavy tailed distribution. Indeed, a 82% trajectories avoid all traps, wander for 10-12 moves, and terminate with returns clustered near -11 (the mode); = 12% hit least one obstacle and end below -14, forming long left tail; the remaining 6% reach the goal along the eight step optimal path (return -8), producing sparse right tail. The pooled sample exhibits excess kurtosis 7.1 and Hill tail-index 2.3, attributes of heavy teive-tailed distribution.
Mean rewards
a 0.2
a = 0.15
a 0.1
a 0.05
0.01
Iterations
Figure 8: The start state fixed at (0. 0) [GREEN], and the goal is at (4,4) [RED]. [eftt] Plot shows the mean rewards obtained for various levels of the threshold for e [Right] Optimal path chosen reach the goal.
DaaR vietation
Risceneurral
800
2500
Probability mass
600
2000
Random
400
Risk-neutral
0.2
1000
EVaR
500
-14
-12
-10
-8
-6
-4
-2
0
Return bin
(a) State-visit frequency
(b) Return-probabiliiy heat-map
Figure 9: Tail-risk behavior in the 5x5 Grid-World. [Left:] The EVaR-optimized agent follows the safe diagonal corridor and rarely steps trap cells, whereas a risk-ncutral policy sprcads visits more widcly and still wanders onto traps, showing that it has not fully internalized low-frequency hazards. [Right:] Probability mass allocated to each 2-point return bin (darker = higher). EVaR optimized policy sharply suppresses the catastrophic bins (< 12), while shifting mass toward the settrratthhhhh-eetttr region (-8...-6)
The agent achieves the shortest collision-free path while successfully navigating random obstacle placements (Figure 8 by minimizing EVaR cumulative cost, demonstrating quantitative risk awareness. Maximizing EVaR exponentially tilts the reward distribution: under the tilted measure, every trajectory is re-weighted by exp( (p)()((T) (5). High-return events (R > -8) receive cxp(-8.3) times more emphasis than the modal -11 outcomes, whereas catastrophic returns (s 114) are suppressed by exp(-143). Subject to the KL constraint, the optimization therefore drives the policy to (i) avoid whichever six traps appear in the current layout and (ū) reach the goal quickly, because only such trajectories migrate probability mass into the right tail. By contrast, risk neutral agent weighs outcomes linearly; the rare -2 penalties are diluted by their low frequency, so the baseline oscillates wetween risky shoricuts (which sometimes intersect traps) and conservative detours.
We also evaluate our algorithm for various levels of a (0.2, 0.15,. 0.1, 0.05, 0.01). Interestingly, we observe that decreases, the agent. converges to optimality more slowly. This behavior can be attributed to the
17

---

Published in Transactions on Machine Learning Research (10/2025)
effect of the function (3): for smaller values of a, (3) exhibits flatter region near its base, forming saddle point basin. Indeed, as decreases, the log term dominates, flattening the curvature. As a result, the gradient-based optimization process experiences slower convergence to the true EVaR, requiring extensive exploration to discover rare high-reward paths to refine the policy. The policy spends more time sampling Irajectories to accurately estimate the tail of the reward distribution.
3.2 MuJoCo
We consider the OpenAl Gym environments LIVERTEA-DDUBLE-PPADDLLL and SWIMMER/VV /v4 from the MuJoCo framework framework (Towers et al. 2023). These canonical tasks span markedly different return geometries. INVERTED- DOUBLE-PENDDEE PENDULUM/V / v4 presents a hcavy-tailed mixture of frequent tip-overs and rare full-horizon balances, exhibits an even sharper bimodal distribution with sparse yet large terminal bonuses, while SWIMMER/V4 is almost Gaussian thanks to its smooth quadratic reward and fixed-horizon episodes. These contrasts let us probe how the exponential tilting behind EVaR trades off gradient variance and tail exploitation. All experiments use identical network architectures and optimizer hyperparameters: only the risk level a and the underlying lail structure differ, allowing unbiased comparison of EVaR against CVaR and VaR across light-, mixed-, and heavy-tail regimes. For comparability, EVaR is reported level a, whereas CVaR and VaR are reported at 1 - a, aligning the measures toward right-tail risk.
Tassa al., 2018) and MOUNTAIN- CAANNIAA-CAR-CONIIIIN WWW--OOOTNNNOSOOOOOOI from the Box2D Gym
MOUNTAIN-
CONTICNOTTUUUUS/VV COUNTAIN-CA--COONINUUUS
metr d-1.
metr am2.9
metr un0.9
Rick
metric
Iteratons
(4)
66
(a) evolution of risk objective during training and (b) realised return for EVaRo.1. CVaRo.9. and VaRo.9 optimal policies. The mixture-heary-tale return distribution-fii frequuuut early crashes contrasted with rare high-reward balance episodes ppisodes-faouurs EVaR's exponential tail re-weighting, enabling it to extract informative gradients from sparse succosses and converge faster than the quantile-basod objectives.
Figure 10: InvERTEEDDUSSLEEEDDUUU
Monsmuu Conrieinss
rik motric sw0.1
riX motric awo.
a=Q.9
rBk moticc var dway
dw0.9
Iteration
Herailona
(
Figure 11: MOUNTAIN LAR-COOTINUOUS. Direct comparison of EVaRo.1, CVaRg.9, and VaR9.9 for (a) their respective risk objectives during training and (b) average realised returns for their respective optimal policies. The entropic criterion climbs sharply and reaches higher plateau, whereas the quantile-basee objectives converge more slowly-illusie EVaR's advantage in the mixture-heavy-tail return regime characteristic of this benchmark.
18

---

Published in Transactions on Machine Learning Research (10/2025)
0.23
0.23
120
100
Ecoodo roum
return distribution: dense left mode (-90 to -70) corresponds to failed climbs that accumulate only per-step torque penaltics, while a narrow right spike at R + +100 represents the rare episodes that reach the goal and receive the terminal bonus. The resulting bimodal, mixture-heavy distribution illustrates the extreme positive outliers that EVaR amplifies, whereas VaR/CVaR initially down- weights ignores them explainnng the risk-criterion gap observed in subsequent learning curves.
Figure 12: COUNVAAN-CAR-CONIIIOOSS,
In MOUNTAIN- COOUNTIINCARR-ONTIIIN CAR-CONTINUOUS)) the per episode return R exhibits mixture induced heavy right tail rather than a genuine power-laye tail. An episode earns step penalty -0.1 of with lay| < 1 until the car's position surpasses the goal, at which point it receives a one-shot + +100 bonus and terminates early. Returns therefore follow
Frail [-100,- -70]. with prob. 1 P.
R
Pauce [+95, + 100), with prob. p.
Although the support is bounded, the spike at +000 produces high empirical skewness and kurtosis, making the distribution pseuto-heawy--ailled. Further, since supp( Rt) finite, the MGF exists v e TR and
log E an [ (1 - p) log Ee Mtaa + P. 100 + o(p)
is dominated by the +00 mass even for modest 3 > 0. Hence, EVaR(R) amplifies successful trajectories exponentially, yielding a sharp optimization signal once the first few successes appear. Also, EVaR uses all trajectories with exponential weights, so the gradient pivots towards the rare success mode after only handful of successful episodes; variance remains controlled because Rmax Rmin 200. However, both CVaR and VaR discard the boltom (1 (1-a0 a) fraction of relurns, and early gradients are driven solely by the narrow [-100,- -70] slab, providing little incentive to explore the costly "reverse treverse-swing" maneuver. Hence, the bimodal "mixture heavy tail" of
creates marked difference in gradient-sigee quality: EVaR converts the right-tail spike into high-amplitude training signal, whereas VaR/CVaR suppresses it until success becomes common, explaining the performance gap observed in Fig. 13.
Saimmer
evar metric cvur mak meint 2-0.9
Iterationa
Perations
Figure 13: SWIMMER/V4. In this light-tailed task, EVaRo.1 tracks CVaRg.9 and VaRg, for both objective (a) and average return (b), showing that sub-Gaussian rewards neutralize EVaR's usual entropic edge.
19

---

Published in Transactions on Machine Learning Research (10/2025)
The SWIMMER /v4 benchmark induces light-tailed, almost Gaussian return distribution because each time-step reward is smooth quadratic of forward velocity minus bounded torque costs, and the episode always runs its full horizon without absorbing failure states. Summing these sub-Gaussian rewards over T steps yiclds a cumulative return whose noment-generating function admits the second-order approximation
log DRJ
DR) -B 33
where j and o² are the mean and variance of R. Plugging this into the entropic risk functional shows that (Va,,, collapses to first-order correction of CVaR, (R), the gap shrinking to O(o² In(1/a)). Because the worst-ar tail well-populated (roughly 10% of samples for 0.9), CVaR and VaR enjoy low-variance gradient estimates while retaining nearly the same risk sensitivity sharpens tail control therefore offers little extra benefit, yet still incurs the cost of evaluating exponential moments, so SWIMMER/VE benign reward structure does not favour EVaR as illustrated in Figure 13
EVaR. The entropic tilt that normally
3.2.1 a-Sensitivity
EVaR's performance is highly sensitive to the confidence level especially in heavy-tailed reward scttings. Lowering a increases tail emphasis - the policy gradient effectively reweighs trajectories by an exponential factor, amplifying the contribution of top-return outcomes. This sharpened focus on the extreme tail can initially accelerate learning by extracting signal from rare high-reward episodes, but it also increases gradient variance when those outcomes are scarce. Conversely, a larger (weaker tail focus) yields more stable, low-variance updates by using a broader sample of returns, at the cost. of under-weighting The result is tradc-off between sample efficiency and stability: too aggressive tilt (small a)) may noisy gradients and convergence to suboptimal policies, while too mild tilt (large a:) converge slowly or miss the highest-return strategies. We examine this trade-off in two continuous-control domains with heavy-tailed return distributions, MOUNTAIN-CAR-COM MAAR-CONTINUOUS/VA and INVERIEE--DUBBLE-PE PEW.OOOBBBLEEEEUUU///// which feature mixtures dense low-return outcomes and occasional large returns.
payoffs.
InverteiDoubleennduuuvvv
110
375
rewards
obj.
0.1
a 0.1
0.01
0.01
0.05
a 0.05
iterations
iterations
(a)
(b)
Figure 14: INVERTED-DOUBLL PENUBBLR-PEDUUUUM/AAA. EVaR sensitivity to a. (a) EVaR, objective values during training for a {0.10, 0.05, 0.013 and (b) Average episodic return for the corresponding optimal policies. The balanced tail emphasis a = 0.05 ultimately surpasses both the more aggressive a = 0.01 and the conservative a = 0.10, finishing with the highest objective and return plateau. The extreme tilt (a = 0.01) accelerates carly learning but suffers higher gradient variance and platcaus lower, whereas the mild tilt (a 0.10) wnder-weights the rare high-return trajectories and converges slowest. These results indicate that intermediate exponential re-weighting achieves the best long-horizon performance in this heavy-tail environment.
20

---

Published in Transactions on Machine Learning Research (10/2025)
0.05
Probability mass
a
0.4
0.01
Risk
0.2
0.10
0
40
80
120
160
200
240
Return bin
Figure 15: Heat-map of risk-rcturn tradc-offs across confidence levels in LNVERFED-DOUBLL-PENNDUU The intermediate lail emphasis a 0.05 concentrates the greatest mass in the extreme-return band (* 240) while retaining spread across neighbouring bins. The aggressive tilt a = 0.01 narrows onto a single high- return mode (< 200), and the conservative the heat-map visualizes how balanced exponential re-weighting (a 0.05) best exploits rare high-value trajectories without sacrificing distributional robustness.
0.10 leaves most mass in mid-range returns (* 160). Thus,
Mous Lams-aap---- Costuccon
Bikk values
-35
0.03
0.05
150
Figure 16: MOUNTAIN- CAR- CONTINUOUS - EVaR risk level sweep. (a) Evolution of the EVaR, objective during training and (b) corresponding mean episodic return under the EVaR-optimized policy for risk levels a E (0.1, 0.05, 0.01). In this mixture-heavv-Lail domain, moderate tail emphasis maximizes reward: The a 0.10 setting balances sample efficiency and tail focus, ultimately delivering the highest risk-adjusted value and the best returns; the more aggressive tilts a 0.05 and a 0.01 overweight the scarcest +100 trajectories too early, suffer high gradient variance, and converge to lower plateaux.
MOUNTAIN- CAR- CONTINUOUS/v0 favored a higher a (more conservative tail weighting) to cope with its binary-success structure, whereas traded off learning speed stability. The underlying principle is that EVaR's exponential tilting should be tuned to the return distribution. An that is too conservative for given domain can lead to overly timid policy that fails to accomplish the task. Suppose the returns are extremely "spiky" (e.g., mix of very frequent low returns and very rare huge returns). In that case, an overly aggressive tilt will overweight those spikes too early, destabilizing training. Conversely, if the returns allow incremental improvements toward the tail, well-chosen intermediate can significantly improve sample efficiency by focusing the gradient on those improving lail outcomes. In all cases, a balanced helps manage the variance bias trade-off: it grants sufficient. emphasis the lucrative tail of returns to drive policy improvement, while still leveraging enough of the sample data to maintain the accuracy of the gradient estimate. This leads to superior long-run return performance and more reliable convergence, as evidenced by the learning curves and return distributions in our experiments.
benefited from mid-range that best
3.3 Comparison with CVaR-based Baselines
To quantify the empirical advantages conferred by the entropic risk mcasure, we compare our procedure (referred EVAR-SA) against three canonical CVaR-based algorithms: CVaR-PG (Chow et al.) 2015). SDPG-CVaR (Singh et al. 2020), and simplified D4PG-CVaR (Barth-Maron al. 2018). All four methods
21

---

Published in Transactions on Machine Learning Research (10/2025)
operate in fully controlled tabular setting with identical finite-horizon MDPs, tabular state-action value tables initialized to xero, e-greedy exploration (f = 0.1), discount factor 0.99, and fixed learning rate of 0.1. By limiting all algorithms to 500 episodes per seed (truncated at 200 steps) and averaging over eight independent random seeds, we ensure that any performance differential arises exchusively from the choice of risk criterion and its estimator, rather than from architectural capacity or extensive hyperparameter luning. The experiments are conducted on two benchmark environments. Cliff Walk (4x12) requires the agent traverse from the lower-left start cell the lower-right goal while avoiding a "cliff" of ten cells (columns 10) that imposes catastrophic penalty of -100 and resets the agent. All other moves incur
in the bottom cost of 11, producing a heavy-tailed cost distribution that challenges agents to avoid rare but catastrophic failures. Windy Grid World (7x 10) involves navigating from (3, 0) to (3, 7) under stochastic upward winds of column-dependent strengths {0, 0, 0, 1, 2, 2, 1, 1), with each step costing -1 until the goal. The random drift amplifies the likelihood of large deviations, making robust risk sensitivity desirable. More details in A Appendix
Navigation
Mart
Table 1: Mean final rewards and episode lengths (u o) over 8 seeds. EVAR-SA demonstrates robust perfor- mance under tail-risk, avoiding catastrophic failures in Cliff Walk and maintaining stable returns in Windy Gorld
3.0
1.0
Poation
Amgorithm
Final Reward
Final Length
Windy GridWorld: Adaptive Navigation
Cliff Walk
Wind=2
EVAR-SA
-47.23 + 12.67
17.39 + 0.92
Start
CVaR-PG
-20.53 + 0.96
19.09 + 0.20
SDPG-CVaR
-54.23 = 7.00
18.73 + 1.65
Posiion
D4PG-CVaR
-24.04 + 5.02
21.95 t 5.15
Windy World
EVAR-SA
-2..78 = 0.31
26.78 + 0.31
CVaR-PG -23.41 1.11
24.41 + 1.11
SDPG-CVaR
-26.14 0.49
27.14 + 0.49
D4PG-CVaR
-25.82 + 1.29
26.82 + 1.29
10
X Position
Figure 17: Visualization of environments (left) and performance summary (right). In Cliff Walk, EVAR-SA safely avoids cliffs, reducing catastrophic returns. In Windy GridWorld, it maintains adaptive paths under stochastic drift. Performance metrics show EVAR-SA achieving competitive rewards with tighter confidence intervals.
Notably, EVAR-SS occasionally sacrifices average reward (by 15 25%) in favor of reduced variance and worst-case performance. This safety efficiency trade-off is expected and desirable in domains like autonomous navigation or healthcare.
Table and Figures 18
20 show that EVAR-SA achieves consistently lower worst-case costs and tighter confidence intervals than all CVaR-based baselines.
Cliff Walk Figure 19 (top row) and Table report that EVAR SA achieves mean final reward of -47.2 + 12.7 and converges to an average episode length of 17.4 + 0.9 steps. In contrast, CVaR-PG attains -20.5 + 0.9 reward in 19.1 + 0.2 steps and D4PG-CVaR -4.00 + 5.0 reward in 21.9 + 5.2 steps both suffer frequent catastrophic resets into the cliff region. SDPG-CVaR is more conservative (-54.2 7.0, 18.7 but still undorperforms EVAR-SA Two-sample t-tests confirm that EVAR-SA's gains over CVaR-PG (A -26.7, p < 10-4) and D4PG-CVaR (A = 223.2, p < 40-4) are highly significant, with marginal advantage over SDPG-CV&R (p 0.07).
22

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Quantitative comparison of EVAR-SA and OVaR-based baselines on Cliff Walk and Windy Wordd Metrics are derived from experiment statistics (success rates, convergence speed, and variance). EVAR-SA exhibits strong risk aversion, rapid convergence, and conservative path efficiency.
(a) Cliff Walk
Metric
EVAR-SA
CVaR-PG
SDPG-CVaR
D4PG-CVaR
Risk Aversion
Strong (>-55%)
Moderate (68%)
Moderate (72%)
Weak (41%)
Convergence Speed
Rapid (17.4+0.99
Moderate (20.1+1.2)
Moderate (19.8+1.0)
Slow (22.0=1.7)
Path Efficiency
Conservative (+88)
Ncar-optimal (A -3%)
Balanced (A -1%)
Variable (High variance)
Exploration Spread
Low (Var=0.05)
Moderate ( (Var=0.15)
Moderate (Var=0.12)
High (Var=0.29)
Catastrophic Failure Avoided
Yes (0% failures)
Partial (22%)
Frequent (10%)
Partial (18%)
(b) Windy GridWorld
Metric
EVAR-SA
CVaR-PG
SDPG-CVaR
D4PG-CVaR
Risk Aversion
Strong (>95%)
Weak (43%)
Moderate (66%)
Weak (47%)
Convergence Speed
Rapid (15.2+1.1)
Moderate (17 1.3.
Moderate (17.0=110)
Slow (19.3=18))
Path Elficiency
Conservalive (+5%)
Near-optinal (A -2%)
Balanced (A -1%)
Variable (High variauce)
Exploration Spread
Low (Var=0.04)
Moderate (V3)=0.13)
Moderate (Var=0.10)
High (Var=0.25)
Catastrophic Failure Avoided
Yes (0% failures)
Partial (27%)
Frequent (8%)
Partial (21%)
Env.
EVaR-SA vs.
A
p-value
Sig.
CVaR-PG
-26.7
< <10-
Cliff Walk
SDPG-CVaR
7.0
0.0735
D4PG-CVaR
-23.2
< 10-1
CVaR-PG
-2.37
10-4
Windy GridWorld
SDPG-CVaR
0.36
0.0276
D4PG-CVaR
0.04
0.8967
Table 3: Reward differences (two-sample I-tests) where, A (EVAR-SA baseline) and 0.05,
p < 0.01, *** p < 0.001 denote significance.
Windy GridWorld: : In Figure 19 (bottom row) and Table 1 EVAR SA reaches 25.8 + 0.3 reward in 0.3 steps, matching or slightly exceeding the baselines in mean performance while dramatically reducing variance. CVaR-PG achieves -23.4 + 1.1 reward in 24.4 + 1.1 steps but exhibits wider dispersion, whereas SDPG-CVaR (-26.1 t 0.5, 27.1 + 0.5) underperforms in mcan return. D4PG-CVaR shows no significant difference (A = 0.04, p = 0.90) but suffers from high variability. In Windy Grid EVAR-SA delivers comparable mean rewards but exhibits dramatically narrower upper-tail cost distributions, highlighting its ability to enforce safety without sacrificing efficiency. The advantages of EVAR-SA arise from two key properties. First, its entropic objective inherently emphasizes tail-risk mitigation by placing exponential weight on the worst a-fraction of returns. This makes it particularly effective in safety-criticll tasks where rare catastrophic outcomes must be eliminated. Second, the use of finite-difference stochastic approximation and state-adaptive step sizing ensures smooth and monotonic lcarning curves, unlike quantile-based CVaR updates, which often oscillate when thresholds shift.. The computational cost. of this gradient estimation is modest, requiring only (11) additional rollouts per update. Also, Figure 19 illustrates that EVAR-SA's mean-reward and episode-length learning curves (smoothed +10) converge more rapidly and smoothly than CVaR-PG and D4PG-CVaR, whose muantile-based updates induce oscillations when thresholds shift. SDPG-CVaR's convergence speed lies between these extremes. Moreover, EVAR-SA consistently maintains superior 5th-percentile returns, underscoring its robust tail-risk control.
Figure 20 presents a multi-pana panel risk-seeking evaluation. In the boxplots (left), EVAR-SA attains both a higher median final reward and a markedly narrower interquartile range than CVaR-PG, SDPG-CVaR, and D4PG-CVaR, indicating stronger tail-risk mitigation. The scatter plot (center) shows EVAR-SA at the lower-risk (sid. dev.) frontier for comparable mean returns. Finally, the convergence curves (right) reveal that
23

---

Published in Transactions on Machine Learning Research (10/2025)
350
250
-06
EVAR-GA
226
SOPG-CVAR
150
220
250
290
100
300
139
200
-50
00
-0
EVAR-SA
200
CVaR-Pr
SOPD--CWW CVAN
DEPO-CVOR
402
150
108
Figure 18: Trajectory visitation heatmaps in Cliff Walk (top row) and Windy GridWorld (bottom row) for EVAR-SA and three CVaR-based baselincs. Color intensity represents the cumulative number visits per grid cell across all episodes and seeds (darker = more frequent). EVAR-SA exhibits concentrated visitation along safe paths, avoiding high-risk regions (cliff cells and wind-affected upper rows). contrast, CVaR-PG and D4PG-CVaR display broader dispersal, indicating greater exposure to tail risks. SDPG-CVaR exhibits intermediate behavior. Those results highlight EVAR-SA's capacity for robust tail-risk mitigation in stochastic environments.
D6-1 XDans
Serouthed Mear Peoord
38
Mgproo 5th Presertte Reware
Aepres 129 Deraedte Stweeet
D40 Yours
Exl (Dun)
JONC Chu
Dex Mean
Do Coor
(a) Windy Grid World
(b) Cliff Walk
Performance comparison across environments. (i) smoothed mean episode reward with #o confidence bands, (ii) approximate 5th-percentile return trajectories, and (iii) bar charts of final mean reward + standard deviation, for EVAR-SA and three CVaRt-base baselines over 500 episodes and 8 seeds. EVAR-SA converges faster with tighter worst-case guarantees in both Windy GridWorld (left) and Cliff Walk (right).
Figure 19:
24

---

Published in Transactions on Machine Learning Research (10/2025)
Clfff Welk: Reward Distribution
cliff Walk: Risk- Return Trade-off
Cliff Walk: Convergence Stability
-20
0000 con
-30
Hial Haaltpilaee bofoeeeeee noweed
-20
-30
-30
10 cos
-50
12
310
200
300
400
Windy GridWorld: Reward Distribution
GridWorld: Risk- Return Trade-off
Windy GridWorld: Convergence Stabilitt
-235
BWR
D4PG.CVUR
Hoa takkoodo
-40
Expected
-20
-23
-15
-7-
-233
-20
-260
Diaked
04
2.0
3.0
12
220
200
300
400
bed
R5kk15t Dovi
Eblodd
Figure 20: Risk-secking analysis. (Left) wox-ser-whisker plots of final episode rewards over 8 seeds: EVAR-SA achieves a tighter distribution and higher returns than CVaR-PG, SDPG-CVaR, and D4PG-CVaR. (Middle) Risk-return scatter: EVAR-SA attains lower tail-risk (std. dev.) for a given mcan reward. (Right) Convergence stability: EVAR-SA's finleediiflerence SA estimator yields faster, smoother learning with narrower to bands.
EVAR-SA's finieeddifeerence stochastic-pproxxination gradients produce smooth, monotonie improvement and tighter to confidence bands, unlike the oscillations observed in quantile-based CVaR updates.
3.4 Glycemic Control
We demonstrate our algorithm's ability to manage high-risk insulin administration for Type-1 Diabetes Mellitus (TIDM) using the Simglucose simulator (Xie) 2018), which evaluates EVaR's efficacy in safety-critical RL. Simglucose mimics real-world scenarios, providing a controlled enviroument to test control algorithuns before clinical deployment. Tn our experiments, a PID controller regulates insulin based on blood glucose levels, aiming to keep them within a safe range. We evaluate performance on both adult and adolescent patient profiles to minimize the risk of hyper- and hypoglycemia, illustrated in Figure 21 The glycemic control task typically uses highly nonlinear reward function to reflect clinical risk. Small deviations from the normoglycemic range might incur mild penalties, but crossing critical glucose thresholds triggers disproportionately large negative rewards (e.g., a penalty spike or episode termination when glucose < 70 mg/dL or > 250 mg/dL for a sustained period). This nonlinear penalty structure creates the heavy-tailed return distribution. Glycemic control is a high-stakes problem where inaction or overly conservative actions can lead severe health risks such as prolonged hyperglyoemia. A risk-averse policy (SAC) favors safe, incremental insulin adjustments, but this could result in suboptimal glucose regulation. This is illustrated in Figure 21 The EVaR-optimized insulin policy keeps the patient within "admissible levels of risk", quickly correcting course whenever glucose breaches hypo- or hyperglyeemic thresholds. In other words, if blood glucose starts trending to a red zone, the EVaR policy will take bold corrective insulin actions to bring it back to safe levels without prolonged exposure to risk. A single trajectory where blood glucose goes to a life-threatening level can dramatically decrease [[ (for some , > 0) and thus lower the EVaR metric. The agent learns to steer away from such outcomes because they are catastrophically bad under the entropic risk measure.
25

---

Published in Transactions on Machine Learning Research (10/2025)
Rot mite amadi (ulmn CHO (000ggmm 85 05
202
BG
Seft thoon bend
Insulin Variability
Glucase (9G) Variability
150-
200-
CHO
20-
Insulin
10-
Rick
Risk Index
Our Method
Hypo Risk
1byper
200
Hyper Risk
Rak Index
Indee
SAC
Jan 01
12:30
13:00
13:30
14:00
14:30
Time
Our Method
Time
SAC
Figure 21: Comparison of our proposed method with SAC (Haarojj et al, 2018 demonstrating reduced variability in blood glucose levels and more robust insulin administration. We observe that when blood glucose reaches risky levels (red), our method effectively course-corrects insulin administration without prolonged exposure to risk, ensuring better patient stability.
3.5 Portfolio Optimization
The portfolio optimization problem seeks an optimal portfolio allocation among N assets by maximizing the EVaR of the portfolio returns R, which captures the upside tail of the return distribution. Here, policy represents the action chosen, which includes sell, buy, hold. Constraints are kept. to ensure that the portfolio weights are nonnegative and sum to one, representing fully invested portfolio. For our portfolio (top 10 stocks of DJIA), weights w E I are constrained such that > w; = 1, w; e Ovi. A constant transaction cost of 0.1% is applied, computed
Cost 0.001 E wupew woyhd| x Portfolio Value. In Table 4 we provide the backtesting [Wong 2010 results of our EVaR strategy compared against other risk seeking strategics as follows:
Portfolio
Cum Ret
Exp Ret
Ann Vol
Sharpe
EVaR
265.7%
13.3%
15.0%
0.75
VaR
250.0%
12.6%
14.1%
0.75
CVaR
257.5%
12.8%
10.2%
0.99
Table 4: Portfolio backtest summary
Over years (DJIA top-10 stocks, 1970 1977), an EVaR-optimized dynamic strategy achieved the highest terminal wealth and growth rate. Its cumulative return +265.7%, outperforming both CVaR-optimized strategy (+257.5%) and VaR-optimized strategy (+250.0%) over the same period. In annualized terms, this corresponds to an average return of 13.3% for EVaR, versus 12.8% for CVaR and 12.6% for VaR. This superior long-run performance aligns with the intuition that EVaR's objective lets the agent capture more upside within a certain entropy distance. Notably, EVaR did accept slightly higher volatility to achieve those gains: the EVaR portfolio's annualized volatility was 15.0%, a bit above the VaR-based portfolio (14.1%) and higher than the very low volatility of the CVaR portfolio (10.2%). In risk-adjusted terms (Sharpe ratio), the CVaR strategy had the highest Sharpe & 0.99 by virtue of its tight risk control (essentially sacrificing return to minimize variance), whereas EVaR and VaR both came in around Sharpe & 0.75. This indicates the EVaR agent deliberately took on extra volatility consistent with "risk-sccking" approach - but translated that risk into higher return so that its Sharpe remained on par with the VaR strategy and quite noteworthy in absolute terms. In other words, EVaR maintained a middle ground on the efficient frontier (Figure 27 : it did not maximize Sharpe ratio (as CVaR's extremely cautious approach did), but it achieved a markedly better growth rate for only a moderate increase in volatility. Indeed, the efficient frontier analysis shows that the EVaR-optimal portfolio yields excellent returr balance, essentiall ially maximizing return for given
26

---

Published in Transactions on Machine Learning Research (10/2025)
downside risk level. By identifying portfolios that "maximize returns while minimizing downside risk", the EVaR frontier dominates what VaR or CVaR alone can achieve.
Compounded cumulative
223.0h
155.00
Margine
45
101.01
201--1
19744
201431
20144
Uncompounded dr drawdown
6.0%
ANT
-04PM
13.48h
33.00
61
Ku.Ei Lod Sence -22.0% Mociran Drandiver 30.00%
65.00%
16.17%
43.00%
Figure 22: [Left:] Drawdown analysis of our portfolio. EVaR-optimized portfolio advances on a steadily rising equity curve while absorbing macro shocks and keeping peak-to- losses shallow and short-lived. [Right:] Portfolio allocation according to the optimal EVaR policy.
Also, since the EVaR objective multiplies every daily loss by an exponential weight, even a handful of large negative returns pushes the risk metric down sharply. This triggers the optimizer to reduce its exposure to high-risk holdings before a market slide deepens, so peak-to-trough declines stay small (* 4% the 1973-74 crash, versus 48% for the index). When conditions improve, the same exponential tilt captures the rebound rally, guiding the portfolio back into higher-beta assets and restoring its previous high in a few weeks. This is illustrated in Figure 22 where the equity curve rises steadily while any dips are both shallow and quickly recovered.
3.6 Sensitivity to Learning rate
To analyze haper-parameter sensitivity, consider an episodic, two action MDP with |S|=50 states arranged ring. Action 0 (safe) keeps the agent in place and yields deterministic reward 0.5; action 1 (risky) moves one step clockwise and delivers a heavy-tailed reward distributed as Pareto(k=11 b=2). Episodes last = 5 time steps with discount y = 0.95. The policy is a one aue-paraeeter Bernoulli, Pr(a;=1) = a()), so the return distribution is regularly varying, which stresses the EVaR objective.
With the baseline hyperparameters, the bias decays like t 0-200 the drift remains at 1/8, and the variance of the stochastic gradient contracts 10000 bringing (0,, close to their limits by iteration 50. Increasing the outer step-size accelerates early progress but leaves wider asymptotic band because the noise term af Var dominates more slowly. A steeper perturbation decay eliminates bias faster yet stalls convergence once the finite difference signal falls below the simulation noise floor. Growing the inner batch length suppresses the drift from 10-1 to 10-2 in fewer than twenty iterations, yiclding the tightest bands at the expense of a quadratic increase sample complexity. These trends corroborate theoretical predictions: de sets the speed varianc trade-off, 6 governs asymptotic bias, and N, controls late late-stage variance once the iterate enters the EVaR basin of attraction.
We further analyze the impact of learning rates algorithm performance using InvertedPendullmmDDobbel... We evaluate learning rates 0.01,0.1,02 0.2, and 0.5 by averaging results over 5 independent batches, each with 50 episodes of 500 steps. The optimizer runs for 500 iterations with the risk parameter a = 0.1. Using (ingma 2014) improves iterate stability, with 10 2 yielding the best performance. this context of our gradient. estimator with ADAM, our learning rate parameter update rule is:
m
0++1 =
T
27

---

Published in Transactions on Machine Learning Research (10/2025)
1.0
baseline
N-1+1
besaline
c 0.20
a-02tt-1y--00
basolino stoady
2-02
exp 0.20
0.8
0.6
0.4
-5
0.2
0.0
40
60
100
20
40
80
100
iteralion
iteralion
10
100
baseline
taselin
--0.2(++1)000
a-0.21t41))00
exp 0.20
exp 0.20
6 x 10-3.
b. (10g)
ge (log)
4x 10-3.
x 10-3.
2 x 10-3.
10-1
40 60
100
40
80
100
iteration
iteration
Figure 23: Parameter sensitivity. The figure shows the evolution of the policy parameter 6,, the EVaR shape parameter 2t, the fiiteeddifeerenc bias by (log axis), and the drift term on 50-state hcavy-tailed MDP. The baseline schedule |4, = 0.10((+1)-06 4 = 0.10((+1)-0.10) 6, = & 0.05((+1)- -0.6, N2 8] is compared with three single-factor variants: (i) larger outer step size = 0.20(++1)-04 (ii) a faster perturbation decay (x and (iii) a linearly growing inner batch N; = min 15). Solid lines show the mean over six i.idd. runs; shaded envelopes indicate standard deviation.
where: me the bias-corrected first moment estimate and é, is the bias-corrected second moment estimate.
As learning rate sensitivity affects both convergence and variance, higher sensitivity lead to faster initial convergence but may impact long-term stability. Very high sensitivity can increase the upper bound on variance, potentially leading to less stable convergence. The adaptive nature of ADAM helps mitigate these effects by adjusting the effective learning rate based on the moments the gradients. From Fig. 24(a),, which depicts the movement of the iterates, and Fig b.). the expected, it is evident that the introduction of an adaptive learning schedule for the gradient estimator EVaR controls the rapid movement of the iterates and is resilient against environment dynamics. When compared against the non-adaptive case, Fig, 24(c) c)
and (d), we clearly see increased movement as the initial learning rate decreases, depicting high susceptibility to the initial choice of the learning rate.
4 Conclusion
In this paper, we introduce a novel multi-timescale stochastic approximation algorithm for risk-seeking reinforcement learning, optimizing the Entropic Value at Risk (EVaR) objective that provably converges. EVaR, coherent risk measure derived from exponential tail bounds, enables agents to prioritize high-reward trajectorics while managing tail risk through Kullback-Leible divergence constraint. and thus provides a lighter control on tails. By employing a randomly perturbed finite difference approximation, we seek the optimal EVaR policy. Across grid navigation, MuJoCo locomotion, glycaemic regulation, and dynamic portfolio allocation, the resulting policies consistently achieved competitive performance, limiting worst-case drawdowns, yet capturing larger upside returns. However, four practical challenges remain: (1) performance is sensitive to the confidence level and inner-loop batch size; (2) the perturbation-base gradient estimator
28

---

Published in Transactions on Machine Learning Research (10/2025)
105
expected
0.03
1.51
0.1
150
iterations
Figure 24: Sensitivity of the optimizer to the learning rate placed at [0.01, 0.1, with ADAM for the top row for JEVAR perturbation (a) and the expected returns (b) and similarly, the bottom row shows the similar sctup without ADAM (c) and (d).
for EVaR requires twice the on-policy trajectories per update, limiting sample efficiency and preventing replay buffer reuse; (3) numerical instability arising from exponential weighting in the EVaR objective; and (4) EVaR objective may be ill-posed if the moment-generatinn function of the returns does not exist (e.g. power-law tails). To address these, promising directions include: adaptive schedules for and batch size to balance bias-variance trade-offs automatically; off-policy corrections (e.g,, importance- crities) to enable gradient estimation from cached data, reducing simulation costs; numorical stabilization via reward scaling or log-domain arithmetic (e.g., log-sum-exp); and truncation of the return distribution.
References
Mohamadreza Ahmadi, Ugo Rosolia, Michel D Ingham, Richard M Murray, and Aaron D Ames. Constrained risk-averse markov decision processes. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 11718 11725, 2021.
Amir Ahmadi-Javid. Entropic value-at-risk: A new coherent risk measure. Journal of Optimization Theory and Applications, 155:1105 1123, 2012.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXio preprint ar 1iv: 08617, 2018.
Michel Benaim. Dynamics of stochastic approximation algorithms. In Seminaire de probabilites XXXIII, pp. 1-68. Springer, 2006.
Vivek S. Borkar, Convex analytic methods in markov decision processes. Handbook of Markov Decision Processes, pp. 347-375, 2001.
Vivek S. Borkar. Q-learning for resk-sensitive control. Mathematics of Operations Research, 27(2):294 311, 2002.
Vivek S Borkar. Stochastic opprozimation: dynamical systems vieupoint, volume 9, Springer, 2008
Vivek S Borkar. Stochastic approzimation: dynamical systems vieupoint, volume 48. Springer, 2009.
29

---

Published in Transactions on Machine Learning Research (10/2025)
Vivek S. Borkar. Risk-constramned markov decision processes. Stochastics, 82(3))337-331, 2010.
Vivek S. Borkar and Sean P. Meyn. Risk-sensitive control with discounted cost in discrete time. SIAM Journal on Control and Optimization, 40(3):681-707, 2002.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXi::1606.0154 0..000 2016.
Dany Cajas. Entropic portfolio optimization: disciplined convex programming framework. Avoilable at SSRN 3792520, 2021.
Dany Cajas.
Riskfolio-lib (6.1.0), 2024. URL
Jay Cao, Jacky Chen, John Hull, and Zissis Poulos. Deep hedging of derivatives using reinforcement learning. arXiv preprint 0rX1:2208..1449, 2021.
Rolando Cavazos- Cavazos-Cadena Cadena and Daniel Bernandez-Hernandee Risk-sensitive control of discrete-time markov systems with feedback information. SIAM Journal on Control and Optimization, 49(4): 2011.
Venkat Chandrasekaran and Parikshit Shah. Relative entropy optimization and its applications. Mathemalionl Programmiing, 161:1-30 32, 2017.
Robert Chares. Cones and interior-point algorithms for structured convex optimization involving powers andexponentials, 2009.
I-Ju Chen, Markus Aapro, Abraham Kipnis, Alexander Ili,, Peter Liljeroth, and Adam Foster. Precise atom manipulation through deep reinforcement learning. Nature Communications, 13(1):7499, 2022.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for evar optimization in mdps. Advances in neural information processing systems, 27, 2014.
Yinlam Chow, Aviv Tamar, Shic Mannor, and Marco Pavone. Risk-scnsitive and robust decision-making: cvar optimization approach. Advances in neural information processing systems, 28, 2015.
Stefano Coraluppi and Steven I Marcus. Risk-sensitive and minimax control discrete-time, finite-state
markov decision processes. Automatica, 35(2):301-300 309, 1999.
Giovanni B. Di Masi and Lukasz Stettner. Risk-sensitive control of discrete-time markov processes, Mathe- matics of Operations Research, 24(3):552 568, 1999.
Giovanni B. Di Masi and Lukasz Stettner. Risk-sensitive control of continuous-time markov processes with unbounded cost rates. SIAM Journal on Control and Optimization, 45(33:106111077) 2007.
Anushri Dixit, Mohamadreza Ahmadi, and Joel W Burdick. Risk-sensitive motion planning using entropic value-at-risk. In 2021 European Control Conference (ECC), pp. 1726 1732. JEEE, 2021.
Angelos Filos. Reinforcement learning for portfolio management. arXiv preprint arXiv:1909 09571, 2019.
Wendell H. Fleming and William M. McEneaney. Risk-sensitive control and differential games. Applied Mathematics and Optimization, 31(3):235-22) 1995.
Javier Garcia and Fernando Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-114) 2015.
Christian Gollier. The economics of risk and time. MIT press, 2001.
Shangding Gu, Alap Kshirsagar, Yali Du, Guang Chen, Jan Peters, and Alois Knoll. A human-centered safe robot reinforcement learning framework with interactive behaviors. Frontiers Neurorobotics, 17:1280341, 2023.
30

---

Published in Transactions on Machine Learning Research (10/2025)
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-pooicy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Jia Lin Hau, Marek Petrik, and Mohammad Ghavanzadeh. Entropic risk optimization in discounted mdps.
In International Conference on Artificial Intelligence and Statistics, pp. 47-76. PMLR, 2023.
Matthias Heger. Consideration of risk in reinforcement learning. In Machine Learning Proceedings 1994, pp- 105-111. Elsevier, 1994.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https://gitte wte.iitbb.oooo.omme.mmommmmoooot
2018.
Ronald A. Howard and James E. Matheson. Risk-sensitive markov decision processes. Management Science, 18(7):356-36)) 1972a.
Ronald A Howard and James Matheson. Risk-sensitive markov decision processes. Monagement science, 18 (7):356-369, 1972b.
Hisashi Kashima. Risk-sensitive learning via minimization of empirical conditional value-at-risk. IEICE TRANSACTIONS on Information and Systems, 90(12):2043 2052, 2007.
Elia Kaufmann, Leonard Baucrsfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982-999) 2023.
Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning evar policy. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 4436-4443, 2020.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: survey. Int. J. Rob. Res., 32(11)) 1274, September 2013. ISSN 0278-3649. doi: 10.1177/022733641134499 bttpps//daon org/ 10. 01711777777777719999198777
URL
Sven Koenig and Reid G Simmons. Risk-sensitive planning with probabilistic decision graphs. In Principles of Knowledge Representation and Reasoning, pp. 363-373. Elsevier, 1994.
Thanh Lam, Arun Verma, Bryan Kian Hsiang Low, and Patrick Jaillet. Risk-aware reinforcement learning with coherent risk measures and non-linear function approximation. In The Eleventh International Conference on Learning Representations, 2022.
Michael L Littman and Csaba Szepesvári. A generalized reinfrrement-learming model: Convergence and applications. In ICML, volume 96, pp. 310 318, 1996.
Harry M Markowitz and G Peter Todd. Mean-variance analysis in portfolio choice and capital markets, volume 66. John Wiley & Sons, 2000.
José Antonio Martín H. and Javier de Lope. Learning aulonomous helicopter flight with evolutionary reinforcement learning. In Roberto Moreno-Diaz, Franz Pichler, and Alexis Aheesada-rrencibl (eds.), Computer Aided Systems Theory EUROCAST 2009, pp. 75-82, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg. ISBN 978-3-542-04472-55
John L Maryak and Daniel C Chin. Global random optimization by simultaneous perturbation stochastic approximation. JEEE TRANSACTIONS ON AUTOMATIC CONTROL, 53(3):781, 2008
Oliver Mihatsch and Ralph Neuncier. Risk-sensitive reinforcement learning. In Advonces Neural Information Processing Systems, pp- 172-178, 2002.
31

---

Published in Transactions on Machine Learning Research (10/2025)
Xinyi Ni and Lifeng Lai. Policy gradient based entropic-var optimization in risk-sensitive reinforcement learning. In 2022 58th Annual Allerton Conference on Communication, Control, and Compuling (Allerton) pp. 1- 6. LEEE, 2022.
Takayuki Osogami. Risk-sensitive reinforcement learning by relative entropy stochastic search. In Advances in Neural Information Processing Systems, pp. 1647-1655, 2012.
LA Prashanth. Policy gradients for evar-constrained mdps. In International Conference on Algorithmic Learning Theory, pp. 155-169. Springer, 2014.
Martin L Puterman. Markow decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Marc Rigter, Bruno Lacerda, and Nick Hawes. Risk-averse bayes-adaptive reinforcement learning. Advances in Neural Information Processing Systems, 34:1142 1154, 2021.
R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distributions. Journal of banking 8 finance, 26(7):1443-144) 1474411, 2002.
R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional valuo-at-risk. Journal of risk, 2:
21 21-42, 2000.
Mark Rowland, Rémi Munos, Mohammad Gheshlaghi Azar, Yunhao Tang, Georg Ostrovski, Anna Harutyun- yan, Karl Tuyls, Marc G Bellemare, and Will Dabney. An analysis of quantile lemporal-difference learning. Journal of Machine Learning Research, 25:1-47, 47, 2024.
Robert Serfling. Approrimation theorems of mathematical statistics. John Wiley & Sons, 2009.
Harshit Sethy, Amit Patel, and Vineet Padmanabhan. Real time strategy games: reinforcement learning approach. Procedia Computer Science, 54:257-264 2015.
Kun Shao, Zhentao Tang, Yuanheng Zhu, Nannan Li, and Dongbin Zhao. A survey of deep reinforcement learning video games. arXiv preprint erxxii1:122 10944, 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distributional reinforcement learning. In Learning for Dynamics and Control, pp. 958-968. PMLR, 2020.
James C Spall. Multivariate stochastic approximation using simultancous perturbation gradient approxima- tion. IEEE transactions on aulomatic control, 37(3):332 341, 1992.
Richard S Sutton and Andrew G Barto. Reinforcement learning: introduction mit press. Combridge, MA, 22447, 1998.
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the evar via sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXie preprinl 1011:0080..00600, 2018.
Mark Towers, Jordan Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL
https://zenodo.. org/recorrd/8120025
Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainly, 5:297-323, 1992.
32

---

Published in Transactions on Machine Learning Research (10/2025)
Guangyu Wang, Xiaohong Liu, Zhen Ying, Guoxing Yang, Zhiwei Chen, Zhiwen Liu, Min Zhang, Hongmei Yan, Yuxing Lu, Yuanxu Gao, et al. Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concept trial. Nature Medicine, 29(10):6633 2642, 2023.
Shaun Wang. Premium calculation by transforming the layer premium density. ASTIN Bulletin: The Journal of the IAA, 26(1):71-92 1996.
Peter Whittle. Risk-sensitive linear beeaar/quddaaticggauusss control. Advances in Applied Probability, 22(4): 764-774, 1990.
Woon K. Wong. Backtesting value-at-risk based on tail losses. Journal of Empirical Finance, 17(3):526 538, 2010. ISSN 0927-5398. doi: Bciencedirect. htt.a///aaaaac...ca/////////////
URL https://www
Jingda Wu, Chao Huang, Hailong Huang, Chen Lv, Yuntong Wang, and Fei-Yie Wang. Recent advances in reinforcement learning- based autonomous driving behavior planning: A survey. Transportation Research Part C: Emerging Technologies, 164:104654, 2024.
Peter R Wurman, Samuel Barrett, Kenta. Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, (02(7896):223-222) 2022.
J. Xie. /smg////mmgucose::: A type-1 diabetes simulator implemented in python for reinforcement learning purpose. http:////gthuuu...o//.../aa. http://ixxx/3//mmmcmcoaaaaa
simglucose 2018. Accessed: 2024-05-22.
Jesse Zhang, Brian Cheung, Chelsea Finn, Sergey Levine, and Dinesh Jayaraman. Cautious adaptation for reinforcement learning safety-critical settings. In International Conference on Machine Learning, pp- 11055-11065. PMLR, 2020.
Qiyuan Zhang, Shu Leng, Xiaoteng Ma, Qihan Liu, Xueqian Wang, Bin Liang, Yu Liu, and Jun Yang. Cvar- constrained policy optimization for safe reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
33

---

Published in Transactions on Machine Learning Research (10/2025)
A Proofs
Consider the filtration Fi = o(D1, we, Se, We prove Theorem 4 in two parts.
A.1 Proof of Theorem 2
Proof of Theorem 2: Part
Proof. Consider the recursion
0z+1 + 6, ( (eR(t)o- -0.)
0. + 6, E [ + eR(t)/o -E { -8.)
(25)
0, + og (1,) + M4+1),,
where h2(10 = E e R( 9, and M4+1 = R((r//e E R(T)//
It is easy to verify the {M,} is a martingale difference adapted to the filtration {Fe} i.e., M, is Fr-measurable and E [M,++IFFF = a.s. Also, note that h : R R Lipschitz continuous. Further, by Borkar-Meyn
Theorem (Theorem Chapter of Borkar (2099) ) one can show that sup, [0c] < ao a.s. Hence, by Theorem
of Chapter of Borkar (2009); the iterates (1t) asymptotically tracks the following ODE:
i ET
[(())//] 0.
(26)
Since h2 is linear, we conclude that
lim 0, (1*0)(0*) 0} = lim 0r = E, ET~rO 28(T)
(27)
1-20
Now consider the stochastic recursion
We+1 wi + 5c ( (())
wi + 8e (B(()) - wie R(r)/ (((()) wee R(F)/0) + ( R(r)/o wre R(r)/a
Rewriting the above equation, we get
Wi+1 = + 8r (we) + MY+1)
(28)
where (w) E, (((r)e we n(T)/2 and
(29)
M R(T)) whe R(r)/ (((T) wse R(T)/o F
(30)
Now,
E My [F
E R(T)) R(T/// - E R R(T)e
wn Fi F.
E (( wee R(T)/2 F.
- R(T)) (R(O) Fi
34

---

Published in Transactions on Machine Learning Research (10/2025)
Hence (ML)) is a Martingalo-diferencc noise adapted to the filtration (Fi). Also since y E [0, and we have [R(T)] Re. Hence,
w > 0 s.t. E [ < (1+ (w)(2)))
(31)
Also, for w1,42 E IR, we have
(w1(w) 11 (w2))
Er ( lws - wel
$ exp
( Roo
(a(1- ( lus w2l-
Hence 12 is Lipschitz continuous.
Now, we will show that the iterates 0, are stable, i.e., sup, |ws/ < 0c a.s. Hence, we consider the following scaled functions
ho(w) (cw)
C. = 0.
(32)
Now consider the co-ODE given by
is (w).
(33)
where
h((w)
1 Er ((T)))(()) owe
11 (w:)
lim 1e G-
lim 6-0000
lin
c c
WB, (()))
(34)
Hence ODE 33
becomes
#s = -Uw, where = Ex e R(T))
(35)
Note that since 3> > 0, we have 0. Hence, the oc-ODE given above has unique globally asymptotically stable equilibrium point. Hence, by Borkar-Meyn Theorem (Theorem Chapter 3 of Borkar (2099 we have
sup [
(36)
Now by Theorem 1 of Chapter 2 of (Borkar 2009 the sequence (wf) converges to compact connected
internally chain transitive invariant set of the ODE given by
h" (w).
(37)
Since h is a linear function, the only compact connected internally chain transitive invariant set is (w) =
0). Hence
(w) = 0 = E, ((T)) WE, [(T)//s =
E, [ R(T))
E, R(T)//
Thercfore
E2 R(T R( E, R(T)/ax
lim Wi
(38)
35

---

Published in Transactions on Machine Learning Research (10/2025)
The function 0" (x, 0) Eyano |e [e is Lipschitz continuous E [mmin: Imaz], i.c., 30g 0 such that:
Lemma 3.
(x1. - 0* (I2, 0) & LotØ x21.
Va D2 E [Tmin: Imaz]
Proof. We compute the derivative of U (x, 0) with respect tot a as follows:
E, [-
R(T) (G)
(x, 0)
dx
x²
Since [ &
Rse and Tmin, we have:
Rio Er R(1)//
Ric
0 (z, 0)
< (1 -))x
dr
(1 (1-7) ) min
Rie (1- Fmmn
Let La
Rs. (1-7)ammn
By the Mean Value Theorem, Ven, 222 E [Eminn Tmnx]
100 (x1,0) - O# (x2, 0)1 & Lo|ø1 Ø2//
Hence, 8d (x, 0) is Lipschitz continuous in r.
E-- (()))))())))))))) xB. [<B(r)/a]
Lemma 4. The function w*(x, 0) erists Lu such that:
Lipschitz continuous on x [Tmins Fmar), i.e., there
(x1, - w (x2, 0)l < Lu| - x22, Væ 22 E [Eminn Fman].
Proof. Let (xx) = (-( and D(x) = xE, #t/// Then: (*1,r) 0) N(x) D(z)
Their derivatives are:
(²(x) 11 D'(x) E, E. R(r)/o
Br [(((T) R(r)/o.
Using the quotient rule, obtain:
N(x) - ((x)
-(
w (x,0)
dr
D
Ri Bounding each term ([R(T)) < (1-9)+ I 2 Xmin) from above as follows:
R2
Rv. 73mmmn
[N'xa] < (1 22
min
[D(a)] > Imin Cja
Ruo [D'(x) Vamm
Rwo
Tmin
Rso el (1-73mmin.
[N(x)] < (1-
Substituting these bounds, we get:
d de n (1.00)
R
28 MJmmn + Roo c 71mm (1 (1 - 2min
28 R
2
28 m1mm & Lus-
222 min
(1 (1-))
m
Thus I (21,0) w* (1z, 0)) & Ln 101 zzl.
Lemma The first order derivative ('(z) log (x,0) (x,0) is Lipschitz continuous on e [Imin Fmar], i.e., TLey O such lhat:
[C'(x)) G'(x2) Lor sal. V01, [Eminy Fman].
36

---

Published in Transactions on Machine Learning Research (10/2025)
Proof. Consider the derivative:
(0x,0) 0)
(x, oУ 1* (r. 0)
log
dx
Using Lemmna 3
obtain:
(+
log
$ Lac Amenth 7)
dx
From Lemma w (x, has Lipschitz constant Lus Therefore:
(G'(z1) - (122) & (geemmnlity + L.s) 121 xzl + L 1 x2/-
Proof of Thcorem 2 Part II:
Proof. On the slower timescale, the x upupae approximates the ODE:
-G'xx) = -(11g P T R(r)/
ET (RTT))
B wory R(T)/
By Theorem 1, G(z) is m-strongly convex. Using G(x) as Lyapunov function:
dG =
with equality only at (0). By LaSalle's invariance principle,
2t x (0) almost surely.
Now, regarding the joint convergence, note that the time scale separation ensures
02 v (re, 0) and We - (x,, 0) before Ia updates significantly. Thus, the system converges
00 (x* (0). 0), w (x (0), (0)) a.s.
A.2 Proof of Theorem 3
We prove each part of Theorem 3 individual lemmas here.
Lemma 6. Let & grre (frac12, 1) with 1. Then for any fized x
K
E [1, - 0z 1
with K > 0, > 1.
t
Proof. Let the error be eΦ = 0, 8* (x, 0), where 8* 0) = E The update rule becomes:
e41= (1 (1 Sent411
(39)
where 1 = R( (x, 0) martingale difference sequence.
Square both sides and take conditional expectations:
(= (1 -1- (,) + 8, 2(1 6,) =/
(( )2 [F] = (1 - (c.) (1
1 [F).
37

---

Published in Transactions on Machine Learning Research (10/2025)
2/1zp v²_
since [[ = 0 and E[rE+1] & e (1-m))mm
Taking total expectations, we get
B[( ) E (1-6,)) 6,)² 20-,))/((( [[(
+ + àz22
(40)
Now define et = {) Hence, from Eq. 40),
vt+1 = (t + 1) [B((((++)))
<
+ +
= (t+ 6.)² tt 1 + (1( 1)
(41)
Take 6, m, (0.5, 1). Also, using the binomial expansion (L + 1)a = 17(1 + & + 0((-)) and (1 6,)²
1- - 26, + 0, = -1- 1-2cct 2ct"" + e2tt2r
-5) -)
2ct + r + Ott
(42)
t
Because >> 0.5, t- >> t-1 honce for all t3 Ti = (22 2r 1/(1-r)
2/-2//
k == 2c 2 > 0 (since c c> 7
A < 1 kt
(43)
For the additional term in Eq. 41 we have
(L + 8) = + & 26²r (12 2).
(44)
Set To = max((T,22 23. Then, from Eqs. 42 and 44 we have for t & To,
Ut+1 & kl U) + 207
(45)
Now, let we vt C, where C.
2 Then,
U++1 + +C & (1 - nt xtt + C+ 2-
(1 - kt 1)t 2/ C xC
(since RC 2v²c?)
ot + C x-= (v: (V=-)
Ut + C =t 1i Wi kt Ct.
(46)
If U2 C, then from 46 we have
U141 C & U2 + Wi+1 & Wt.
(47)
Otherwise (ve C = wi < 2C),
W++1 Ut+1 + C Ut +,+C+ (C v.) U. + +++
2C,
since nt_ & 1, for t > sufficiently large T2.
(48)
From the above two cases, it implies that in either decreases or remains bounded by 2C, for t sufficiently large enough To max (To, T2). So wi & max (2C, suprzr w W.. c0, Vt Ty Therefore is W for all large 1 > Tg. Therefore,
(+> To). Wa
E[(E?)
For t < Tú let M = max {[ Then M & M M t-. Now choose K1 max(Wa, M MT ) Then
K1 t
E (((
1 1.
38

---

Published in Transactions on Machine Learning Research (10/2025)
Lemma 7. For given policy we, let the step size de = fr E (2 with 2Rx [c
Then, the
slochastic voriable wt salisfies:
< K2 for some K2 > 0 and t > 1.
E [l w² (x,
Proof. Define the error ox == wi - (r, 0). The update rule of we becomes:
c++ = et (1 deze R(ret)/ + eg R( (R(Te+1) - nw (z. 0))
Square both sides and take expectations and using (6 (R(Te+1) - x wx: 0)) e 0, we obtain:
1 = Er (1-
dy 2 [(e])]]] 8 2 (R(T) xw (x; 0))
Using [R]] < T-y
and x 20 mmin: we bound:
RRg e2 ( 02.
E, 2 (R(T) arw (x))²²
(1 7)2
Let X R(r)/a (> 0), and e₁ = (7))) Then,
E E[(1 6,x)²1 + 0.
(49)
ei+1
A
Note that - 2-
8
1 6,X,
0 6,X).
Then,
1- At,,, A, < 1 E
where x cE[X > O.
Substituting Eq.
49
we get
et11 & (1-At")ez + 22
(50)
Multiply 50 by (t+ ()r and set 1/2 tt
ue+1 = (1 1)'et11
< < 1) - -
= =1+1))( 0z 66 (1+ +
1
= =1 0(0 \)) v / ( 0(1
where we used the binomial expansion (L+ 1)" 1" (1 +
Bocause r (2. 1), the term - == o(t-r), so for sufficiently large
++E+0(13) Xt 11 1t where x 2- r
0 (from Lemma assumption).
A Absorbing the factor 1 + O(L-1) in the noise term into a constant 2 yields, for all large enough 4
Ut+1 & (1- kl kt 22000 TT
k x
r 2
0.
(51)
39

---

Published in Transactions on Machine Learning Research (10/2025)
2c"02 and set Wi 04 M (1 112 0). Now using 02 = Wi M in we get
Let M
W1+1 4++1 + M & (1 - kt (w; - M) + 2c. o2, t + M
w++1 & [1: M) kt (w: M) + 3t + M
Wt kt Wi + (kt M + (t²))
wi kt J wi + 2nt M
we kt - (w) 2M).
(52)
If W4 2M, then from 52
we have
W4+1 W/ kt= (wi 2M) wt.
(53)
Otherwise,
W++1 = W/ + kt. (2M w/) & + 2M Wi 2M, since kt. & for 1 sufficiently large.
(54)
From the above two cases, it implies that either decreases or remains bounded by 2M, for t sufficiently large enough To- So w, & max (2M, superrr wx) = Wx < so, VA 3 Therefore U, & W - M for all large > To. Finally, since et ve/t"
Wa tr
a &
for all t t> To.
For the finite prefix 1 & I < To, let M' = max | Cx. K₂ = max MTTT); } which yields
Then et & M & MTTT Finally choose
K₂ tr
E [((+1) )
V1 0.
(55)
Theorem 5. Let step sizes 6, +, & $ with b ($.1) 1) and d 1 b. Under the Assumptions
and 3 the iterates E IR asymptotically satisfy:
K3
[[sx-- xz)2
where x* arg ming G(x), and the constant K3 0.
Proof. Define = j2, x*1 The update rule is:
Tt+1 Dt Ee (G' + no)
where the gradient estimation error 7/ log (क())
(wr w* (x:)
Using the Lipschitz continuity of log(-) near 0 (z)):
0, log (+())
$ Inle?h
where L1 Inl, Jā( (aa((1))))mm) Then:
Ime & T,lc,21 -|1--
Squaring and taking expectations:
21,K, 2Kg
< 21/ + 21
t
40

---

Published in Transactions on Machine Learning Research (10/2025)
Let C= = + 2K2. Expand Vi+1 (z+ (z)) + mi) - x²)²² Taking conditional expectations:
E|V++ F & V - 2z, - x²) + [G' (2.) + +m)² [...
Using strong convexity G' - x*) > vV,/ with
a and
[G'(z) & (1-m(1>)) + logal La
28
max
B[V++1] (1 2 Bl + E 1, (1+ ()
Substitute d.
2
+ d²
L2² + +)
E[V4+1] 1
E[Vi]
x t max
f2b
Unroll recursively for T >
2ōd
2 C/t
2
B & E[V
+ d
1
(56)
t=1
r.3
max
1b
2
kmr+1
x
max
kb
Using 1 - & E gel
T-1
2
2
T
E cxp
23
2
max
max
Approximate the geometric sum
>T11
1
pl
Hence,
16
1-b-
2 T
1
exp
(57)
xmax 1
7
Choose d = (1-b) b so that 27d = v max
b(1 - b). Hence,
C/t
T-1
2od
2 12+0 C
T-1
(1- 6(1 b(1-b) b)
d
II
II
(58)
22
Tmmx bb
12b
k.b
k=
1=1
k=1+1
To bound the above term, we let Pt. bound (1 x1= ee-
Ih (1 b(1 - b)k=
Because < b(1 - b) < 1, and using
(0 < x < gives
T-1
- -b(1 - b) Σ x )
PeT & exp
(59)
k=t+1
Since I + -b is decreasing,
T-1 Σ
T
T1-b - (t + 1)
kー
x dr
b
k=t-1
J--1
Substituting this in Eq. 59 yiclds
Per & expf-b(77 - 1) 1)²
(60)
Let t0 [T/2]. We consider two cases:
Case & t & to: Because x --- is increasing, (++1) & ( + 1) so from 2 2- 1b-1)r1-1p Consequently
60
Pi, & expl-b(1 -
ta PLT
Per 22
((2b) espl-b(1= 2 2--177
(61)
2b
41

---

Published in Transactions on Machine Learning Research (10/2025)
Case <t < T: Let s = = t (= I,.... to). By applying the mean-value theorem the function _!=b,
obtain 71-b - (T + -1)1=b > (1 0T - 1). So Eq. 160) implies Pir exp(- exp(->6,7*) > 8 1)3 with
As = b(1 6)2". Since t = T - > T/2, we have
7 Σ
PLT
2
7g (s-1) 222TT-20
1 1 c- AoT
&
t-fo
2
T
(62)
where we used e e- < 1 x 1).
Now combining Eqs.
66
and
62
we obtain
1
6(1
b)
dL +c][<126)
( -01--2-')
+ 2 T
(T
(63)
kb
Ab
1-t+1
Finally, combining Eqs.
[66]
57
and
63
we get [[Vr] O11/T)). This implies that
[10>
K 0, 14 >
T (sufficiently large).
(64)
Now using the finite comparison trick from Lemma
can show that
K3
B [Ir - x x²)²| &
vt vt 1, K3 >
(65)
7
A.3 Proof of Theorem 4
Recall the definition,
V (evva((0)) Jevan(O; + c,ii) JEVAR - GAi)
(66)
2,,
where =(0t) = =G(xt) (e,) and 2(0,)) G(x,)) JEVAR (0,)
Lemma 8. Let ( Jevar 000 30th 20 exist and faxi supa EVaRei (0)l S.. e. Then VO interior(O)
by (0,) = VJ (.)) JJEVaa Fi = O(e)).
Proof. By the continuity of Jeva and A. being a Bernoulli random variable, we have, by Taylor's theorem,
Juns (f. + e (x) & (O,) + a AT (eevaa(O.) ar a ('EVaR, (0)),, + V (e
where O, lies on the line segment between O, and + a... Hence, (0, + ced,)) Jena (0t aa)) AJ (e. A,
of v (Jena(0i) JEvun(0.)) a OS.OS. 124.
20
Now,
6, (0.) E V Jevan (0)) Veeaa((0)) (0t) Fe
JEVAR (0, + c. A.) Jevaa(Or ceD.). 2c,
E
J (0.) Fz
42

---

Published in Transactions on Machine Learning Research (10/2025)
Let by, denote the tth term of the bias vector by Then
bi E
Au Au
JEVaR (0.)
of
Vi (Jevaa(0)) + JevaR(O,)) A (
A.A.S..00... V JEJEv -eJJaa (0.)
(67)
12
where 8 lies on the line segment between d, and 0t - a&c. Now note that,
E
[ V
JEVAR
Fi]
(
[F] +...+ + (VJeva (VEvaR(B.)))p (B [A, L., Fi [A,
(Since, V.Vav (0c) is measurable w.r.t. Fi)
A L
A Du
A
Dtp
(Vievan(e)))
F
+ (V.EvaR (
Fi
=
AT, An
EA 61 EAC EA 12 BAC,
EA, 12 EAa
= (Veaaa
Je (0:))p
EA,, 1p EAD,
(Since, EA,, Vi [1...p) p] and As, is independent of Di,Vi- # j)
0
(VJevaR(O:))
(V (ev))
V JEVAR (0s)
(68)
Therefore, from Eq. 67 and Eq.
68
we get,
by
E
1
(Evaaa)( +
JevaR(())) V (
(69)
12
Du
We can bound the term on the right- right-hand side of Eq.69
69 in magnitude as follows:
1
be (0.)
E
(V3
(0.) + 3 JEVaa (1)) á 2 1.00001 Fr
12
Au
cc
Di,, Dr.. tig
pca O
E
(70)
k1
Duy Drgy Aig Bg
The first inequality follows as 3 Jevaa (0) & e, 4O and the latter inequality follows since
$ 1.
43

---

Published in Transactions on Machine Learning Research (10/2025)
Proof of Theorem 4
Proof. Consider the recursion from Step 12 of the algorithm:
O1+1 Oc + dr
E EVVaRR(O.))
V VJvvaR Fr IR
V (JvvaR(Oe) V JEVaR (0t) Fr
V (JevaR(O..) =(0,)) 2(0
2c,
0c + at (br + Pe + (evan VJEvaa - E V (1)) Fi
Or + or (br + ex + spr + ()EvaR(O:)), where et VJ (0t) EE
V EVaR (0t) | Ft
z(0,t) =(,,7
and 4p
(71)
2r,
Then
=(0 z x(0F)
1
E
[[
E
Fr
E (=(0t) 200 FF
20, A.
2cr
| ((==(1)) + (2(0
2c
1
|
-1 ||x } + E ||,, (1 Fb]
20t
LG
LG EE [| x* [0 (Since G is Lipschitz
E ||-,
||
2
2ct
2c,
with constant LG which follows from Theorem
1
E [
E [ x² (0 (0t)
$ 2c
(by Cauchy-Schwart Inequality)
VK
(by Theorem
N
Now, by the Monotone Convergence Theorem, we have
< Σ &VK K
E
Σ
aelvpel
=
Σ [1
a
<0.
t11 N.
12
Therefore
P
Σ as spll <
=1
= Σ
ar9t < 0 0.3.
Σ
aris
0.
(72)
k
12k
Define
E1+1
Octrt t 2 0.
(73)
Then
E [[
E
0.c F
Σ
a.E Fe a,
E VJ (JEvaa(01)]
- EE
VJ
(ev)a(()))
(il)
Ex. (74)
1=0
1=0
44

---

Published in Transactions on Machine Learning Research (10/2025)
This implies that (Et) is martingale with respect to filtration (Fi). Also, since JEVaR is contimously differ- entiable, we have & is square-integrable, VI, i.e, I [|| 0o, V.. Again, by the continuous differentiability of JEVER and Assumption 4 we obtain
- [la < do on the set (sup ||0ell z0).
[e
Therefore, by the Martingale convergence theorem, we get
lim & exists on the event (sup |0.1 < so). 60
(75)
Hence, by Theorem 2, Chapter of Borkar (2008), the asymptotic behavior of the sample paths belonging the event (sup, |6i]l o0 is equivalent to the long-term behavior of the dynamical system induced by the ODE
d0(t) di
= V JEVaR (0(t), 0.
(76)
This further implics that the iterates 8, corresponding to the sample paths belonging to the event (sup, |6.1l < оo) converge to any the compact transitive invariant sets connected internally in chains of (76). Invariant sets are subsets of the state space that remain unchanged under the flow of the dynamical system. The dynamical system (6) driven by the gradient of the JEVAR is gradient flow where the only possible invariant. sets are the subsets of H {0) VEVaR,, (0) 0} (Lemma 1, Section 10.2 of [orkar (2008)). Further, by invoking the LaSalle invariance principle and the Lyapunov theorem, one can obtain that the asymptotically stable points inside H are given by {0 e H JEvan)))) < 0}.
B Comparison of EVaR estimation methods
One also estimate EVaR of discounted cumulative rewards for sample trajectory using disciplined convex programming characterization as stated by Cajas (021) as follows
JEVAR (0) = min t - 3 (nn)). 3,t.u
(77)
subject to B >
Uj and (Jevaæ(O); - t, U3) e Kexp, Vj e [1,N]
where [JEvaR (0) are N realizations of (0), B, t and are variables and Kexp is an exponential cone
[Chares 2009) which is defined as follows
Kexp = ((),b,)) by 0, bexp ( (la,b.c) a a 0,0 0, 0}.
(78)
The above optimization can be solved efficiently to any desired level of accuracy via interior interior-point point methods due to the existence of computationally tractable barrier functions, which enable the efficient exploration of the solution space /Chandrasekaran & Shahl 2017). In Algorithm 2 give EVaR optinization algorithm where the EVaR estimate computation is done using the disciplined convex cone method by solving the optimization problem. This requires access to N trajectories for the computation and which decides the desired accuracy of the estimate. This process is not an online process with respect to the estimation, and the movement of the samples has little influence over the EVaR estimate. When this process is compared with the Stochastic Approximation (SA) version of EVaR estimation, we see in Figure 25 is more resilient changing environment dynamics, and higher error band justifics the online nature and adaptability to the samples.
Remark. The rewards for the encironment are negaled to comply with the structure of the solver in Algorithm
45

---

Published in Transactions on Machine Learning Research (10/2025)
6
EVaR with Exp Cone EVaR with TSSA
-8
ostinnte
-10
-12
-14
(over batch
Figure 25: EVaR estimation using two-time scale schotastic approximation, where the blue represents the estimate using the convex optimization and the orange band represents the estimate using the SA method. Run for 10 batches with CartPole as the environment with episode length of 100.
Algorithm EVaR Optimization using Disciplied Convex Cone
Require: risk level a (0, 1), initial 0g E IR, step-sizes {at, Cr 8,, Ee), inner lengths Ne Initialize policy network (To) parameters θ
for / = .....,77
do
Draw A, ( IID.
4: 0 02 A., 0r - 0, A:
5: for k l.....,, N₂ do
+- EVaR estimation for "
6:
Sample trajectory Th Top compute R Ev T end for
8: Solve using the Interior point method
N
Gity - Solve min -- I1n (a:N, subject to B 3,8,u
Σ
Uj
and RT,, t,3, Kexpi Vj =j=1, 1....,
N₂ do 10: Sample trajectory T To., compute R EH rr
for k == 1,
EVaR estimation for "-"
11: end for
12: Solve using the Interior point method
Gū Solve( min ビ- I1 (a:N, subject to B Uj
and R t, Kexp Vj ...
G - G A,1
13: 9.
2ct
14: 0.11 Or 0+ g.
15: end for
16: return 0r
46

---

Published in Transactions on Machine Learning Research (10/2025)
C Experiments Detail
C.1 Finite Difference Gradient Estimation
The hyperparameters used in finite difference-based gradient estimation have significant impact the algorithm's performance. The timeout parameter controls the time allotted for each function evaluation, ensuring efficient computation. Iterations govern the number of optimization steps, balancing convergence speed and accuracy. Learning rate decay and power adjust the step size dynamically, allowing the learning process to slow down as the model converges. Perturbation size affects the extent of exploration in the gradient estimation; while ils decay and power ensure that perturbations shrink over time, refining the gradient's precision. Momentum helps maintain stable updates by incorporating past gradients, and the Adam parameters (Beta and Epsilon) enhance robustness, partieularly in the face of noisy gradient estimates, providing smoother and more reliable updates. Together, these hyperparameters create dynamic and adaptable optimization process, crucial for navigating uncertain and noisy environments.
Hyperparameter
Value
Timeout
10
Iterations
10,000
Learning Rate Decay
1 x 10
Learning Rate Power
0.5
Perturbation Size (px)
2.0
Perturbation Decay
x 10-2
Porturbation Power
0.161
Momentum
0.9
Beta (Adam Parameter)
0.999
Epsilon (Adam Parameter)
10
Table 5: Hyperparameters for Finite Difference Gradient Estimation
C.2 MuJoCo and Gridworld
We evaluate proposed algorithm various continuous control tasks from the OpenAT Gym suite Brockman et al. (2016), where augment the environment with random normal noise to introduce uncertainty. For the discrete setting, we use a custom GridWorld environment with randomly placed obstacles covering approximately 30% of the grid. Importantly, we ensure that a path always exists from the start to the goal state, even with obstacles. The hyperparameters used across all experiments are summarized below
Hyperparameter
Value
Learning rate
0k (4+14A Ck (k+1)o
Constant A
10-7
Constant
0.999
Random noise parameter Ò 1-a
Action sampling
a (T(- (0,
Step size å
0.12
Step size 5
x 10-10
Table 6: Hlppepparameters used in experiments
Remark. The confidence value that used for EVaR is flipped, i.e is used for the same experiment for CVaR and VaR estimates to align the measures the direction of the upward risk, which is used for most reward distributions.
47

---

Published in Transactions on Machine Learning Research (10/2025)
C.3 Implementation Details for baseline comparison
The tabular implementations of EVAR SA and the three CVaRt-base baselines, present their mathematical update rules, and summarize all hyperparameters. By re-implementing each algorithm in an identical finite-horizon MDP framework, we ensure that performance differences stem solely from the risk criteria and their estimators.
C.3.1 Environments and Reward Structure
Cliff Walk (4x 12). States form a 4x grid. Start at (3, 0), goal at (3, 11). Stepping into any cliff cell (3, for < i & 10 yields -100 and resets to start; all other moves incur r -1; reaching the goal yiclds 0 and termination.
Windy GridWorld (7x10). States form 7x10 grid. Start at (3, 0), goal at (3, 7). At each step, the agent chooses one of four cardinal moves, then experiences upward wind of strength wy E (0, 0, 0, 1, 2, 1, in column J, paying -1 per step and terminating with r = 0 upon reaching the goal.
C.3.2 Algorithmic Update Rules and Hyperparameters
All four algorithms are implemented within an identical tabular Q-learning framework. Specifically, at each time step, they perform the update
Q++1(s, = Qx(s,a) + ar(ur - a)(
where y the aggorithm-specffii TD target and a is the step size as described in Table 7 Action selection follows an e-greedy policy with & 0.1, and future returns are discounted by y 0.99. Each method runs for 500 episodes per random seed (with maximum of 200 steps per episode), and results are averaged over eight independent seeds. All Q-tables are initialized to zero, and reproducibility is ensured by calling np. random. .reeee) (sedd) at the start of each seed.
Table 7: Algorithmic update rules and hyperparameters. All methods use tabular Q-learning with base 0.1, γ 0.99, and e = 0.1.
Algo
Risk Objective
TD Target Vt
Params
EVAR SA
EVaR.
max Q(s(,0))
0.05, N
infe In( B(a
20
G(R) 1 In( 1 2 R4)
CVAR-PG
(V)
(1 A)(r + max. Q(s',a')
A 0.3,
1 D Boy & |aN|
CR. (R)
15
ming sup, 2|R u(s)] + [[[(((s] R D4PG CVaR (V)
-VaR
= us + afff- uel.
dual step = 0.1
+ max, y Q(x ,d))
0.6 + max, Q(6', ..))
window
1 Boy [aN]
0.4 a.
In E'VRR-SA, gradients of the entropic value-at-risk are estimated using a two-point finite-difference stochastic approximation on recent returns, allowing for unbiased updates without backpropagation through analytic risk gradients. The method also adapts its learning rate per state, ensuring convergence stability across heterogencous state visitation frequencies. In contrast, CVaR-PG augments standard temporal-difforenc (TD) targets with the empirical CVaR of the worst a-fraction of returns, SDPG-CVaR solves a saddle-point dual formulation with per-state dual variables, and D4PG-CVaR employs sliding-window window return samples for quantile averaging within distributional Q-learning framework. The Cliff Walk risk landscape contrasts a high-penalty "cliff path" with a uniformly safe alternative; EVAR-SA's 's entropic objective naturally avoids the peak risk region. In Windy GridWorld, per-column wind strength h yields uneven state-transition uncertainty, which EVAR-SA accommodates via its adaptive step size in the SA update.
Note: All algorithms evaluated in this work confirm that the observed empirical advantages of EVAR-SA
arise from its risk criterion and eslimator alone.
48

---

Published in Transactions on Machine Learning Research (10/2025)
C.4 Glycemic Control
The environment is modelled an MDP in such that the state space consists of multiple noisy glucose measurements at various time points in the past, the carbohydrate intakes, and other relevant information about the patient. The action is the amount of insulin to be delivered to the patient, also a scalar that indicates the insulin to be administered. We observe from our experiments that our algorithm can keep the patient at an admissible level of risk. There are instances where the glucose breaches into the hypo or hyper region, the policy course corrects to maintain the stable condition.
150
10x-
CTM
CGM
MaMM Un Wwwllhhhudd
Hiper
00:00 3.66 04:00 00.30
06:00
13:00
11:00
S0o0
1850
2000 2000
23:00
12:00
3200
1000
2000
21:00
20:00
01.00 $0:00
(4)
(b)
(c)
adu10000
300
2G
Foll Indic
Fliclnde
11:00 30:00 60
$5.00
00.33
0720
09:00 0420 11839
15:90
32:00
10h30
(f)
Figure 26: In plots (b).(b) and (c) represent the patient profile of adoloscent001 and the rest of the plots represent the palient. adul1002. Both patients show up to be stable and alive under the influence of the administration of insulin via the controller, where the signals are optimizod by the EVaR optimizer.
We show that our algorithm is equipped to handle high-risk scenarios like administering insulin to Type-1 diabetes patients of type #02, adolescent registered on the Gym environment using the simulator developed by Xie 2018 for RL control tasks. The simulator emulates PID controller that provides the insulin to maintain the Blood Glucose levels, which is the action for the agent, and we try to optimize by increasing the longevity of the patient using the finite difference EVaR optimization. The patient profiles are given below, where CF: Carbohydrate Factor, CR: Carb Ratio (often referred to as Carb-to-Insulin Ratio), and TDI: Total Daily Insulin.
Name
CR
CF
Age
TDI
a00lescent+#401
12
15.0360
18
36. 7339
#002
8
9.2128
65
57.8688
Table 8: Patient profiles
C.5 Portfolio Optimization
Problem Definition We focus on the top 10 US DJIA stocks- [KO, AAPL, MSFT, JPM, WMT, UNH, v, PG, JNJ, HD] using market data from January 1, 2014, to Jamuary 1, 2024, which provides a decade of market information encompassing various economic cycles. Our custom PortfolioEnv class, designed to adhere to the OpenAl Gym interface (Brockman et al., (016)) and compatible with Stable Basclines Hill al. 2018), represents a 50-day window of historical returns as a matrix O e 100010 To minimize the EVaR value of the negative returns, which in case are the portfolio values represented R under
49

---

Published in Transactions on Machine Learning Research (10/2025)
the constraints that the weights of the portfolio always up to 1 after each allocation step. We find the optimal EVaR portfolio with N-assets represented by the optimization problem:
max 8=0
lo
J)
(79)
s.t.
Wi B 0, 2
(80)
(81)
here the weights of the N assets in the portfolio is represented by the weight vector w [w....., Wp] that Σ Wi = 1, where 0 & W; < 1. An asset weight of 0 indicates zero holdings of particular assct in a portfolio, whereas a weight of means that the entire portfolio is concentrated in the said asset.
such
Action Space: For each action A A, the action represents portfolio weights for portfolio allocation on
N assets. The weight constraint makes cach action represented as A [ai...... ayl so that Σ 0g where 0 E a; & 1. In alternative implementations of this framework, a, < 0 would permit short-selling an asset, while a; would allow for leveraged positions. However, in scenario, limit the actions to non-leveraged, long-only positions. To enforce these constraints, apply the softmax function to the agent's continuous actions.
State Space: The state space S is represented by a matrix St E R=x*, where w is a predefined window size and n is the number assets. Each element 8tj of the matrix represents the return of the asset j at the time step w Formally:
S. trigri [ [t- w + +1,4,j [1,
(82)
where ri, the return of asset time The observation space bounded, with S [-1, 1xN At each time step 4, the state S, represented by a matrix S, R=x", where:
is the window size (number of historical time steps considered)
the number of assets in the portfolio
Formally, we define the state matrix follows:
Tt-w+111
Tt-w+12
Tt-w+1,n
Tt-w+21
Tt-w+22
Tt-w+2,
S, =
(83)
Tt.1
T2,2
where rigj represents the return of asset j at time step i. Each element stj of the matrix S, corresponds to the return of specific asset at specific time:
Si = rt-wtiy Pr-weij - Pa-wiil1
(84)
Pe-wti-1,5
where Pt.j is the price of asset at time l. To ensure numerical stability and consistent scale across different assets, we bound the elements of the state matrix St e []1,1]=xn, The rows of the matrix S, represent different time steps, with the most recent returns in the bottom row and the oldest returns in the top row. The rows of the matrix S, represent different time steps, with the most recent returns in the bottom row and the oldest urturns in the top row. This structure allows the agent to identify temporal patterns or trends in asset returns, potentially.
Reward Function : The reward function is designed balance portfolio return with risk. We use the Entropic Value at Risk (EVaR) as risk measure. The reward R2 time & is defined
50

---

Published in Transactions on Machine Learning Research (10/2025)
R EVaR (R+)
(85)
where the negative portfolio returns are given by R, = (Vo V.)/Vo is the portfolio return up to time t, and a is the confidence level for EVaR calculation. The negative sign before EVaR ensures that minimizing risk corresponds to maximizing reward.
An episode terminates when either:
The end of the available market data is reached (t = T), or
The portfolio value drops zero (Vi & 0).
The enviromment functions comprehensive interface to the market, utilizing technique known as market replay to traverse historical data. Additionally, it operates as both a broker and an exchange; at each timestep, it processes the agent's ' actions to rebalance the portfolio according to the latest prices and specified allocations. As the trading day progresscs and new price data is acquired, the environment provides these updates to the agent as observations, accompanied by the Entropic Value Risk (EVaR) reward. For the scope of this study, we permit instantaneous rebalancing of the portfolio. Given an action at the environment updates the portfolio value Vn according to:
Transistion Dynamics:
V4+1 = V2 (1+ Σ oes T++1. c. Σ ar, a2-1,)))
(86)
where Tt+1, the return of asset at time f + 1, and e the transaction cost rate, which kept constant in our case. After rebalancing, the environment creates the next state Si+1 and procoeds to the next timestep
+ 1. It calculates the new portfolio value based on V4+1 and computes the reward Re, which it returns the agent.
C.5.1 Algorithm
We use Soft Actor Critic Haarnoja et al. (018 for policy optimization and incorporate the stochastic recursive updates for the convergence of EVaR sample estimate by augmenting the critic function. We employ the SAC algorithm with its default MLP policy as implemented in Stable Baselines 3. The actor (policy) and critic (value) networks consist of two hidden layers with 256 units each, using ReLU activations. The model is trained for 1000 timesteps, which we found sufficient for convergence in our environment.
Table 9: Hyperparameters used for training
Hyperparameter
Value
Learning rate
0.0003
Buffer size
1,000,000
Batch size
256
Gamma
0.99
Train frequency
TrainFreq(froqueecyy- unit=STEP)
Gradient sleps
Entropy coefficient (ent coef)
auto
Target entropy
-10.0
Tau
0.005
Policy kwargs
{use sde: False)
We see that our EVaR estimation more robust as adapts to the samples and the error band consistent, which shows contained variance and provides more conservative estimate of the sample averages, which is visualise portfolio and how it fairs
desirable in minimal risk portfolio setting. In Figs 28a 27a
51

---

Published in Transactions on Machine Learning Research (10/2025)
Efficlntt
614
10
2000
sepected kisk
(a)
Figure 27: (a) Efficient Frontier of EVaR at a = 0.01.
against other risk mcasures like VaR and CVaR, where we employ RiskFolicCajar (2024) The importance of these plots is discussed below.
to generate the plots.
Date
AAPL
HD
JNJ
JPM
KO
MSFT
PG
UNH
WMT
2014-01-03
-2.20%
-0.16%
0.90%
0.77%
-0.49%
-0.67%
-0.11%
0.71%
0.07%
-0.33%
2014-01-06
0.55%
-0.96%
0.52%
0.58%
-0.47%
-2.11%
0.24%
-1.15%
-0.60%
-0.56%
2014-01-07
-0.72%
0.49%
2.12%
-1.15%
0.30%
0.78%
0.97%
3.06%
0.76%
0.31%
2024-01-26
-0.90%
1.23%
-0.04%
-0.38%
0.36%
-0.23%
0.33%
1 1.99%
-1.71%
0.88%
2024-01-29
-0.36%
0.11%
-0.09%
0.26%
0.61%
1.43%
0.01%
0.27%
2.13%
0.47%
Table 10: Snapshot of portfolio assets and their adjusted returns over decade.
C.5.2 Our Portfolio and Results
Returns histogram provides visual representation of the distribution of portfolio returns, allowing for a comprehensive analysis of the performance of our EVaR agent compared to other risk mcasures such as VaR and CVaR. This comparison is crucial as it reveals how our EVaR-based portfolio exhibits a more conservative allocation, which characterized by a lower frequency of extreme negative returns. By evaluating the shape and spread of the distribution, we can gauge the robustness of our portfolio against adverse market conditions, highlighting the potential benefits of utilizing EVaR risk measure.
The comparison of various risk measures is essential for understanding the effectiveness of our portfolio strategy in managing risk. By analyzing the performance of the EVaR portfolio against traditional measures like VaR and CVaR, we can illustrate the advantages of adopting an entropic approach to risk assessment. Figure (28b illustrates that these comparisons indicate that our EVaR strategy not only limits potential losses but also adapts more dynamically changing market conditions. This adaptability is reflected in the reduced variance of estimated returns, which is desirable for investors seeking minimal risk.
Efficient Frontier: is key concept in modern portfolio theory, representing the set of optimal portfolios that offer the highest expected return for given level of risk or the lowest risk for given level of expected return. By analyzing the Efficient Frontier in the context of our portfolio optimization with EVaR measure, we can visually assess the trade-offs between risk and return for various portfolio allocations. Figure (27a) illustrates the Efficient Frontier for our EVaR-bascd portfolio, demonstrating that it achieves a favorable balance between risk and return. This indicates that our optimization strategy effectively identifies portfolios that maximize returns while minimizing downside risk.
52

---

Published in Transactions on Machine Learning Research (10/2025)
Protiatalyy Syyats
Compoand Annual Growth Rade (CAGR)
67076%
Portciko Peturns Malogra
Sygllhance Level
1,9999
21.2%%
Sine Nancaren fatura
STanda Desiatin [X]
6412
18.27
Semt Standard Devirtion (2)
12 335315%
0.852244
Recond Leane Partiel Mameen (ALPM))
14 11007
Value (Waft)
Conditional Bak (Cuad)
133302
0.204466
11.0% www.eoaaaeeeeeeooooo MELL 11EM
Entropic vaise mak (EVAR) 12)
0145236
13.30 3.5%
Reletivinte Value at Rist (RoVel 112
60908%
Sersmess
MJSE Meaauree based on Drondoanee 13)
Awerage Drandoon (AD0)
47911%
2312300
Conditiionl Granndome Rick (COwR)
14 1..500
0331195
Drewdren Bisk
1.996
Relabivttic
Rai (RLDaa))
13 19.3425%
0546120
Max Brawdown (M60))
25 22.06255
0328114
(b) Figure 28: (a) Returns histogram with comparison of VaR, CVaR and EVaR which shows EVaR portfolio being
(a)
the most conservative allocation. (b) Comparison of various risk measures using risk-adjusted portfolio.
D Reproducibility Details
The experiments were conducted on NVIDIA DGX A100, having an AMD EPYC 7742 64-core processor operating at 1.5 GHz ~ 3.39 GHz, with GDDR5 32 GB RAM, NVIDIA A100-SXM4-40 GB GPU at 1.41 GHz, and memory clocked 1.21 GHz. The operating CUDA version for PyTorch 1.13.1 is 11.6 for Python version 3.10.13. The supplementary material provided includes all the experiments with their obtained values, which are reported here in visual format.
53


Reviews:
[{"review_id": "B3TFGRuUZR", "reviewer": "Reviewer_aV7C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strength\n\n1. The paper investigates the stochastic approximation scheme of EVaR risk-measure in RL algorithms, which seems to be new and has been unexplored in the RL community. Moreover, the experimental results justifies the usage of EVaR risk-measure in RL.\n\n2. The paper is well-written and provides sufficient background to understand the paper.\n\n3. The authors provide an extensive experimental results ranging from Mujoco to portfolio optimization.\n\n- Weakness\n\n1. The algorithm’s implementation depends on four separate step-size parameters, making its performance potentially sensitive to their selection.\n\n2. In the prediction stage, the algorithm requires two long single trajectories for implementation, similar to a Monte Carlo approach, which may lead to high variance.\n\n3. In the optimization stage, the paper presents only asymptotic results, leaving non-asymptotic analysis as an open problem. This raises concerns about the sample efficiency of relying on an SPSA-based method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates a risk-sensitive reinforcement learning algorithm that employs the entropic value-at-risk (EVaR) risk measure, which is more conservative than VaR or CVaR. The authors formally define the EVaR measure and present analyses for both the prediction and optimization components. In the prediction stage, the authors develop a stochastic approximation scheme to estimate EVaR and establish both asymptotic and non-asymptotic convergence guarantees. In the optimization stage, which uses the simultaneous perturbation stochastic approximation method, asymptotic convergence to a stationary point is provided.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Can the authors comment more about when Assumption 1 and Assumption 2 are satisfied, and whether the assumptions are used in related literature?\n\n2. The authors do not provide any sketch of proof for Theorem 2 or 3 in the main manuscript. Brief steps to prove convergence analysis of ODE or its non-asymptotic analysis can be helpful in understanding the paper.", "broader_impact_concerns": "na", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DlQ4S5DB13", "reviewer": "Reviewer_BeF1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe experimental results span discrete (GridWorld) and continuous (MuJoCo) environments, as well as high-stakes applications like glycemic control and portfolio optimization. The results demonstrate that EVaR policies achieve higher cumulative returns and better risk management compared to baselines. \nThe paper provides a thorough theoretical analysis, including convergence proofs for the proposed algorithm under specific assumptions.\n\nWeaknesses: \nThe experimental results comparing CVaR and VaR should be moved to main text. It will also be beneficial to compare against other baselines such as D4PG-CVaR, SDPG-CVaR or QR-DQN for respective continuous or discrete action spaces. Also, it is not clear if or how to tune \\alpha?\nA discussion on limitations, sample efficiency, computational considerations for high-dimensional action spaces, and practical implication of using EVaR is missing. \nEquation overflow at the bottom of page 12.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel approach to risk-sensitive reinforcement learning (RL) by optimizing the Entropic Value at Risk (EVaR). This is a relatively new and less explored risk measure compared to traditional measures such as VaR and CVaR. The paper is well-written, with clear motivations, problem formulations, and a logical flow from theory to experiments. The theoretical part is dense and could be summarized first for clarity. Formal convergence proofs, together with experiments spanning tabular, continuous-control, healthcare and finance tasks, support the claim that EVaR can serve as a competitive risk-sensitive objective. The multi-timescale stochastic approximation algorithm for EVaR optimization is a significant contribution, especially in model-free RL settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The experimental results comparing CVaR and VaR should be moved to main text.\n\n2. A discussion on limitations, sample efficiency, computational considerations for high-dimensional action spaces, and practical implications.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aDWpweoVAE", "reviewer": "Reviewer_Ey3E", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is generally clearly written.\n* The proposed algorithm is theoretically grounded and demonstrates promising empirical performance.\n\n**Weaknesses**\n\n* While the algorithm is theoretically appealing, I found the connection between its advantages (e.g., managing tail risk) and the experimental setups (Grid World and MuJoCo) unclear. I would appreciate it if the authors could elaborate further on how these environments illustrate the benefits of EVaR-based optimization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of risk-sensitive reinforcement learning (RL) and proposes a multi-timescale stochastic approximation algorithm for optimizing the Entropic Value at Risk (EVaR) objective. The authors provide a theoretical analysis of the algorithm’s asymptotic behavior and evaluate its performance across several environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The notation $m$ is used to denote different concepts in various parts of the paper: the size of the action space (in the Problem Statement), the modulus in Theorem 1, and the grid size in Section 3.1. Please consider using distinct symbols to avoid confusion.\n* In Theorem 4, what does the abbreviation \"tss\" stand for?\n* In Section 3.1, the sentence \"The agent exhibits risk-aware efficiency\" is unclear. What specific notion of risk is being referred to, and in what way does the agent’s behavior reflect risk awareness?\n* Also in Section 3.1, the term \"true EVaR\" is used. How is this computed?\n\nMinor changes:\n* At the bottom of page 12, the equation is rendered off the page. Please adjust the formatting.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)