Risk-Averse Total-Reward Reinforcement Learning
Authors: Xihong Su
Jia Lin Hau
Gersi Doko
Kishan Panaganti
Marek Petrik
Venue: NeurIPS 2025 poster ()
Keywords: Q-learning
ERM
EVaR
total reward criterion
Primary area: reinforcement_learning

Abstract: Risk-averse total-reward Markov Decision Processes (MDPs) offer a promising framework for modeling and solving undiscounted infinite-horizon objectives. Existing model-based algorithms for risk measures like the entropic risk measure (ERM) and entropic value-at-risk (EVaR) are effective in small problems, but require full access to transition probabilities. We propose a Q-learning algorithm to compute the optimal stationary policy for total-reward ERM and EVaR objectives with strong convergence and performance guarantees. The algorithm and its optimality are made possible by ERM's dynamic consistency and elicitability. Our numerical results on tabular domains demonstrate quick and reliable convergence of the proposed Q-learning algorithm to the optimal risk-averse value function.

Risk-Averse
Total-Reward Reinforcement Learning
Xihong Su Department of Computer Science University of New Hampshire Durham, NH 03824 xihong. sucunh. edu
Jia Lin Hau
Department Computer Science University of New Hampshire Durham, NH 03824
jialin hau@unh edu
Gersi Doko
Kishan Panaganti
Department of Computer Science University of New Hampshire
Department of Computing Mathematical Sciences California Institute of Technology
Durham, NH 03824
(now at Tencent AI Lab, Seattle, WA)
www.@unh.occoooooo gersi gersi.doko@unh.oo edu
kpb. whb.researrch@maai.cc
Marek Petrik
Department of Computer Science University of New Hampshire Durbam, NH 03824
marek. petr kōunh. edu
Abstract
Risk-averse total-reward Markov Decision Processes (MDPs) offer a promising framework for modeling and solving undiscounted infinite-horizon objectives. Existing model-based algorithms for risk measures like the entropic risk measure (ERM) and entropic value-at-risk (EVaR) are effective in small problems, but require full access to transition probabilities. We propose Q-learning algorithm to compute the optimal stationary policy for total-reward ERM and EVaR objectives with strong convergence and performance guarantees. The algorithm and its optimality made possible by ERM's dynamic consistency and elicitability. Our numerical results on tabular domains demonstrate quick and reliable convergence of the proposed Q-learning algorithm to the optimal risk risk-averse value function.
Introduction
Risk-averse reinforcement learning (RL) an essential framework for practical and high-stakes applications, such as self-driving, robotic surgery, healthcare, and finance [6, 12, 16, 17, 19, 22, 25, 26, 29, 30, 34, 41, 42, 45, 47]. risk-averse policy prefers actions with more certainty even it means a lower expected return [30, 38). The goal of risk-averse RL is to compute risk-averse policies. To this end, risk-averse RL specifies the objective using monetary risk measures, such as value-at-risk risk (VaR), conditional value-at-risk (CVaR), entropic risk measure (ERM), entropic value-at-risk (EVaR). These risk measures penalize the variability of returns interpretably and yield policies with stronger guarantees on the probability of catastrophic losses [14].
major challenge in deriving practical RI. algorithms is that the model of the environment is often unknown. This challenge is even more salient in risk-averse RL because computing the risk of random return involves evaluating the full distribution of the return rather than just its expectation [17, 40]. Traditional definitions of risk measures assume known discounted [6, 18, 19, 22, 25, 26, 29] or transient [44] MDP model and use learning approximate global value or policy function [32].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent works have presented model-free methods to find optimal CVaR policies [23, 27] or optimal VaR policies [17], where the optimal policy may be Markovian or history-dependent. Another recent work derives risk-sensitive Q-learning algorithm, which applies nonlinear utility function to the temporal difference (TD) residual [38].
In RL domains formulated as Markov Decision Processes (MDPs), future rewards are handled differently based on whether the model is discounted or undiscounted. Most RL formulations assume discounted infinite-horizon objectives in which future rewards are assigned a lower value than present rewards. Discounting in financial domains is typically justified by interest rates and inflation, which make earlier rewards inherently more valuable [20, 36]. However, many RL tasks, such as robotics or games, have no natural justification for discounting. Instead, the domains have absorbing terminal states, which may represent desirable or undesirable outcomes [4, 15, 28].
Total reward criterion (TRC) generalizes stochastic shortest and longest path problems and has gained more attention [1, 9, 12, 13, 21, 31, 43, 44]. TRC has absorbing goal states and does not discount future rewards. A common assumption is that the MDP is transient and guarantees that any policy eventually terminates with positive probability. While this assumption is sufficient to guarantee finite risk-neutral expected returns, it insufficient to guarantee finite returns with risk-averse objectives. For example, given an ERM objective, the risk level factor must also be sufficiently small to ensure a finite return [35, 44].
This paper derives risk-averse TRC model-free Q-learning algorithms for ERM-TRC and EVaR-TRC objectives. There are three main challenges to deriving these risk-averse TRC Q-learning algorithms. First, generalizing the standard Q-learning algorithm to risk-averse objectives requires computing the full return distribution rather than just its expectation. Second, the risk-averse TRC Bellman operator may not be contraction. The absence of contraction property precludes the direct application of model-free methods in an undiscounted MDP. Third, instead of the contraction property, need to rely on other properties of the Bellman operator and an additional bounded condition to prove the convergence of the risk-averse TRC Q-leaniin algorithms.
As our main contribution, we derive a rigorous proof of the ERM-TRC Q-learning algorithm and the EVRR-TRC Q-learning algorithm converging to the optimal risk-averse value functions. We also show that the proposed Q-learning algorithms compute the optimal stationary policies for the ERM-TRC and EVRR-TRC objectives, and the optimal state-action value function can be computed by stochastic gradient descent along the derivative of the exponential loss function.
The rest of the paper is organized follows. We define our research setting and preliminary concepts in Section 2. In Section 3, we leverage the elicitability of ERM, define a new ERM Bellman operator, and propose Q-learning algorithms for the ERM and EVaR objectives. In Section 4, we give rigorous convergence proof of the proposed ERM-TRC Q-learning and EVRR-TRC Q-learning algorithms. Finally, the numerical results presented in Section 5 illustrate the effectiveness of our algorithms.
2 Preliminaries for Risk-aversion in Markov Decision Processes
We first introduce our notations and overview relevant properties for monetary risk measures. We then formalize the MDP framework with the risk-averse objective and summarize the standard Q-learning algorithm.
Notation We denote by IR and N the sets of real and natural (including 0) numbers, and R R (- J denotes the extended real line. We use R and 14+ to denote non-negative and positive real numbers, respectively. We use tilde to indicate random variable, such ā: Ω RR for the sample space Ω. The set of all real-valued random variables is denoted as X == Ro Sets are denoted with calligraphic letters.
Monetary risk measures Monetary risk measures generalize the expectation operator to account for the uncertainty of the random variable. In this work, we focus two risk measures. The first one is the entropic risk measure (ERM) defined for any risk level B > 0 and E X as [14]
ERMg [é] == -B-1 log E exp (-3 .),
and can be extended to ,3 E 00, oo) as ERM, [a] = lim FRM [
R[ and ERM [+] =
lim ERMg [ā]] = ess inf [Z]. ERM is popular because of its simplicity and its favorable proper-

---

ties in multi-stage optimization formulations [17-19, 44]. In particular, dynamic decision- with ERM allows for the existence of dynamic programming equations and Markov or stationary optimal policies. In addition, in this work, we leverage the fact that ERM is elicitable, which means that it can estimated by solving a linear regression problem [7. 11].
The second risk measure we consider is entropic value-at-risk (EVaR) which is defined for a given risk level a E (0, 1) and ã
EVaRa [+] := sup - log (a-IEexp (-B8)
sup ERMg g [2] + 8 log a, 3>0
B>0
and is extended to EVaRo [+] ess ess in] and EVaR, [x] = [[]] [2]. It is important to note that the
supremum in (2) may not be attained even when R is a finite discrete random variable [3]. EVaR addresses several important shortcomings of ERM [17-19, 44]. In particular, EVaR is coherent and closely approximates popular and interpretable quantile-based risk measures, like VaR and CVaR [2, 19].
Risk-Averse Markov Decision Processes We formulate the decision process Markov Decision Process (MDP) (S, A.prr μ4) [36]. The set S = {1, (....... represents a sink state. The set = (1.2..... A.] is the finite set of actions. The transition function p: S x A + As represents the probability p(s, a, s') of transitioning to s' E S after taking a e A in s E S. The function r: S x A transitioning from E S and E A to s' S. The vector μ E As is the initial state distribution.
S.e} S, is the finite set of states and e
+ IK represents the reward r r(s, a, )' E R associated with
As the objective, we focus on computing stationary deterministic policies II == A that maximize the total reward criterion (TRC):
max lim Risk TM FEll 50000
Σ r(kk Ces 5k+1)
where Risk represents either ERM or EVaR risk measures. The limit in (3) exists for stationary policies but may be infinite [44]. The superscript ı indicates the policy that guides the actions' probability, and the y indicates the distribution over the initial states.
Recent work has shown that an optimal policy in (3) exists, is stationary, and has bounded return for sufficiently small / as long as the following assumptions hold [44]. The sink state e satisfies that p(e, a. e) = 1 and r(e, a, e) = 0 for each E A, and pe = O. It is crucial to assume that the MDP is transient for any II:
pa.. [3, = 59 < oo,
S (e}.
(4)
Intuitively, this assumption states that any policy eventually reaches the sink state and effectively terminates. We adopt these assumptions in the remainder of the paper.
With risk-neutral objective, transient MDPs guarantee that the return is bounded for each policy. However, that is no longer the case with the ERM objective. In particular, for some / it is possible that the return is unbounded. The return is bounded for sufficiently small / and there exists an optimal stationary policy [44]. In contrast, the return of EVaR is always bounded, and an optimal stationary policy always exists regardless of the risk level a [39].
Standard Q-learning To help with the exposition of our new algorithms, we now informally summarize the Q-learning algorithm for risk-neutral setting [4]. Q-learning is an essential compo- nent of most model-free reinforcement learning algorithms, including DQN and many actor-critic methods [10, 24, 46). Its simplicity and scalability make it especially appealing.
Q-learning iteratively refines an estimate of the optimal state-action value function q SSx A + R, i N such that it satisfies
1+ (5t, ū.) (5, a.) 7. ã = ā. 5/) + max q. (3,, a') (, ā.).
(5)
d'E
Here, E; is also known as the TD residual. The algorithm assumes stream of samples (5 a. =1,..... sampled from the transition probabilities and appropriately chosen step sizes î7, to converge to the
3

---

optimal state-action value function. Note that qi is random because it is function of random variable. We restate lemma C.13 from [17] as the following lemma because it shows the connection between Q Q-learning and gradient descent.
Lemma 2.1 (Lemma C.13 in [17]). Suppose that f: RR is a differentiable u-strongly convex function with an L- Lipschitz continuous gradient. Consider x, R and gradient update for any step size & (0, 1/L]:
Fi+1 24 E. (
Then 31 E (1/4, 1/1) such that g/l E (0, and
Ti+1 = E/1) x; + E/l. x²,
where x arg min (x) is unique from the strong convexity of f.
The standard Q-learning algorithm can be seen stochastic gradient descent on the quadratic loss function f [5, 17, 38]. We leverage this property to build our algorithms.
Q-learning Algorithms: ERM and EVaR
In this section, we derive new Q-learning algorithms for the ERM and EVaR objectives. First, we propose the Q-learning algorithm for ERM in Section 3.1, which requires us introduce a new ERM Bellman operator based on the elicitability of the ERM risk measure. Then, we use this algorithm in Section 3.2 propose an EVaR Q-learning algorithm. The proofs for this section are deferred to Appendix A.
3.1 ERM Q-learning Algorithm
The algorithm we propose in this section computes the state-action value function for multiple values of the risk level / E B for some given non-empty finite set B C R4+.. The set B may be a singleton or may include multiple values, which necessary to optimize EVaR in the next section.
Before describing the Q-learning algorithm, we define the ERM risk-averse state-action value function and describe the Bellman operator that can be used compute it. We define the ERM risk-averse state-action value function gæ: S x A + R for each 3 E B as
[-- Σ
O' as a. 3) lim ERMA (s,a) 1-000
r6k āgs BaT1) "n (s,,,,,3)= max (8, sell
(6)
B
for each E S,a E 3, B. The limit in this equation exists by [43, lemma D.5].
The superscript (s, in (6) indicates the initial state and action are &p 8, âg a, and the policy determines the actions henceforth. We use Q = SxA to denote the set of possible state-action value functions.
To facilitate the computation of the value functions, we define the following ERM Bellman operator By: RSXA SRSA
(s, = :EMM 3 r(5,a,31) + max g0x, (3),
Vs S A,E. (7)
Note that the Bellman operator B3 applics to the value functions q(:.;, 3) for a fixed value of 3. We abbreviate B3 to B when the risk level is clear from the context.
The following result, which follows from Theorem A. shows that the ERM state-action value function can be computed as the fixed point of the Bellman operator.
Theorem 3.1. Assume some 3 E B and suppose that (s, a, 8) > -00, E S, E A. Then of (6) the unique solution to
= =B94 where = =".,,.1118)
(8)
Although the Bellman operator defined in (8) can be used to compute the value function when the transition probabilities are known, inconvenient the model-free reinforcement learning setting where the state-action value function must be estimated directly from samples.

---

We now turn to an alternative definition of the Bellman operator that can be used to estimate the value function directly from samples. To develop our ERM Q-learning algorithm. we need define Bellman operator that is amenable to computing its fixed point using stochastic gradient descent. For this purpose, we use the elicitability property of risk measures [7]. ERM known to be elicitable using the following loss function Eg: IR + IR:
Ez(=) = 3 (exp
(g(=) = 1 exp (-33
())()) - 3-exp (-xp))-- (-B (9)
The functions l' and e" are the first and second derivatives, which are important in constructing and analyzing the Q-learning algorithm.
The following proposition summarizes the elicitability property of ERM, which we need to develop our Q-learning algorithm. Elicitable risk measures can be estimated using regression from samples a model-free way.
Proposition 3.2. For each andd? 3 0:
ERMg [
arg min Ella y)],
(10)
yER
where the minimum is unique because Eg is strictly convex.
Using the elicitability property, we define the Bellman operator B3: Q + Q for 3 Ba
(o (1(s,,,,)) max ((5) a', y/ ]] Vs e S (1I)
(Bag)(s, a) = arg min E yER
d'EA
Following Proposition 3.2, we get the following equivalence of the Bellman operator, which implies that their fixed points coincide.
Theorem 3.3. For each 3 > and q E Q, we have that
B3q Bzg.
We can introduce the ERM Q-learning algorithm in Algorithm 1 using the results above. This algorithm adapts the standard Q-learning approach to the risk-averse setting. Intuitively, works as follows. Each iteration i processes a single transition sample to update the optimal state-action value function estimate q The value function estimates are random because the samples are random. The value function estimates are updated following stochastic gradient descent along the derivative of the loss function lg defined in (9). Each sample used to simultaneously update the state-action value function for multiple values of B E B.
Algorithm 1: ERM-TRC Q-learning algorithm
Input: Risk levels B C R++, samples: (3,, å; 8,), step sizes īj, E N, bounds Sming 2max Output: Estimate state-action value function q
(oss a, 8) + O, Ys E S, A:
2 for e N, E S, 8, do
if 8; Aa 0; then
(3) + r(s, a, 3,) + max a', ,), qi((sss,, 3)
if (2min 2min (3) < Emax) then return q 8) -00;
(s, 3) t a)
(exp (-8 (x()) - -1);
7 else (1+1(5, a, 3) + (1 a, ;);
There are three main differences between Algorithm and the standard Q-learning algorithm de- scribed in Section First, standard Q-learning aims to maximize the expectation objective, and the proposed algorithm maximizes the ERM-TRC objective. Second, the state-action value function update follows sequence of stochastic gradient steps, but it replaces the quadratic loss function with the exponential loss function fg in (9). Third, values in the proposed algorithm need an additional bounded condition that the TD residual (3) in [2min Zmax]]
Remark The q value can unbounded for two main reasons. First, as the value of 3 increases, the ERM value function does not increase, as shown in Lemma A.2, and can reach -o0, as shown in Theorem A. Second, the agent has some probability of repeatedly receiving reward for the same action, leading to an uncontrolled increase in the value for that action.
5

---

Note that if (3) is outside the [amin: zmax range, it indicates that the value of the risk level / is so large that q unbounded. This aligns with the conclusion that the value function may be unbounded for a large risk factor [35, 44]. Detecting unbounded values of q caused by random chance is useful but is challenging and beyond the scope of this work.
Remark 2. We now discuss how to estimate 2min and 2max in Algorithm 1. Assume that the sequences { and {(5i, ai, 8)) used in Algorithm 1, B > have
Emin (3) -2 max {|c- 8 dl, |c] ||rlloe 2maax (3) 2 max 3 dl, + ||r|
Where || max,, a (rs s') The constants and d can be estimated by Algorithm 3 in the appendix. For more details on the derivation, we refer the interested reader to Appendix A.4.
Now we explain how to leverage the elicitability and monotonicity of ERM to design Algorithm as stochastic gradient descent. Let (s,a, a, 8), E &,a E A, 8 B, and > 0, and define operators and as
① Ea,. (eo (Fs(,,))))))) + max (51 b)] Iy=a(e,,,))) a'EA
(G)
(12)
By
(G) = a
dy Eleo (r(s,,,))) ( + maxq(s d', () -
w)] ly=o(s.a.0)
(13)
d'EA
The operation (G) can be interpreted as the average gradient step where š1 is a random variable representing the next state. The operation (b) can be interpreted as the individual gradient step for a specific next state s'.
(Hq)(b) = (bb) - (G)) (Hx = q() - E (G)
(14)
The operation (b) can be interpreted as the q-update for a random variable &1 representing the next state. (b) can be interpreted the q-update a specific next state s'.
Lemma 3.4. Let b (s, a, 3). s S, a E A, B E B, 1 E ER is the vector with all components equal to I, and > 0, then the operator defined in (14) satisfies the monotonicity property such that
x(b) < (b) = (Hz) - (H)
(a)
(b) = 9 (6)
(b)
(H4) - 1. g S H - 1- S (g(b) + +1 g) & (Hq) +1.9.
(c)
The proof has two main steps. First, rewrite (b) in terms of the ERM Bellman operator in (11). Second, prove the monotonicity property of (b), which is necessary for the convergence analysis of Algorithm Fix 3 > O and some b = (s, a, 8), s S, E A, fix q and define
fly) = E. [e(r(ss a,) max a' v)]. -)) d'EA
(15)
The function f is E-strongly convex with L-Lipschitz-continoous gradient from Lemma 4.4. Let y" arg minyeR fly = (b) and 31 [1/L, such that
(b))(b) = (q - (b) @2) (bb) - (q(b) (b) (1 - (/1)g(()++(())) y
(16)
④ (1-E/1a0) + E/l (Bq)
Step (a) follows by algebraic manipulation from the definitions in (12) and (15). Step (b) follows from Lemma 2.1. Step (c) follows from the fact that β is the ERM Bellman operator defined in (11)
and y* is the unique solution (10).
3.2 EVaR Q-learning Algorithm
In this section, we adapt our ERM Q-learning algorithm to compute the optimal policy for the static EVaR-TRC objective (3). As mentioned in Section 2, EVaR is preferable to ERM because is coherent and closely approximates VaR and CVaR. The construction of an optimal EVaR policy follows the standard methodology proposed in prior work [19, 44]. We only briefly summarize it in this paper.

---

The main challenge in solving (3) is that EVaR is not dynamically consistent, and the supremum in (2) may not be attained. We show that the -VRR-TRC problem can be reduced sequence of ERM-TRCC problems, similarly to the discounted case [19] and the undiscounted case [44]. We define the objective function h: R++ R:
1(T,, 3) == ERM 1a" (5u, au, [)] + 8 log(a)
[
(17)
lim ERM I4 B
Σ
r(kk aks 5k+1) + 10g'a))((a).
2-000
Recall that so is distributed according to the initial distribution and ão is the action distributed according to the policy л. The equality in the equation above follows directly from the definition of the value function in (6).
We now compute a 8-optimal EVaR-TRC policy for any å > 0 by solving sequence of ERM-TRC problems. Following prior work [44], we replace the supremum over continuous / in the definition of EVaR (2) with maximum over a finite set ((8,, 6) of discretized /3 values chosen as
[(3..0) 6) = {Bo, ,k..,...,,kk
(18a)
where 0 < 8o <1 < ... <
BK, and
Br log
log 1 a
log(O)
Bx+1 =
BK 2
K
(18b)
log _ Ba8
log(1 0)
for 0 := -8000/og((a)) and the minimal K that satisfies the inequality. Note that the construction in (18) differs in the choice of By from B( 6) in [19].
The following theorem summarizes the fact that the EVaR-TRC problem reduces to a sequence of ERM-TRC optimization problems.
Theorem 3.5 (Theorem 4.2 in [43]). For any > O and sufficiently small So >> 0, let
(m', (*)
arg max (xx8)) ((86:)
1((T, (3).
Then, the limits below exist and satisfy that
[-1
lim EVaR + 4000
Σ rBkk az 8k+1)
2 sup lim sup ( B) - 6.
(19)
well 40000 B>0
Algorithm 2: EEVRR-RR Q-learning algorithm
Data: desired precision d O, risk level 1), initial Bo > 0, samples: (5,, ā. 35), N
Result: d-optimal policy n² E 11 Construct B(3,, 8) as described in (18);
2
Compute (xī. ht (3)) by Algorithm for each 8 E B(B,, where h h²( max ((T,8);
3 Let 3* E arg max (eB(..) h (B);
return Tz.
Algorithm 2 summarizes the procedure for computing 8-optimal EVaR policy, Note that the value of ((T, 3) may be -o,, indicating either that the ERM-TRC objective is unbounded for 3 or that the Q-learning algorithm diverged by random chance.
Remark Note that Algorithm assumes a small 30 as input. The derivation 30 is shown in A.5. It is unclear whether obtaining prior bound on 3y without knowing the model is possible. We, therefore, employ the heuristic outlined Algorithm that estimates lower bound Emin and an upper bound Fmax the random variable of returns. Then, we set 8o = 86/(samm "
4 Convergence Analysis
This section presents our main convergence guarantees for the ERM-TRC Q-learning and EVaR-TRC Q- Oearming algorithms. The proofs for this section deferred to Appendix B.

---

We require the following standard assumption to prove the convergence of the proposed Q-learning algorithms. The intuition of Assumption 4.1 is that each state-action pair must be visited infinitely often. Note that when an individual episode terminates upon reaching the sink state, the agent restarts a new episode from a random initial state.
Assumption 4.1. The input Algorithm and Algorithm 2 satisfies that P s' 1 St, a,, md p(š;, 0,, s'), ss' E E ViEN,
almost surely, where Gi-1 == (m,, (5i, ār. 8)))
The following theorem shows that the proposed ERM-TRC Q-learning algorithm enjoys convergence guarantees comparable to standard Q-learning. As Remark 2 states, 2min and 2max are chosen to ensure that q values bounded.
Theorem 4.2. For 3 E B, assume that the sequence (ī))) and (8s, 0z 8, used in Algorithm / satisfies Assumption 4. and step size condition
Σ
n CO,
00.
where i E Ei E (,) = (s, a) 2, [Emin- Zmaxl almost surely, then the sequence (g:);
produced by Algorithm 1 convergences almost surely to 9oo such that goo B3
The proof of Theorem 4.2 follows approach similar to that in the proofs of standard Q-learning [8] with three main differences. First, the algorithm converges for undiscounted MDP, and B, is not a contraction captured by [8, assumption 4.4), which is restated as Assumption B.2. Instead, we show that B3 satisfies monotonicity in Lemma 4.3. Second, Assumption B.2 leads to a convergence result somewhat weaker than the results for contraction. Then, a separate boundedness condition [8, proposition 4.6] is imposed. Third, using the exponential loss function lg in (9) requires a more careful choice of step-size than the standard analysis.
Lemma 4.3. For B E B, let 1 E Th" represent the vector of all ones. Then the ERM Bellman operator defined in (11) satisfies the monotonicity property for some 9 > 0 and q" RSA - SXA and for all SSA + RSA it satisfies that
x S y Bøq
=
B&s S Bay
(a)
q¹
(b)
B3q- - g-
S BB(q - 9 1) S B s
B3q + g
(c)
Here, the relations hold element-wise.
The following lemma shows that the loss function Eg in (9) is strongly convex and has a Lipschitz- continuous gradient. These properties are instrumental in showing the uniqueness our value functions and analyzing the convergence of our Q-learning algorithms by analyzing them as form of stochastic gradient descent.
Lemma 4.4. The function Eg: [zminy Zmaxx - R defined in (9) is l-strongly conver with = eexp (-8 3 2max) and its derivative (2)2) is T-Lipschitz-contimuuus with = B exp (-ß- 2min).
Corollary 4.5, which follows immediately from Theorem 3.5, shows that Algorithm 2 enjoys conver- gence guarantees and converges to 5-optimal EVaR policy.
Corollary 4.5. For 10, and 5 O, assume that the sequence (fx) and (8i, ā. 8 used in Algorithm satisfies Assumption 4. and step size condition
Σ n = 00,
Σ 1=0
< 00,
1
where E e (š,,ā,) a), }. if 2 [zmin Zmax]. Then Algorithm 2 converges to the
5-optimal stationary policy x² for the EVaR- objective in (3) almost surely.
The proof of Corollary 4.5 follows similar approach to the proofs of Theorem 4.2. However, we also need to show that one can obtain an optimal ERM policy for an appropriately chosen that approximates an optimal EVaR policy arbitrarily closely.

---

1
1
t
1
P1
1
1
td
1
c
g
t
C
Figure I: EVaR optimal policy with = 0.2
Figure 2: EVaR optimal policy with 0.6
0.8
ENalt vouho
0.9
probabaitt
0.6
0.0
0.4
0.2
0.0
0.0
0000g
00000
-0.5
0.0
00.5
1.0
1.5
3.0
1.0
20 10f
30 36"
return
number of samples
Figure 3: Comparison of return distributions of two optimal EVaR policies.
Figure 4: Mean and standard deviation of EVaR values with a = 0.2 on CW domain
Numerical Evaluation
In this section, we evaluate our algorithms on two tabular domains: cliff walking (CW) [4] and gambler's ruin (GR) [44]. The source code is available https://gite EVaR_Q. Q. First, we evaluate Algorithm 2 on the CW domain. In this problem, an agent starts with random state (cell in the grid world) that uniformly distributed over all non-sink states and walks toward the goal state labeled by g shown in Figure 1. At each step, the agent takes one of four actions: up, down, left, or right. The action will be performed with a probability of 0.91, and the other three actions will also be performed separately with probability of 0.03. If the agent hits the wall, it will stay in its place. The reward zero in transitions except for the states marked with e, d, and The agent receives 2 in state g. 0.004 in state d, -05,, -0.6, -0.7. -0.8 and -0.9 in cliff region marked with e from the left to the right. an agent steps into the cliff region, it will immediately be returned to the state marked with b. Note that the agent has unlimited steps, and the future rewards are not discounted.
com/ mmm//u222119 19 ERM_
We use Algorithm 3 to compute e and d in Remark 2 and estimate the bounds of (,) in Algorithm 1. Figure 1 and Figure 2 show the optimal policies for EVaR with risk level a = 0.2 and a 0.6 separately. Since the optimal policy is stationary, it can be analyzed visually. The arrow direction indicates the optimal action take in that state. Note that the optimal actions for states e and g are the same and are omitted. In Figure I, the agent moves right to the last column of the grid world and then moves down to reach the goal. When the agent is close to the cliff region, it moves up to avoid risk. In Figure 2, different optimal actions are highlighted in blue. As we can see, for different risk levels a, when the agent is near the cliff region, it exhibits consistent behaviors to avoid falling off the cliff. For the remaining states, the agent behaves differently.
Second, to understand the impact risk aversion on the structure of returns, we simulate the two optimal EVaR policies over 48, 000 episodes and display the distribution of returns in Figure 3. The r-axis represents the possible return the agent can get for each episode, and the y-axis represents the probability of getting a certain amount of return. For each episode, the agent takes 20, 000 steps and collects all rewards during the path. When a = 0.6, the return -0.074, and its standard deviation is 0.228. When 1.92, and its standard deviation is 0.228. Overall, Figure shows that for the lower value of a, the agent has a higher probability of avoiding falling off the cliff.
(-1, 2). and its mean value 0.2, the return is in (0, 2]. its mean value is

---

1.5
EVaR value ditferenos
EVail value differenen
1.0
0.8
0.0
0.4
0.0
10
2.0 10"
10
number of samples
number of samples
Figure 5: EVaR value converges on CW
Figure 6: EVaR value converges on GR
Third, to assess the stability of Algorithm 2, six random seeds generate samples, compute the optimal policies, and calculate the EVaR values on the CW domain. In Figure 4, the risk level is 0.2, the axis represents the number of samples, and the axis represents the mean value and standard deviation of the EVaR values. The standard deviation is 0.015, and our Q-learning algorithm converges similar solutions across different numbers of samples. Note that our Q- learning algorithm could converge to a different optimal EVaR policy depending on the learning rate and the number of samples.
Finally, we evaluate the convergence of Algorithm which approximates the EVaR value calculated from the linear programming(LP) [44] on CW and GR domains. The axis represents the number of samples, the axis represents the difference in EVAR values computed by LP and Q-learning algorithms, and the risk level is 0.2. Figure and Figure 6 show that the EVaR value difference decreases to zero when the number of samples is around 20,000 on the CW and GR domains. We illustrate the convergence results for 0.2, but the conclusion applies any value. Overall, Q-learning algorithm converges to the optimal value function.
Conclusion and Limitations
In this paper, we proposed two new risk-averse model-free algorithms for risk-averse reinforcement learning objectives. We also proved the convergence the proposed algorithm under mild assump- tions. To the best of our knowledge, these are the first risk-averse model-free free algorithms for the TRC criterion. Studying the TRC criterion in risk-averse RL is essential because has dramatically different properties in risk-averse objectives than the discounted criterion.
important limitation of this work is that our algorithms focus on the tabular setting and do not analyze the impact of value function approximation. However, the proposed algorithm is not limited tabular representations. It a general model-free gradient-bart based method that can combined with any differentiable value function approximators. This work lays out solid foundations for building scalable approximation reinforcement learning algorithms. particular, strength of Q-learning is that it can be coupled with general value function approximation schemes, such as deep neural networks. Although extending convergence guarantees such approximate scttings is challenging, convergence results show that Q-lcarning algorithm is sound foundation for scalable, practical algorithms.
Our theoretical analysis has two main limitations that may preclude its use in some practical settings. First, we must choose the limits Zmins Zmax. If these limits are set too small, the algorithms may fail to compute a good solution, and if they are too large, the algorithms may be excessively slow to detect the divergence of the TRC criterion for large values of B3. may possible to use existing TD algorithms to improve how quickly we detect the divergence. The second limitation is the need to select the parameter 3n that guarantees the existence for 3o. proposed in [44], requires several runs of Q-learning just to establish appropriate 3o.
8-optimal EVaR solution. natural choice
10

---

Acknowledgments
We thank the anonymous reviewers for their detailed reviews and thoughtful comments, which significantly improved the paper's clarity, This work was supported, in part, by NSF grants 2144601 and 2218063.
References
[1] Mohamadreza Ahmadi, Anushri Dixit, Joel W Burdick, and Aaron D Ames. Risk-averse stochastic shortest path planning. IEEE Conference on Decision and Control (CDC), pages 1199,5204, 2021.
[2] A. Ahmadi-Javid. Entropic Value-at-Risk: A new coherent risk measure. Journal of Optimiza- tion Theory and Applications, 155(3):1155-1123, 2012.
[3] Amir Ahmadi-Javid and Alois Pichler. An analytical study of norms and banach spaces induced by the entropic value-at-risk. Mathematics and Financial Economics, 11(4):527-550, 2017.
[4] Barto Andrew and Sutton Richard S. Reinforcement learning: an introduction. The MIT Press, 2018.
[5] Kavosh Asadi, Shoham Sabach, Yao Liu, Omer Gottesman, and Rasool Fakoor. TD convergence: An optimization perspective. Advances in Neural Information Processing Systems, 36, 2024.
[6] Nicole Bäuerle and Alexander Glauner. Markov decision processes with recursive risk measures. European Journal of Operational Research, 296(3):953-92)) 966,, 2022.
[7] Fabio Bellini and Valeria Bignozzi. On elicitable risk measures. Quantitative Finance, 15(5):725- 733, 2015.
[8] DP Bertsekas. Neuro-dynamic programming. Athena Scientific, 1996.
(9] Alon Cohen, Yonathan Efroni, Yishay Mansour, and Aviv Rosenberg. Minimax regret for stochastic shortest path. Advances in neural information processing systems, 34:28350-28361, 2021.
[10] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096-1105. PMLR, 2018.
[11) Paul Embrechts, Tiantian Mao, Qiuqi Wang, and Ruodu Wang. Bayes risk, elicitability, and the expected shortfall. Mathematical Finance, 199 100111999-111111 1217, 2021.
[12] Yingjie Fei, Zhuoran Yang. Yudong Chen, and Zhaoran Wang. Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning. Advances in Neural Information Processing Systems, 34:20436-20446, 2021.
[13] Yingjie Fei, Zhuoran Yang, and Zhaoran Wang. Risk-sensitive reinforcement learning with function approximation: A debiasing approach. In International Conference Machine Learning, pages
PMLR, 2021.
[14] Hans Follmer and Alexander Schied. Stochastic finance:: introduction in discrete time. De Gruyter Graduate, 4th edition, 2016.
[15] Haichuan Gao, Zhile Yang. Tian Tan. Tianren Zhang, Jinsheng Ren, Pengfei Sun, Shangqi Guo, and Feng Chen. Partial consistency for stabilizing undiscounted reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 34(12):10359-103 2022.
[16] Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient risk- averse reinforcement learning. Advances Neural Information Processing Systems, 35:32639 32652, 2022.
11

---

[17] Jia Lin Hau, Erick Delage, Esther Derman, Mohammad Ghavamzadeh, and Marek Petrik. Q-learning for quantile MDPs: A decomposition, performance, and convergence analysis. arXiv preprint arXiv:2410.24128 2024.
[18) Jia Lin Hau, Erick Delage, Mohammad Ghavamzadeh, and Marek Petrik. On dynamic pro- gramming decompositions of static risk measures in Markov decision processes. In Neural Information Processing Systems (NeurIPS), 2023.
[19] Jia Lin Hau, Marek Petrik, and Mohammad Ghavamzadeh. Entropic risk optimization in discounted MDPs. In International Conference on Artificial Intelligence and Statistics, pages 47-76. PMLR, 2023.
[20] Wenjie Huang. Erick Delage, and Shanshan Wang. The role of mixed discounting in risk-averse sequential decision maassinnnmaiin.... 2024.
[21] Lodewijk Kallenberg. Markov decision processes. Lecture Notes. University of Leiden, 2021.
[22] Tyler Kastner, Murat A. Erdogdu, and Amir-massoud Farahmand. Distributional model equiva- lence for sensiiene reinforcement learning. In Conference on Neural Information Processing Systems, 2023.
[23] Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning CVaR policy. In AAAI conference on artificial intelligence, pages 4436 4443, 2020.
[24] Prashanth La and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive MDPs. Advances in neural information processing systems, 26, 2013.
[25] Thanh Lam, Arun Verma, Bryan Kian Hsiang Low, and Patrick Jaillet. Risk-aware reinforcement learning with coherent risk measures and non-linear function approximation. In International Conference Learning Representations (ICLR), 2022.
[26] Xiaocheng Li, Huaiyang Zhong, and Margaret L Brandeau. Quantile Markov decision processes. Operations research, 70(3): 1428-1444774 1447, 2022.
[27] Shiau Hong Lim and lyas Malik. Distributional reinforcement learning for siskiiensitvve policies. Advances in Neural Information Processing Systems, 35:30977-30989, 2022.
[28] Sridhar Mahadevan. Average reward reinforcement learning: Foundations, algorithms, and empirical results. Machine learning, 22(1):159-195, 1996.
[29] Alexandre Marthe, Aurélien Garivier, and Claire Vernade. Beyond average return in Markov decision processes. Conference on Neural Information Processing Systems, 2023.
[30] Majid Mazouchi, Subramanya P Nageshrao, and Hamidreza Modares. A risk-averse preview- based Q-learning algorithm: Application to highway driving of autonomous vehicles. IEEE Transactions Control Systems Technology, 31(4):1803-1188 2023.
[31] Tobias Meggendorfer. Risk- stochastic shortest path. In AAAI Conference on Artificial Intelligence, pages 9858-9867, 2022.
[32] Thomas M Moerland, Joost Brockens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trends in Machine Learning, 16(1):1-118, 2023.
[33] Yurii Nesterov. Lectures on Convex Optimization. Springer, 2nd edition, 2018.
[34] Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial rein- forcement lcarning. In International Conference on Robotics and Automation (ICRA), pages 8522-8528. IEEE, 2019.
[35] Stephen Patek. On terminating Markov decision processes with a risk-averse objective function. Automatica, 1379-13779 2001.
12

---

[36] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2005.
[37] R. Tyrrell Rockafellar and Roger JB Wets. Variational Analysis. Springer, 2009.
[38] Yun Shen, Michael Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement learning. Neural computation, 26(7): 299-1138, 2014.
[39] Xihong Su. Efficient algorithms for mitigating uncertainty and risk in reinforcement learning. arXiv preprint arXiv:25 10. /7690, 2025.
[40] Xihong Su and Marek Petrik. Solving multi-model MDPs by coordinate ascent and dynamic programming. In Uncertainty in Artificial Intelligence, pages 2016-2033 2025. PMLR, 2023.
[41] Xihong Su, Marek Petrik, and Julien Grand-Clément. Evar optimization MDPs with total reward criterion. In Seventeenth European Workshop on Reinforcement Learning, 2024.
[42] Xihong Su, Marek Petrik, and Julien Grand-Clément. Optimality of stationary policies in risk- averse total-reward MDPs with EVaR. In JCML Workshop: Foundations of Reinforcement Learning and Control-Connection and Perspectives, 2024.
[43] Xihong Su, Marek Petrik, and Julien Grand-Clément. Risk-averse total-reward MDPs with ERM and EVaR. arXiv preprint arXi:22408.17286:1 2024.
[44] Xihong Su, Marek Petrik, and Julien Grand-Clément. Risk-averse total-reward MDPs with ERM and EVaR. In AAAI Conference on Artificial Intelligence, pages 20646-20654, 2025.
[45] Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning, arXiv preprint arXiv:2102.05333 2021.
[46] Gwangpyo Yoo, Jinwoo Park, and Honguk Woo. Risk-conditioned reinforcement learning: A generalized approach for adapting to varying risk measures. In AAAI Conference on Artificial Intelligence, pages 16521,, 2024.
[47] Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean-variance policy iteration for risk-averse reinforcement learning. In AAAI Conference on Artificial Intelligence, pages 10905-10913, 2021.
13

---

A Proofs of Section 3
A.1 Standard Results
For the model-based approach, Theorem A.I shows that for an infinite horizon, the optimal expo- nential value function = is attained by stationary deterministic policy and is a fixed point of the exponential Bellman operator. Following this theorem, we show that the ERM state-action value function can be computed as the fixed point of the Bellman operator shown in Theorem 3.1 in the main paper.
Theorem A.1 ( ([44), Theorem 3.3). Whenever w 20. > - there exists nt = (d)) E lss such that
2 = -0 = L o
and u is the unique value that satisfies this equation.
Lemma A.2 ((18), Lemma A.7). The function - ERM, [X] for any random variable X E X and B > 0 is continuous and non-increasing
Lemma A.3 ( (118), Lemma A.8). Let E X be a bounded random variable such that Imin < <
Tmax Then, for any risk level O, ERM, [ can be bounded as
B(Imax mmin)
[[
ERM [X] S E[X]
8
A.2 Proof of Theorem 3.1
Proof. The exponential value function w ws() is the unique solution to the exponential Bellman equation [44, theorem 3.3]. Therefore, for a state v"(s) (s), and the exponential state value function w² w=)s,, B E B, we have that
and the optimal policy u, state value function
v" (s) -8-- log( [og(-w"> (s))
(20)
Because of the bijection between w* (s) and o* (s), we can conclude that v*(s) is the unique solution to the regular Bellman equation.
The optimal state value function can be rewritten in terms of the optimal state-action value function in (21)
v² (s) max q (s, a, aεA
(21)
Then q² as, exists. The value q² (s, a, (8) is unique directly from the uniqueness of v*.
A.3 Proof of Proposition 3.2
Proof. The proof is broken into two parts. First, we show that Ea - y) is strictly convex, and minimum value of lg exists. Second, we show that the minimizer of E[ls(ā y)] is equal to
ERMg [z].
Take the first derivative of fo - y) with respect to
06g(ā v) a ( 8 (2--)) -
(5 v))
Oy (v) = B B- -.-1-112--1 3.12-
Oy
(22)
(v) -B-12-v) -1
Take the second derivative of la(ā y) with respect to y
826 O(e - (22-v)
0y²
Oy
(v) = e- - .
(23)
(1)) 3e -3-(2-1) 0
14

---

Therefore, Eg is strongly convex with respect to and the minimum value of Eg exists. Second, take the first derivative of [[(e(( y)] with respective y
0E (e (- - 1) + (i - y)
Oy
E [e 8-(2-0) - -1]
E 8-(2-v)
[e [1] E [ [[e
E B.
--y
E e -B-2
-3- logg
Therefore, the minimizer of B(eg( y)] is equal to ERM [+].
A.4 Derivation of Remark 2
Algorithm 3: A heuristic algorithm for computing 2 bounds
Input: Risk levels 3, 1-10, samples: (5i, ai, 8)), step sizes 7i E N Output: Estimated expectation value Tmin. and Tmax
To(s, a, B.) + 0, a(s, a, (3a) + O, Vs E
for i N S, A
it's = 5, A o, then (.e)) + r(s, (8, + ( - î (exp (-8 (()))) - 1); a,s, Be) --s,,,, +r(sa))
max. Bb) qi(s, Be):
else (1,1 3x) ((s,
8 + max goo (s, S,,
Tmin < min (s, S
10 Tmax f- max i(8 Be), ES
11 (mmax mmin) 1//8.
From line of Algorithm 1, we have
(B) r(s, a, $/) + max ( B) d'EA
A
Do some algebraic manipulation,
(,) < ||r||sc + 14 + /m/xxxx
(3) < |||||00 2/gl max
where
||r|
max S.EEE
Ir(s, a, s')l and |glmax = max (ā,(s, a, 8)), N.
sES,uEA
Let us estimate (4/ mmm-- For any random variable š, for any risk level B E B, ERMg [+] is non- increasing Lemma A.2 and satisfies monotonicity Lemma A.3,
B (( Tmun)² )2/8 & ERMg [ B[i
That
E[æ] (x Tminn ) g00 E[E)s E S
Then, for any B,
1g
max
1000 (5, B)]
sES.aEA
max
[
[[Imax 2minn² 1/81, ||
15

---

As mentioned in Section 2,
ERM [T]
lim ERMs [ --+0
Then
B ~
max Tce (8, (*)) sEB,aEA
We estimate [[ I maxx Emin by setting B. 1 in Algorithm 3. Therefore, for any 8 E B. Zmin (3) - 11 - 2 maxiic- 10c--a -d,, lc]h, Zmax (3) = 1|vll00 - max{/cc - 3 |c|
where max 700 (s, a, Bc), E S,a A and d (Xmax Imin /
A.5 Sg Derivation
Let us derive 36. For a desired precision 8 > 0, 30 is chosen such that
E
x ERM J [ S 8
(24)
From Lemma A.3, for random variable . and B O, we have
ERMg
[ < 8 (Imax Xmin) xmin)2/ )2/8
Then we have
9o [+]] S 80 (Fmax mmmin )2/8 When the equality conditions in (24) and (25) hold, have
ET
ERM WM
(25)
Bo (Xmax Pmin) )2/8 /
88
8o (xmax Eminn )
where Tmax and Tmin are estimated in Algorithm 3.
B Proofs of Section
B.1 Standard Convergence Results
Our convergence analysis of Q-learning algorithms guided by the framework [8, section 4]. We summarize the framework in this section. Consider the following iteration for some random sequence R where N 2,,,2, ... .n). then Q-learning is defined
Fst1 (b) = =(1 - -(1-0.()) - -.(()) + (b) ((HF:) + 7,(1),, 1=0,1,..
(26)
= = (b))) ((H)) + (. - -.(b)), i= 0, 0,
for all E N, where H RV + W some possibly non non-linear operator, 0, Ω + R4+ is step size, and Φ; + R is some random noise sequence. The random history Fi at iteration i = 1, 1,... is denoted
F (Fo.
Fs, 0o,
1 0o, 0.).
The following assumptions will be needed to analyze and prove the convergence of our algorithms. Assumption B.1. [assumption 4.3 in [8]]
(a) For every and we have
(b) There exists a norm || || on RR- and constants A and B such that
BlA A+ B||
Assumption B.2. [assumption 4.4 in [8]]
16

---

(a) The mapping H is monotone: that is, if S F, then Hr S Hr.
(b) There exists a unique vector r* satisfying Hr* = r".
(c) If e E R" is the vector with all components equal to 1, and 1) is positive scalar, then
Hr- e & H(r n. e) - H(r + n e) - Hr +7
Note that Assumption B.2 leads to convergence result somewhat weaker than the results for weighted contraction. Then we need a separate boundedness condition [8, proposition 4.6]. which is restated here as Proposition B.3.
maximum norm
Proposition B.3 (proposition 4.6 in [8]. Let Ti be the sequence generated by the iteration shown in Equation (26). We assume that the bth component F(b) of is updated according to Equation (26),
with the understanding that (b( = 0 ((b not updated at iteration dc(b) is a random noise term. Then we assume the following:
(a) The step sizes (b) are nonnegative and satisfy
Σ..0 = 20,
Σ 1=0
(b(
< oc,
(b) The noise terms de(b) satisfies Assumption B.I,
(c) The mapping H satisfies Assumption B.2.
If the sequence it is bounded with probability 1, then i converges to r* with probability I.
B.2 Proof of Theorem 4.2
B.2.1 Operator Definitions
Let (s, a, B), s E S,a E A. 3 E B. and & > O, we use some operators G defined in (12) and (13), and H defined in (14) for the convergence analysis of Algorithm 1 and Algorithm 2.
w)] ( (T((, max ( a', a) -v] ly=o(s,0,))
(b) By ga.
a'EA
= Eu.. 06g (r(s,a, max a', .) q(s, 5))]
a'EAA
31 is random variable representing the next state. When & is used in an expectation with superscript, such as E then it does not represent sample 8; with i = 1. Still, instead it represents the transition from 5g to 8n distributed as p(s,a,))
i R.. lo (r((s(( (1s,,,,8) + max max ,l'.'(('', a3, - w)] lu=o(e.a.)) (b) oy
a'EA
E ah (((s,,,))) 5') + max a' gls a, ]]]]
d'EA
(b) == (bb) - g. (G))(t),
(b) ((b) (b)
Consider random sequence of inputs (š,, aj 2 Algorithm We can define real-valued random variable 7(s, a) for E, a E A, 3 > (27).
(1I
(s)),,,a), if, = (s.a)
(s(s, a. B)
(27)
0,
otherwise
ii can be interpreted as the random noise.
/:/S, if (5,, (s,
0, a, 3)
(28)
10, otherwise
denote by F, the history of the algorithm until step which can be defined as
Fi (āo. Th 2o. 00. ,0
(29)
17

---

Lemma B.4. The random sequences of iterations followed by Algorithm 1 satisfies
Q6-1(5, a, 3) = as(s, a, 3) + (, a, 3) (Hq, + o; - qi)(s, a, 9). Vs E S, E where the terms are defined in Equations (12) to (14), (27) and (28).
Proof. We prove this claim by induction on The base case holds immediately from the definition. To prove the inductive case, suppose that E N and we prove the result in the following two cases.
Case I: suppose that = (5, 5) = (s, a. 3) = 6, then by algebraic manipulation
a++1 (b) [.(b) + ((0)))((())((()))))))))))))))))) 6,()) (b))
= @.(b) (((0))))(())))))))) + (Hzq4)()-
(H) (.(b))
@.(b) (()))((())((())))))) (+)))
a.(b) (10++000((0000 -((6))
(.(b) -O0)(((((())))))
(:(b) (0(0s(((8, 8() max ( d', max d'EA
-()-(())))))
Case 2: suppose that (3, 0) # (s, a, 3) = b, then by algebraic manipulation, the algorithm
does not change the q:
1 (b) = (b) + (H(O) + +.))) - (b)))
(b) +0 (b) + +b) (b())
(b)
B.2.2
Random Noise Analysis
We restate Lemma C.16 in [17] here Lemma B.5, which is useful for proving properties of the random noise. Lemma B.5 (Lemma C.16 in [17]. Under Assumption 4.1:
PP by Fil p(5,, āe s'), a.s.,
for each s' E and E N. Lemma B.6. The noise de defined in (27) satisfies
Bi Fi = 0s Vs S,
almost surely, where F, is defined in (29).
Proof. Let b = (s, a, 3). b. = (3i, ā. B) and E N. We decompose the expectation using the law of total expectation to get
(())(b) | = (() Fú by + bj T # b F.. B(ō.
Jr, b. bj PIù = b [F] (30)
From the definition in (27), we have
B(p((b) |.. b, * b) Pl. + bb
Then
E(g.(b) Fr, b. = -[(H=a)))- [F, b]
= E -ET-((G:4)) F = b]
E (1) = bb G--1 + (G))) F, = bj
@ E [IE=
F. b)
g (C) F,, b=
0
18

---

When $1 is used in an expectation with subseript, such as E then it does not represent sample & with 1. but instead represents the transition from sg to $ distributed as p(s, a, +). Step (a) follows the fact the randomness of (Gg; only comes from 8 when conditioning on
Fr, b, = b, 7/. and 91-1-
Then, we have
(b) Fil = (b) Fr, + b] P(b, *b B(A F,,b, - bj .Pb, = b Fil
0- Plù 61 [
0
Lemma B.7. The noise o; defined in (27) satisfies
E((é,(s, 0, F & A + Bl S E BiiN almost surely for some A, B E R F; is defined in (29).
Proof. Let b == (s, 3), b, = (3,, āz B) and e N. We decompose the expectation using the law of total expectation to get
(b) Fi] (b)² Fú, b; + Pb: * b + B(o,(b² Fr, b, b] Plù = b Fl 2b)
Fi, b, b) T0 b Fi)
This is because [E(::()) Fú, by # bj = 0. Let us evaluate [b): Fs, b. = b].
B(².(b)² Fe, b, = b] - E (Hx (HQ)))))) 2 |F,, =b]
= |8|
E (Gx (G)) 2 FF,,b; =b] E (-(=
? | Fr, b. 9 Fr,
82 E
Let us define 8,('', 33 in (31).
8,(5', = r(s,((s s') + max "'''','''))) ()
(31)
a'EA
+6) 2 |F, |
5²
E
E
b, nr Gi-1 Fr, =
[[G5, (G 9 | =
E
E..
(((0))((((11,8)) [.] - E (08 ()11 ) F, b)
E
(5().)()((1111))) B)) {.1)²] 100 '1) F. b)
E
E
IE
B
(6(0)) B)) s.)) |
F b, b
E
max FEES
(6.(5, 2 F.
19

---

(0)
552
E
max
(lg(5,(s', B)) dlz(0) D) F,, b, = b s'ES
(
max( s'ES
5 B) 1)² Fi, b;
e 9
B
-5²
E
max(o:(s'
6:(s', ,1)²² Fú, by =
e
s'ES
)
2..
11 +8. 1x))
@.mmin
Step (a) follows Lemma B.5 given that the randomness of (G (b) + (G) only comes from 8 when conditioning on F,,b; = b, ii and 9 Step (b) follows by substituting (:) (6) with (12), substituting (G) with (13), and replacing by using 5, defined in (31). The equality in step (c) holds because for a random variable v g (5, B)) | $1), the variance satisfies [ [ B[G:]] (E[=)) 2 Step (d) upper bounds the expectation by maximum. Step
(e) uses (0) = 0 from the definition in (9). Step (f) uses Lemma 4.4 to bound the derivative difference function of the step size. Step (g) derives the final upper bound since
||
max
|r(s,a,))),
11ãiloo = max [Tc( a, 3) aES,nEA
ES,EAA
max E'ES
(s' s (11rllo + 21l
s {1rllo> + 20 + (11 - 200²
(32)
=2|rr + 111
Then, have
El², Fi] (b)² F,, by = bj PP.p( = b Fil
B )2. (2 ||r + 8 11 1
@82min
Therefore, A = g2.
3 (,-6 "
)2. (2 ||r113) and B = 862 (g=Fmm )
B.2.3 Monotonicity of Operator H
Let us prove that the operator H defined in (14) satisfies monotonicity.
Proof of Lemma 3.4. First, we prove part (a): fix 3 and some b (s,0,3), A. Fix and define
[vv E. [Lg(r(s, a, 31) + max q(Bi,d', -v) v)] a'EA
(33)
The function f is strongly convex with a Lipschitz-continuon based on Lemma 4.4. Let arg mingeR (3) and 31 E [1/L, 1/2] such that
gradient with parameters & and
(bq) (q (b)
())
(bb) - E. (q(b))
(34)
(1 - (0))//(((()++++ + E/l-
(1 (l))(b))++++)))))()) + E/L (B))())
Step (a) follows the replacement with (12) and (33). Step(b) follows the Lemma 2. Step(c) follows
the fact that B is the ERM Bellman operator defined in (11) and y is the unique solution (10).
Given rbb) < vb), let prove that (Hx)(b) (b) < (H) Since the ERM Bellman operator B is monotone, we have
x(b) v(b) < 0 (bx)()) (B)) 0
20

---

Then we have
(bx) - (H)) = (1 - ( + (Bx)()) -11--//1 E/-w() + t/ (By)(()
- (/()(=(b) v(6) + ([Bx)((()) (By)
- 0
Second, let us prove part (b): (Hq) can be written as follows.
(b) (1 E/1)(bb) + g/l (B))
From [44, theorem 3.3], we know that q" (6) fixed point of B. Then (b) is also fixed point of H. That is, (b) = (b).
Third, let prove part (c), we omit b in the (H))(b) and rewrite it follows.
Hq = (1 g/l)g + g/l Bg
Given e R the vector with all components equal and if e a positive scalar, we show that
Hq c c (1g c. H(q c. Hg c.e
1) Let us prove Hq - e (( c
Hq c (1 {/l) q (E/l) Bq c.e
=(1 {/() .q - (1- - E/1) c (E/l) Bq (E/1) e
(1 (/))(--c-- - + (E/1)(Bq c.
@ (1 {/()(g c. + (g/1) c)))
= (q - c)
Step (a) follows from the law invariance property of ERM [19].
2) Let us prove H(g c < Hgg + c)
Because q S q e and the part (a) of the operator H, we have
H c. e) s H(q - ee
3) Let us prove H(g + c. = H(q) tc
H(g cc
= (()0)(++++e)) (E/)))=-tc
= (E)))(()=)++(()(((0 +.
@ [1-
(E/l) c
= - </1)q + (c- e (E/)
= </1)q +c (g)
(g) c
Step (a) follows from the law invariance of ERM. Then we have
Hg c. < H(g c. e) s H(q c. & H
B.2.4 Proof Theorem 4.2
Proof. Proof of Theorem 4.2 We verify that the sequence of our Q-learning iterates satisfies the
properties in Proposition B.3. The step size condition in Theorem 4.2 guarantees that we satisfy property (a) in Proposition B.3. Lemma and Lemma B.7 show that we satisfy property (b) in Proposition B.3. Lemma 3.4 shows that satisfy property (c) in Proposition B.3.
21

---

B.3 Proof of Lemma 4.3
Proof. First, from Theorem 3.3, we have
Baq Bag
4
ERM ((s,,, 31) + max (3), d', [0]
= arg min Ea,s ar + max ,' -y)
d'EA
yER
a'EA
For the part (a), ERM is monotone [19]. That is, for some fixed B E value, if x < y. then Byz & Bay.
For the part (b), Eg is a strongly convex function because (w()) = B B (- > 0. Then there will be unique y" value such that E=. ( (r(s, $1) + m a' v) attains the minimum
value, and the y" value is equal to q" (s, a, 33,, which follows from Proposition 3.2. So Bøq" = q" For the part (c), ERM is monotone and satisfies the aaw-invariance property [19].
- 1- BB(g 1 BBlg +1-1) BB(()) + +1.
Steps (1) and (3) follow from the law-invariance property of ERM. Step (2) follows from the mononotone property ERM.
B.4 Proof of Lemma 4.4
Proof. We use the fact that Eg is twice continuously differentiable and prove the i-strong convexity from [33, theorem 2.1.11]:
inf (=)=) inf 3exp ( (-3z) = exp (- -2max)
=E =E(Znin:=max)
=E(Fmin Fmox)]
To prove L-Lipschitz continuity of the derivative, using [37, Theorem 9.7], have that
sup
10(())==
sup
1eep 32) = B exp (- B2mn).
26 [Smin (mana))
26 (emii
B.5 Proof of Corollary 4.5
Proof. From Theorem 4.2, for some 8 E B, ERM-Q learning algorithm converges to the optimal ERM value function. Theorem 3.5 shows that there exists 5 5-optimal policy such that is an ERM- TRC optimal for some 3 E B. It is sufficient to compute an ERM-TRC optimal policy for one of those values. This analysis shows that Algorithm converges to optimal EVaR value funciton.
C
Additional Material of Section 5
The machine used to conduct all experiments referenced in Section 5 is single machine with the following specifications:
AMD Ryzen Thread ripper 3970X 32-Core (64) @ 4.55 GHz
256 GB RAM
Julia 1.11.5
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
22

---

Answer:[ Yess]
Justification: The authors believe that the abstract and the introduction represent the contri- butions accurately and precisely.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the work are discussed in the "Conclusion and Limitations" section and after Remark in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: assumptions include the equation (4) in Section 2 and Assumption 4.1 in section 4. The proofs are in the appendix.
Guidelines:
23

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
data files. One data file is available online, and the parameters of the other data file are described in Section 5.
Justification: The experiment based
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
24

---

Justification:
The code and data files are submitted as supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ Sewiideee/CoeSSumissstteee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: gww.//gg//e///eeeeeee /Coeee/modeeubbiaiaa asisnnolicyyy for more details.
/ //nips.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5 includes the experimental setting and details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yes)
Justification:
Section includes the standard deviations for multiple runs of the algorithms.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / /twurriurpp...// cc cc/public/eeeecccccccccc ctw.ieiiiiiiileenmmmecemmmne
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and confirm that their research conforms it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Because of the theoretical focus of this work, the authors have no reason to suspect that their work it poses any immediate societal impact good or bad. The purpose of this work develop tools and techniques for risk averse decision making in reinforcement learning.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The authors do not foresee any safety risks associated with this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: One data file from prior work properly cited this paper. All unattributed work is original to the authors' best knowledge.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer[[NA)
Justification:
Our work does not release any significant new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve any research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
28

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer:[NA]
Justification:
The core contributions of our work did not involve LLMs use.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: / haaprrppsscc//ooeeeeenneee////// /LLM) for what should or should not be described.
29


Reviews:
[{"review_id": "kWG8nYT99u", "reviewer": "Reviewer_27u3", "summary": "The authors proposed a model-free Q-learning algorithm for the Risk-Averse problem with ERM and EVAR metrics. To address the issue of unknown transition, the authors employed the elicitability property to replace the Bellman update with a gradient descent update, thereby proving the convergence of the proposed algorithm for both risk metrics.", "questions": "Please refer to the Strengths and weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality:2\n\nThe main idea of this paper is that the risk-averse MDP with total reward satisfies a Bellman equation with the modified Bellman operator. However, unlike in the discounted case, the Bellman operator is not a contraction. Thus, the authors utilized the monotonicity property and followed the proof idea from standard stochastic iterative algorithms.\n\nThe proof seems solid to me. However, I have a question about the assumption. In the paper, the authors mentioned that the idea of assumption 4.1 is that the state-action pair will be visited infinitely often. However, in the definition of MDP, it is said that there is a sink state, and the MDP is transient for any policy. Do these two statements conflict with each other? Please correct me if I’m wrong.\n\nMoreover, the discussion on the related work on risk-averse total reward MDP is missing. The authors list a series of works on the risk-averse RL or total reward MDP, but there is no comparison between this paper and the literature. \n\nClarity: 2\n\nThe entire writing of this paper is OK. But it is confusing in Algorithm 1 why the elicitability property can be replaced with a single step of gradient descent. Following the authors’ writing, I thought we should solve Eq. 11 in each iteration. However, only a single step is adopted in each iteration. The authors answer the question in Lemma B.8 and Eq. 32 in the appendix. I strongly recommend that the authors move these parts to the main text so that it would be clear that a single-step gradient descent is equivalent to one step of Q-learning.\n\nSignificant: 3\n\nThe risk-averse RL is an important topic in Reinforcement Learning. The authors put forward a tabular Q-learning model-free algorithm to solve the risk-averse RL in total reward MDP, which could be an important step for future work on more complex environments.\n\nOriginality:2\n\nThe technical novelty of this paper is somewhat limited. Firstly, the ERM Bellman operator and Bellman optimal equation are from the previous work [41, theorem 3.3]. In order to solve the unknown transition, the authors utilized the elicitability property. However, according to Lemma B.8 and Eq. 32 in the appendix, the proposed algorithm can be considered as a Q-learning algorithm, and thus, the convergence is guaranteed by the standard convergence analysis for stochastic iterative algorithms.", "was_revised": true, "final_justification": "The authors clearly stated the novelty and contributions of the paper. Thus, I raise my score of originality to 3 and final rating to 4.", "rebuttal": "Rebuttal:\n\nWe sincerely appreciate the detailed review and perspective.\n\n***I have a question about the assumption. In the paper, the authors mentioned that the idea of assumption 4.1 is that the state-action pair will be visited infinitely often. However, in the definition of MDP, it is said that there is a sink state, and the MDP is transient for any policy. Do these two statements conflict with each other? Please correct me if I’m wrong.***\n\nThanks for raising this point. These statements do not conflict with each other, and we will clarify this point in the paper. For Q-learning to converge, it requires data from multiple episodes. Even though an individual episode terminates upon reaching the sink state, the agent must restart and revisit state-action pairs across multiple episodes, satisfying the infinite visitation requirement for convergence.\n\n\n***The discussion on the related work on risk-averse total reward MDP is missing. The authors list a series of works on the risk-averse RL or total reward MDP, but there is no comparison between this paper and the literature.***\n\nThank you. To the best of our knowledge, this is the first paper that studies Q-learning for the risk-averse total reward criterion. Prior risk-averse RL work focuses either on model-based dynamic programs [41] or the average reward, finite-horizon, or discounted criterion [23, 27, 17]. We will emphasize this distinction in the final version of the paper. \n\n\n***But it is confusing in Algorithm 1 why the elicitability property can be replaced with a single step of gradient descent. Following the authors’ writing, I thought we should solve Eq. 11 in each iteration. However, only a single step is adopted in each iteration. The authors answer the question in Lemma B.8 and Eq. 32 in the appendix. I strongly recommend that the authors move these parts to the main text so that it would be clear that a single-step gradient descent is equivalent to one step of Q-learning.***\n\nThanks! We will move Lemma B.8 and Eq.32 to the main text in the final version. We will also add a detailed sketch of the proof of Lemma B.9 to the main text, clarifying the relationship between the elicitability property, the monotonicity property, single-step gradient descent, and one step of Q-learning in the final version.\n\n***The technical novelty of this paper is somewhat limited. Firstly, the ERM Bellman operator and Bellman optimal equation are from the previous work [41, theorem 3.3]. To solve the unknown transition, the authors utilized the elicitability property. According to Lemma B.8 and Eq. 32 in the appendix, the proposed algorithm can be considered as a Q-learning algorithm. Thus, the convergence is guaranteed by the standard convergence analysis for stochastic iterative algorithms.***\n\nWe take this concern seriously but believe that our algorithm and its technical analysis depart significantly from standard Q-learning approaches. To develop the algorithm, we leverage the elicitability property of ERM. That is, we need two definitions of the ERM Bellman operator and Bellman optimal equation. First, for each state-action pair, we define the standard ERM Bellman operator in Eq. (7) and the Bellman optimal equation in Eq. (8). Their correctness can be verified from [41, theorem 3.3]. Second, we leverage the elicitability property of ERM to define a new Bellman operator in Eq. (11) as the minimizer of the exponential loss function. We then prove the equivalence of the two ERM Bellman optimal equations, as stated in Theorem 3.3 of our paper. This demonstrates the correctness of our new ERM Bellman operator and Bellman optimal equation, which are essential to developing the proposed ERM-TRC and EVaR-TRC Q-learning algorithms.\n\nThe main challenge in the technical analysis is that under the total reward criterion, the ERM Bellman operator is not a contraction captured by (see Assumption B.2 in the paper). The lack of the contraction property prevents the direct application of the results from the standard Q-learning and the risk-averse total reward setting, which our paper addresses. Instead, we need to use other properties of the Bellman operator to prove the convergence of the proposed Q-learning algorithms. In addition, under the total reward criterion, q-value functions can be unbounded. Then, an additional bounded condition on temporal difference(TD) is needed to bound the q-value. This work uses the monotonicity of the Bellman operator and a separate bounded condition to prove the convergence of the proposed Q-learning algorithms."}, {"review_id": "tFX18UvKaD", "reviewer": "Reviewer_TEZu", "summary": "The paper proposes a risk-averse total-reward MDP framework and a Q-learning algorithm for learning optimal policies under both ERM and EVaR objectives. It provides theoretical analysis on the uniqueness of the solution, the convergence of the algorithm, etc. The effectiveness of the approach is demonstrated through two tabular task examples. This paper seems a direct extension of [1] (reference [41] in the paper).\n\n[1] Su, X., Petrik, M., & Grand-Clément, J. (2025). Risk-averse Total-reward MDPs with ERM and EVaR. Proceedings of the AAAI Conference on Artificial Intelligence, 39(19), 20646-20654.", "questions": "* In Section 5, the authors compare two optimal policies derived under different risk preference levels (\\alpha). The results show that with \\alpha = 0.6, the returns have a wider range and a lower mean, while the policy with \\alpha=0.2 achieves better overall performance with a similar standard deviation. In general, a controller tradeoffs expected total rewards against risk, and reducing risk often comes at the cost of performance. However, in this case, the more risk-averse policy appears to perform better with even lower risk. I suspect this may be due to the setting where the agent does not incur a penalty for remaining in the grid. Could the authors consider adding a small penalty for staying in place and then show how the expected total rewards and EVaR change as a function of \\alpha?\n\n* In practical applications, what guidance can the authors provide on selecting or tuning the hyperparameters for these optimization problems? Are there general strategies or heuristics that work well in practice?", "limitations": "Agree with the authors on Limitations in Section 6.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* The paper is clearly written and has a clear logical structure.\n* It offers a comprehensive theoretical analysis of the proposed algorithms, covering the problem formulation, uniqueness of the solution, convexity properties, the relationship between the EVaR-TRC and ERM-TRC problems, and convergence guarantees.\n\nWeaknesses:\n\n* Some technical details are missing, which may make it difficult for readers unfamiliar with prior work to fully understand the paper. While space limitations are understandable, including a more detailed sketch of key proofs would help improve clarity and accessibility.", "was_revised": true, "final_justification": "I decided to keep my rating. The authors answered my questions in their response regarding the risk preference parameter. I have no more questions.", "rebuttal": "Rebuttal:\n\nWe sincerely appreciate the detailed review and thoughtful comments.\n\n***Some technical details are missing, which may make it difficult for readers unfamiliar with prior work to fully understand the paper. While space limitations are understandable, including a more detailed sketch of key proofs would help improve clarity and accessibility.***\n\nThanks! We will add a more detailed sketch of key proofs in the main text in the final version of the paper. We will also restate the key theorems we use from the work [41] and provide a detailed explanation of the key connections between [41] and our work in the appendix of the final version. \n\n\n***In Section 5, the authors compare two optimal policies derived under different risk preference levels (\\alpha). The results show that with \\alpha = 0.6, the returns have a wider range and a lower mean, while the policy with \\alpha=0.2 achieves better overall performance with a similar standard deviation. In general, a controller trades off expected total rewards against risk, and reducing risk often comes at the cost of performance. However, in this case, the more risk-averse policy appears to perform better with even lower risk. I suspect this may be due to the setting where the agent does not incur a penalty for remaining in the grid. Could the authors consider adding a small penalty for staying in place and then show how the expected total rewards and EVaR change as a function of $\\alpha$?***\n\n\nThis is a great suggestion, thank you. First, let us review the numerical results from the closest related work(reference [41] in the paper). When the risk level $\\alpha$ increases from 0.2 to 0.7, the probability of getting the highest total reward increases. That is, reducing risk comes at the cost of performance. However, when the risk level $\\alpha=0.9$, the probability of getting the highest total reward is lower than the probability of getting the highest total reward with $\\alpha=0.7$. It is unclear what the risk level $\\alpha$ would be to maximize the probability of achieving the highest total reward in the work [41].\n\nSecond, the q-value functions computed by the proposed Q-learning algorithms converge to the optimal value functions computed by the linear programming in reference [41]. Our numerical results exhibit similar behaviors in the total reward criterion setting. \n\nThird, we will add a small penalty for staying in place and include the results of the expected total rewards and EVaR change as a function of $\\alpha$  in the final version.\n\n***In practical applications, what guidance can the authors provide on selecting or tuning the hyperparameters for these optimization problems? Are there general strategies or heuristics that work well in practice?***\n\nThis is a great question. In our numerical results, a decaying learning rate would be more effective than a fixed learning rate. Regarding the convergence results in Section 5, the learning rate with harmonic decay functions outperforms that with exponential decay functions. Also, we saw that the choice of the bounds $z_{min}$ and $z_{max}$ did not affect the convergence rate of our Q-learning algorithm significantly. We believe that more numerical experiments will be needed to understand the practical efficacy of specific strategies. Still, we expect that the general guidance for standard temporal difference bounds is likely to apply in our setting."}, {"review_id": "W3EFzUxZma", "reviewer": "Reviewer_xc6u", "summary": "This paper introduces ERM-TRC Q-learning which is a model-free RL algorithm in the risk-averse total reward setting using the ERM and EVaR objectives. Finding an optimal policy in the absence of knowledge of transition probabilities (which makes estimating risk averse rewards hard) is the key problem addressed in the paper. The paper proposes a novel Bellman operator leveraging the elicitability property of ERM, enabling a stochastic gradient descent approach for Q-learning. The work guarantees asymptotic almost sure convergence to the optimal policy. Technical challenges in algorithm design and establishing convergence guarantees include (a) generalizing the standard Q-learning algorithm to risk-averse objectives requiring computing the full return distribution rather than just its expectation; (b) risk-averse TRC Bellman operator not being a contraction (c) boundedness of the risk-averse objectives.", "questions": "N/A", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- Risk-Averse Total Reward Reinforcement Learning is an important problem to consider and the total reward criterion poses significant theoretical challenges in algorithm development and analysis\n- While elicitability has been explored for risk measures, its application to derive Q-learning in the undiscounted total-reward setting is novel given the challenges of non-contraction\n- Elegant definition of the Bellman operator $\\hat{B}_{\\beta}$ amenable to stochastic gradient descent\n- Well presented, intuitive structure of the algorithms and guarantees\n\n\n**Weaknesses:**\n- The paper considers only the tabular setting - extending the work to function approximation based methods and model-free gradient based methods will make it stronger\n- While, I understand that this is primarily a theoretical work, the experiments are rather limited lacking evaluation on tasks with larger state-action spaces, function approximation and no comparison against existing model-based algorithm baselines\n- As the authors acknowledge in the conclusion, the assumption that $\\beta$ has be to small enough and estimation of $z_{min}, z_{max}$ limits the practical applicability of the algorithm", "was_revised": true, "final_justification": "Why Weak Accept (4): Important problem, application of elicitability for establishing guarantees of Q-learning given the challenges of non-contraction through the elegant Bellman operator $\\hat{B}_{\\beta}$\n\nWhy not Accept (5): no comparison against model-based algorithm baselines in the experiments which also raises concerns about the practical applicability of the algorithm", "rebuttal": "Rebuttal:\n\nWe sincerely appreciate the detailed review and thoughtful comments.\n\n\n***The paper considers only the tabular setting - extending the work to function approximation based methods and model-free gradient based methods will make it stronger***\n\nThanks for making this point. We would like to clarify that the proposed algorithm is not limited to tabular representations. The algorithm is formulated for arbitrary model-free, gradient-based methods and can be used with any differentiable value function approximator. The tabular implementation serves as a concrete instantiation for experimental validation and aims to provide a clear and controlled validation that the value function learned by Algorithms 1 and 2 converges to the true optimal value function, consistent with the theoretical guarantees (see reference [41] in the paper). \n\nWe agree that extending the work to value function approximation and deep reinforcement learning methods is a valuable direction and would further demonstrate the scalability and generality of our approach. We intend to explore these extensions in future work.\n\n\n***As the authors acknowledge in the conclusion, the assumption that $\\beta$  has be to small enough and estimation of $z_{min}, z_{max}$ limits the practical applicability of the algorithm***\n\nWe expect to leverage existing algorithms of refining temporal difference(TD) bounds to improve practical efficacy of the proposed algorithms. We will discuss possible heuristics to develop a better methodology for establishing these parameters in the future work section of the paper."}, {"review_id": "OdzNMEZEq1", "reviewer": "Reviewer_Cf62", "summary": "This paper proposes Q-learning algorithms for optimizing risk-averse objectives (specifically, the entropic risk measure (ERM) and entropic value-at-risk (EVaR)) under the total reward criterion in transient, undiscounted MDPs. The authors derive convergence guarantees using the elicitability property of ERM and present numerical experiments on tabular domains.", "questions": "- Please provide clarification concerning the impact of truncation in the Cliffwalking experiments (concern 1).\n\nAdditionally, I have a few clarification questions to clear up confusions:\n- Sec1, line 45: the authors mention that \"Generalizing the standard Q-learning algorithm to risk-averse objectives requires computing the full return distribution rather than just its expectation\". Could you use the distributional Bellman operator using e.g., C51 [1], learn the full distribution over returns, and use this distribution to learn the optimal risk averse policy for a specific $\\alpha$?\n- Out of curiosity, since the MDP is transient, and that any policy eventually reaches the sink state (where there is zero reward), isn't this equivalent to the episodic setting?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n---------\n- The authors present a rigorous and well-motivated extension of Q-learning to risk-sensitive objectives in an undiscounted, infinite-horizon setting, with convergence guarantees.\n- The algorithm and its theoretical background are generally well explained, and the exposition is mostly clear.\n\nConcerns\n---------\n- In the cliff-walk experiment, the sink state should always be reached due to random chance (the probability of 0.09 of taking a different action than selected), where the agent receives a reward of 2. However, the evaluation is bounded at 20k steps. For $\\alpha=0.6$, did the agent actually learn the optimal policy? Could it be that the mean reward of $−0.074$ is solely due to episode truncation? If so, the experiment does not properly reflect the tackled setting (undiscounted infinite-horizon). Then, it would be valuable to change the setting such that episode truncation does not come into play, e.g., by increasing the probability of taking a different action than selected.\n- The choice of $\\alpha$ in different experiments (e.g., $\\alpha=0.3$ for stability plots, vs. $\\alpha=0.2,0.6$ elsewhere) seems arbitrary. Since the authors already run simulations for $\\alpha=0.2,0.6$, it would be more consistent and informative to reuse them in all plots.", "was_revised": true, "final_justification": "- The authors present a rigorous and well-motivated extension of Q-learning to risk-sensitive objectives in an undiscounted, infinite-horizon setting, with convergence guarantees.\n- The algorithm and its theoretical background are generally well explained, and the exposition is mostly clear.\n\nDuring the rebuttal:\n- The authors addressed the difficulty of using the distributional Bellman equation\n- Explained the reason for the cutoff at 20k steps for the cliffwalking experiments\n\nMy reasons for not scoring higher include:\n- heterogeneous experimental setup, making it hard to make generalizations or comparisons across experiments\n- the cutoff in cliffwalking experiments resulting in not all episodes ending in the sink-state. This has been partially addressed during the rebuttal, the authors mentioning that with the current cutoff the difference between the EVaR of the simulated returns with their true model-based EVaR is minimal", "rebuttal": "Rebuttal\n\nWe sincerely appreciate the detailed review and thoughtful comments.\n\n***In the cliff-walk experiment, the sink state should always be reached due to random chance (the probability of 0.09 of taking a different action than selected), where the agent receives a reward of 2. However, the evaluation is bounded at 20k steps. For $\\alpha = 0.6$, did the agent actually learn the optimal policy? Could it be that the mean reward of is solely due to episode truncation? If so, the experiment does not properly reflect the tackled setting (undiscounted infinite-horizon). Then, it would be valuable to change the setting such that episode truncation does not come into play, e.g., by increasing the probability of taking a different action than selected.Please provide clarification concerning the impact of truncation in the Cliffwalking experiments (concern 1).***\n\nThis is a great suggestion! For $\\alpha = 0.6$, the agent learns the optimal policy for 20k steps. If we keep increasing the number of steps, the agent will eventually enter the sink state and receive the reward of 2, and the mean reward will increase. The mean reward is affected by both the episode truncation and the optimal policy.\n\nWe will increase the probability of taking a different action and include a comparison of return distributions in the final version.\n\n\n***The choice of $\\alpha$ in different experiments (e.g., $\\alpha = 0.3$ for stability plots, vs. $\\alpha = 0.2,0.6 elsewhere) seems arbitrary. Since the authors already run simulations for , it would be more consistent and informative to reuse them in all plots.***\n\nThanks! We will choose consistent $\\alpha$ values for all plots in the final version.\n\n***Sec1, line 45: the authors mention that \"Generalizing the standard Q-learning algorithm to risk-averse objectives requires computing the full return distribution rather than just its expectation\". Could you use the distributional Bellman operator using e.g., C51 [1], learn the full distribution over returns, and use this distribution to learn the optimal risk averse policy for a specific $\\alpha$?***\n\nIt is a great question! The standard Bellman operator differs fundamentally from the distributional Bellman operator, which has three sources of randomness. They are the randomness in the reward, the randomness in the transition, and the next-state value distribution. We define the standard ERM Bellman operator and leverage the elicitability property of ERM to define a new ERM Bellman operator as the minimizer of the exponential loss function, and prove the equivalence of the two Bellman operators and the existence of a fixed point. We utilize the new ERM Bellman operator to develop the proposed Q-learning algorithms and compute the optimal risk-averse policy for a specific value of $\\alpha$.\n\nRecent work (Shiau Hong Lim,Ilyas Malik. Distributional Reinforcement Learning for Risk-Sensitive Policies. NeurIPS 2022) addresses the problem of learning a risk-sensitive policy based on the CVaR risk measure using distributional reinforcement learning. In particular, they show that the standard action-selection strategy when applying the distributional Bellman optimality operator can result in convergence to neither the dynamic, Markovian CVaR nor the static, non-Markovian CVaR. They introduce a new distributional Bellman operator, but the convergence properties of the proposed distributional Bellman operator remain unknown in the general setting.\n\nOur paper focuses on the ERM-TRC and EVaR-TRC objectives. It is unclear how to modify the regular distributional Bellman operator to fit with ERM and EVaR risk measures. The convergence properties of the modified distributional Bellman operator may be unknown in the general setting. This can be an interesting problem for future work. \n\n***Out of curiosity, since the MDP is transient, and that any policy eventually reaches the sink state (where there is zero reward), isn't this equivalent to the episodic setting?***\n\nGood question! Our setting is very similar to episodic RL. We use the term TRC because it is more specific and precise. We will clarify this point in the paper.\n\nIn our understanding of the episodic setting, the agent interacts with the environment for a fixed number of steps, and the episode ends after a certain number of steps or when a terminal state is reached. Each episode has a clear start and end. Episodes are usually finite. When an episode is forcibly halted, the last state is not considered a sink state. \n\nFor TRC criterion,  the start state is uniformly distributed among the non-sink states, as stated in assumption 2.1 of the reference [41]. Also, the number of steps to reach the goal state is not fixed and can vary significantly. The probability of eventually reaching the goal state is 1, but the expected number of steps required to do so is not bounded."}]

Author rebuttal:

Meta-review: This paper studies risk averse RL for ERM and EVAR risk metrics under the total reward formulation. A Q-learning-like model-free algorithm is proposed and extensive theoretical analysis is provided. Small scale empirical results are also provided to support the usefulness of the proposed algorithm. Overall, the reviewers agree that this paper studies an important problem and the approach is technically solid. I, therefore, recommend accept.
Decision: Accept (poster)