Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning
Authors: Mehrdad Moghimi
Hyejin Ku
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Distributional Reinforcement Learning
Risk Aversion
Spectral Risk Measures
Time-Consistency
Primary area: reinforcement_learning->online

Abstract: In domains such as finance, healthcare, and robotics, managing worst-case scenarios is critical, as failure to do so can lead to catastrophic outcomes. Distributional Reinforcement Learning (DRL) provides a natural framework to incorporate risk sensitivity into decision-making processes. However, existing approaches face two key limitations: (1) the use of fixed risk measures at each decision step often results in overly conservative policies, and (2) the interpretation and theoretical properties of the learned policies remain unclear. While optimizing a static risk measure addresses these issues, its use in the DRL framework has been limited to the simple static CVaR risk measure. In this paper, we present a novel DRL algorithm with convergence guarantees that optimizes for a broader class of static Spectral Risk Measures (SRM). Additionally, we provide a clear interpretation of the learned policy by leveraging the distribution of returns in DRL and the decomposition of static coherent risk measures. Extensive experiments demonstrate that our model learns policies aligned with the SRM objective, and outperforms existing risk-neutral and risk-sensitive DRL models in various settings.

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning
Mehrdad Moghimi
Hyejin Ku
Abstract
can stem from changing the objective from expectation to other risk measures such as the Conditional Value-at-Rishe (Bäuerle & Ott, 2011), coherent risk measures (Tamar et al., 2017). convex risk measures (Coache Jaimungal, 2023), and Entropic-VaR (Ni & Lai, 2022). Another approach is limiting the worst-case scenarios by using constraints such variance (Tamar al., 2012 dynamic risk measures (Chow & Pavone, 2013) in the optimization problem.
In domains such as finance, healthcare, and robotics, managing worst-case scenarios crit- ical, as failure to do so can lead to catastrophic outcomes. Distributional Reinforcement Learn- ing (DRL) provides natural framework incor- porate risk sensitivity into decision-making pro- cesses. However, existing approaches face two key limitations: (I) the use of fixed risk measures at each decision step often results in overly con- servative policies, and (2) the interpretation and theoretical properties of the learned policies re- main unclear. While optimizing a static risk mea- addresses these issues, its use in the DRL framework has been limited to the simple static CVaR risk measure. In this paper, we present a novel DRL algorithm with convergence guaran- tees that optimizes for a broader class of static Spectral Risk Measures (SRM). Additionally, we provide a clear interpretation of the learned policy by leveraging the distribution of returns in DRL and the decomposition of static coherent risk mea- sures. Extensive experiments demonstrate that our model learns policies aligned with the SRM objective, and outperforms existing risk-neutral and risk sensitive DRL models in various settings.
Another area of research that has gained attention for risk- sensitive RL (RSRL) is Distributional RI. (DRL. (Morimura et al., 2010; Bellemare et al., 2017). This paradigm diverges from the traditional RL by estimating the return distribu- tion instead of its expected value. DRL algorithms not only demonstrate notable improvements compared to con- ventional RL methods but also enable variety of new ap- proaches to risk mitigation. In this context, few risk mea- sures such as CVaR (Stanko & Macek, 2019; Keramati et al., 2020), distortion risk measure (Dabney et al., (a), En- tropic risk measure (Liang Luo, 2024), or static Lipschitz risk measure (Chen et al., 2024) have been explored.
In the DRL framework, applying fixed risk measure at each step leads to policies that are optimized for neither static nor dynamic risk measures (Lim & Malik, 2022). In this case, action selection at different states are not necessarily aligned with each other, which can lead to policies that are sub-optimal with respect to the agent's risk preference. This issue, known time inconsistency, is common challenge in risk-sensitive decision-making (Shapiro al., 2014). In- tuitively, this misalignment can be understood the fact that finding optimal policies starting from different states can yield different and inconsistent policies. To mitigate this, dynamic risk measures were introduced (Ruszczyíski, 2010), which evaluate risk at each time step, unlike static risk measures that assess risk over entire episodes. However, dynamic risk measures difficult interpret, limiting their practical applicability (Majumdar & Pavone, 2020; Gagne & Dayan, 2022).
1. Introduction
In traditional Reinforcement Learning (RL), the goal is to find policy that maximizes the expected return (Sutton Barto, 2018). However, considering the variations in rewards and addressing the worst-case scenarios are critical in some fields such healthcare or finance. A risk-averse policy help address the reward uncertainty arising from the stochasticity of the environment. This risk aversion
Department of Mathematics and Statistics, York Uni- versity, Toronto, Canada. Correspondence to: Hyejin Ku <hku@yorku.ca>>
Optimizing static risk measures is more interpretable since it can described as finding the policy that gives the best pos- sible outcome in the worst-case scenario. However, unlike dynamic risk measures, the risk preference that the policy optimizes for at later stages is unclear. Traditionally, calcu-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
lating these risk preferences has been limited CVaR due to computational complexity (Bäuerle & Ott, 2011; 1; Bellemare et al., 2023). However, we demonstrate that by leverag- ing the decomposition of coherent risk measures (Pflug & Pichler, 2016) and the return distribution within the DRI. framework, these evolving risk preferences can also be com- puted for more general spectral risk measures. The intuition behind this approach is that a decision maker selects an initial risk preference, but it may change as new information becomes available over time. It is important to emphasize that, unlike previous works (Chow et al., 2015; Stanko & Macek, 2019) that use the decomposition of CVaR to de- rive optimal policies, we utilize this decomposition only to explain the behavior of the optimal policy, not for policy optimization. In fact, Hau et al. (2023) have demonstrated that the decomposition of coherent risk measures cannot be reliably applied for policy optimization, and the optimality claims in those works are inaccurate.
Bäuerle & Ott (2011) demonstrates that the optimal policy depends on the history through a single statistic. By using the representation of CVaR introduced by Rockafellar & Uryasev (2000), they reduce the problem to an ordinary Markov Decision Process (MDP) with an extended state space:
1 B
122 bl 1)
max CVaRa (Z*) = max max FER
b+
FER BER
1 - 6] 1
= max BER
b+
max B HE
(2)
where u(z): :2 ++ [z 3--D bj denote a utility function that is 0 if x > b, and = = b otherwise. For a fixed policy, the supre- mum is attained at b = 1zz (a). With this representation, the problem is divided into inner and outer optimization problems, with the inner optimization addressing policy search for a fixed parameter b, while the outer optimization seeks the optimal parameter b.
The contributions of our work are as follows:
We propose novel DRL algorithm with convergence guarantees that optimizes static Spectral Risk Measures (SRM). SRM, expressed as a convex combination of CVaRs at varying risk levels, provides practitioners with the flexibility to define a wide range of risk pro- files, including the well-known Mean-CVaR measure.
Bäuerie & Rieder (2014) and Bäuerle & Glauner (2021) extend the idea of state augmentation to the case with a continuous and strictly increasing utility function and SRM as the risk measures. Their work demonstrates that suffi- cient statisties for solving these problems are cumulative discounted reward and the discount factor up the decision time. In each of these studies, state augmentation plays a crucial role in formulating Bellman equation to solve the inner optimization. Regarding the outer optimization in the CVaR case, only the existence of the optimal parameter b is shown. However, for SRM, which needs estimation of an increasing function for the outer optimization, a piece- wise linear approximation is used to allow transforming the problem into finite- dimensiona optimization problem. This is then solved using conventional global optimization methods.
We demonstrate that return distributions in the DRL framework enable the temporal decomposition of SRM,, allowing us to identify intermediate risk measures that preserve the optimality of the policy. These risk mea- sures provide insights into the agent's 's evolving risk preferences over time and enhance the interpretability of our algorithm.
Through extensive evaluations, we show that our model accurately learns policies aligned with the SRM ob- jective and outperforms both nesk-neutral and risk- sensitive DRL models in various settings.
In contrast to Bäuerle & Glauner (2021), we use the DRL framework derive SRM-optimal policies. This framework not only enables the transformation of the problem into finiteddimensional optimization problem but also allows us to use the closed-form solution to the outer optimization. As we will discuss in section 5, the DRL framework is also essential for identifying the intermediate risk measures for a time-consistent interpretation of optimal policy. While Bäuerle & Ou (201 1) demonstrate the existence of these intermediate risk measures for the simple CVaR case, they are not discussed for SRMs in Bäuerle & Glauner (2021).
2. Related Works
In this study, we focus on discovering policies with the highest risk-adjusted value:
max p(Z*). NEW
(L)
Here, p(Z") denotes the risk-adjusted value of the return of policy π and π denotes the set of distory-dependen policies. In general, optimal policies may depend on all available information up to the current time step. In cases such as risk- RL, optimal policies are typically stationary and Markovian. However, in risk-sensitive RL, the situation more complex. For instance, in the static CVaR case,
In the distributional RL framework, Dabney et al. (201 8a) introduce the use of risk measures beyond the expectation for action selection. However, their work does not address the theoretical properties of the resulting renk-sensitive poli- cies. For the static CVaR case, Bellemare et al. (2023) adopt
2

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
the formulation in Equation 2 and solve the problem using state augmentation. Lim & Malik (2022) focus special setting where an optimal Markov CVaR policy exists in the original MDP without requiring state space augmentation. More recently, Kim et al. (2024) study risk-constrained RI., using static SRMs to define the constraints. In contrast to our approach, which leverages a closed-fom form solution for the outer optimization. they rely on a computationally intensive gradient-base based method.
which satisfies So ho.z( 0:2
1o.2 (z) - (z)=2()==14 Fz [2) (
(=-Fz (a)) m(da)). (6)
3.2. Markov Decision Process
In this work, we aim to solve an infinite horizon discounted MDP problem presented by (A, R, P, 7, zo). In this tuple, A' and A denote the state and action spaces, R X x A - (R) the reward kernel, and P : X' x A + + (2) the transition kernel, and 7 E [0, 1) the discount factor. Without loss generality, we assume a single initial state represented by zo. Additionally, we assume that the rewards are bounded on the interval [RMIN, RMAX] and RMIN 0.
3. Preliminary Studies
3.1. Spectral Risk Measures
Let FF,,F represent a probability space and 2 represent the space of F-measurable random variables. The histori- cal information available at different time steps is denoted by a filtration 3 := (F)) where F, C Fr F for 0 S f. We also use p : Z + IR to denote risk mea- sure. In the context of this study, Z and p(2) are interpreted as the return and its risk-adjusted value, respectively. Let Fz = PZZ S 2), = E R denote the cumulative distribu- tion function (CDF), and Fz (u) = inf(2 E R: F2(2) Z :): u E [0. 1] denote the quantile function of a random vari- able Z. The SRM, introduced by Acerbi (2002), is defined
Let GO denote the sum of discounted rewards when starting at Xo and following policy π, i.c., G = - 7 RR. With GEIIN = RMIN (1 - 7) and GMAX = RMAX 10---) 7), it's easy to see that G takes on values in [GMIN, GMAX]] In this work, we aim to optimize the risk-adjusted value of the cumulative discounted reward based on the SRM. Since the formulation of SRM given in Equation 5 is more suitable in the context of policy-dependent returns, we write our objective as
max SRM,(G") = mEn
max max J(T, It) FER KEH.
(RM = ! Fz (u)o(u)du,
(3)
max NEH.
max J ((T;
(7)
TEW
where J(T, =
( [G + So h(o(u)ddu.
where the risk spectrum o [0, 1] + R+ is a left con- tinuous and non-increasing function with for ou)du and denotes the risk preference of the agent. The CVaRa (0, 1] is special case of SRM with the risk spectrum (uu) = _ [o. (u). The SRM can also be defined as convex combination of CVaRs with different risk levels (Kusuoka, 2001). With probability measure :: [0,1] 10, the SRM can be written as
In the remainder of this paper, maX J(T, h) is referred to as the inner optimization and finding the max is referred to as the outer optimization. To solve the inner optimization problem, we can reduce the search space from history-dependee policies in the original MDP to Markov policies in an augmented MDP with an extended state space denoted by X = X' x S x C where S = [GMIN, GMAX] represent the space of accumulated discounted rewards and C = (0. 1] represent the space of discount factors up to the decision time (Rieder & Bäuerle, 2011: Bäuerle & Glauner, 2021: Bastani al., 2022). The Markov policies in this x S C - P (A). where the
SRM, (2) (2) (2)==") / CVaRa
(4)
MDP take the form T X subscript h denotes the dependence of the policy on function h in the inner optimization problem and the space of Markov policies in this MDP is denoted by TM- With Xo = To. So = O, Co = 1, the transition structure of this MDP is defined by At ~ T.(. Xx, Sc, C.), R. ~ R(X,, At), X P(X,, A.), S S. C.Re, and C441 = 7Cr.
is shown that an SRM with a bounded spectrum also has a supremum representation
[[[(Z)}+ ( h(o(w)aau (5)
SRM = sup NCH
where H. denotes the set of concave functions h: R R-+ IR and l the concave conjugate of h (Pichler, 2015). In this formulation, the supremum is attained in h2.2 : R + RR
3.3. Distributional RL
Distributional Reinforcement Learning is sub-field of RI. that aims estimate the full distribution of the return, as op-
For a bounded and differentiable risk spectrum 6, we have dé(u) = - (ddd) and ((a)= = Sa L((du)
Proof is available in Appendix
3

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
posed to solely its expected value. To estimate the distribu- tion of the return, DRL uses a distributional value function, which maps states and actions to probability distributions over returns. With (x(xx, a) denoting the distribution of G* (z, a). the distributional Bellman operator is defined as
Algorithm 1 The QR-SRM Algorithm
Input: A random initialization of To or GTO
for 1 = 1, 2, do
Step 1: (The Closed-form Solution in Equation 6)
h4 arg max, J (= h)
Step 2: (The Inner Optimization (Algorithm 2))
(T")) (z, a) = E, (
x-=
= I,A -= a),
x = arg max; [ h1)
(8) where A' ~ T-) and brey : 2 H r + 72. The push- forward distribution (bng)# (A)) is also defined as the distribution of bR,"(G"(X' (X', There are multiple ways parameterize the return distribution, such the Categorical (C51 algorithm, Bellemare et al., 2017) or the Quantile (QR-DQN algorithm, Dabney et al., 2018b) rep- resentation. Here, we use the quantile representation as it simplifies the calculation of risk-adjusted values. With Ti = ii/Ni = O, .... .N representing the cumulative proba- bilities, the quantile representation is given by (2, a)
end for
preference by assigning different significance to quantiles of the return-distribution of the initial state. Fixing the function h. as displayed in Equation 6, can be interpreted as fixing the estimation of the return-distribution of the initial state. With this estimation, we can solve the inner optimization and find the optimal policy and its associated distributional value function. Since GT is approximated for each state- action pair, we can extract the quantiles of the initial state return-distribution and leverage the closed-form solution presented in Equation 6 to update our estimation of the func- tion / in the outer optimization." 3 Algorithm I presents an overview of our method. Note that an important property of the outer optimization 's closed-form form solution that we will use throughout this section is that Jo ho.,c((u))due 0 for any return-distriinn distribution G.
1 N E 00,(a,a), where the distribution is supported by (1 a) G(a,)) (f)) f = (7-1 + /22 Sis N.
3.4. Decomposition of Coherent Risk Measures
The decomposition theorem presented in Pflug & Pichler (2016) provides valuable tool for identifying conditional risk preferences. This theorem states that a law-invariant and coherent risk measure P can be decomposed as p(2)= sup: Blg Pg (Z| Fi), where the supremum is among all feasible random variables É satisfying [[E] = 1. In this theorem, if 5a is the optimal dual variable to compute the CVaR,,, and 0 s 5a S 1/a, <-- = [5 Fi), and 5 = So Ei ((da), the conditional risk preference is given by
For the inner optimization, let E (R) rep- resent the returnddistribution function over the augmented state-action space. We denote the corresponding return vari- able instantiated from 1J as G. The greedy selection rule, denoted by Gn. highlights its dependence on the function h. The greedy action at the augmented state (rr, 8, is then given by:
CVaR at level a, i... E [
E p(da) Pc (Z Fi) (1) - S: CVaRnes (Z F.)
(G, = arg maxE [b (s + cG(x,,, (10) aEA
(9)
&
Since function h is fixed until the optimal policy associated with it is found, we can analyze the convergence of the inner optimization with the Bellman operator 7 separately and then discuss the convergence of the overall algorithm.
In section 5, we show how the return cecurndiittriuution of each state can be used to calculate <p. This value can be used to calculate the new risk levels (aff)) and their weights (EF) ((( in the intermediate risk preferences. More- over, thorough discussion on the decomposability of risk measures and the time-consistency concept can be found in Appendix F.
We the index k and to show iterations on 7) and h. respectively. Therefore, T/k, denotes the kth iteration of return- cuuun-distibbtionn approximation when fy is used for greedy action selection and 7 denotes the distributional Bellman operator associated with ht. This way, the algorithm begins by setting f,, 8, C, a) = óg for all x E X, a E S, e E C, and a E A, initializzig ho based on Equation 6, and iterating 1,4 = ne.1- This iteration can also be expressed in
4. The Model
In this section, we propose an RL algorithm called Quantile Regression with SRM (QR-SRM) to solve the optimization problem outlined in Equation 7. In our approach, the func- tion h is fixed to update the return- -ittr-idstiibtionnn in the inner optimization, and then the return-distributionn is fixed to up- date the function h. The intuition behind our approach is as follows: The risk spectrum & determines the agent's 's risk
"This is in contrast with the work of Bäuerle & Glauner (2021), which relies on global optimization methods for the outer optimiza- tion. Additionally, estimating G for each state is crucial for iden- tifying intermediate risk measures that ensure a time-consistent interpretation the optimal policy.
4

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
terms of random-variable functions
Algorithm 2 The Sample Loss For The Inner Optimization of QR-SRM
C,,,I(2,8,c,a) - a( (X', Cx,,,,
Input:
v. p. (,,,,,), + + c' 60
(II)
where D shows equality in distribution and 1 denotes the action selection with GR., and lu.
q == # Eis (d' 6.) a' s arg mag. Q(a',(( a,) T900 (x, (2,,,c,a) + + ))) Output: ΣN By [et, (T (,, 0, (2,
For the outer optimization, if the optimal policy derived with fixed function hy is denoted by of and the return-variably of this policy is denoted by G the iteration on function h is given by h = arg max h))
(12)
we can perform a similar search. However, this approach is computationally expensive and impractical. Therefore, we do not adopt this method in our work and instead focus more scalable approach that balances theoretical soundness with practical feasibility.
NEH
Since the supremum in this optimization takes the form of Equation 6, this iteration can be viewed as updating function h with the return distribution of the initial state-action with the highest SRi(G*i). ). The following theorem discusses the convergence of our approach and its proof is provided in Appendix B and C.
Algorithm 2 outlines the sample loss for the inner opti- mization problem. A detailed discussion on the conver- gence of this Algorithm is available in Appendix E. is evident that, compared the sample loss of the risk- neutral QR-DQN algorithm, the only difference lies in the extended state space and action selection. In this algo- rithm, function h is defined by the return distribution of the initial state 0 == Fg (f:), where G == G Also, m. == = _u(da) = ((T--)) (((:)) denotes the signifi- cance of each quantile. The derivation of the action selection in this algorithm from function is presented in Appendix D.
Theorem 4.1. If Tuit denotes the greedy policy extracted from GK, and hú, then for all x E X, S, E C, and E A.
J(Tg,,h hn) 2 max J(T, - (00) GMAX (13) TEMM
A+
Additionally, J(mī,hi hi) is bounded and monotonically in- creases as I increases and provides a lower bound for our objective.
It is important to highlight the distinction between our con- vergence results and those established for the CVaR case in Bellemare et al. (2023). A key property of CVaR that enables convergence to the optimal policy is that the infor- mation required to find the optimal solution can be summa- rized in a single variable. In the CVaR case, let da denote the a-quantile in function h. Under this formulation, the greedy action selection in Fquation 10 can be simplified to the following equation, which aligns with the CVaR greedy policy discussed in Bellemare et al. (2023):
5. Intermediate Risk Preferences
In this section, we discuss the behavior of the optimal policy by identifying the intermediate risk measures for which the policy is optimized. We note that the calculations discussed here do not introduce any computational overhead in the optimization process and are provided solely to enhance the interpretability of our model. Suppose that G and G, rep- resent G (zzo, 0, 1) and GT (Iz, St, cz), respectively, with nT denoting the optimal policy. In the context of static SRM, the agent's risk preference is defined by assigning weights to the quantiles of G. To compute the weights for the quantiles of Gr, we establish the relationship between these two re- turn variables, which is where state augmentation becomes crucial.
Ja -0
a (I,, 8, c) arg maxI aeA
Note that in this case, we only need to track a single vari- able, 9a-s rather than all three variables qa. 8, and e, which simplifies action selection. In Bellemare et al. (2023), this variable is denoted by b, as seen in Equation 2. The signif- icance of using a single variable becomes evident in their work (Bellemare et al., 2023, Lemma 7.26), where finding the initial b requires searching over all possible values of b. This step is crucial for proving convergence to the optimal solution for CVaR.
Suppose the partial random return is denoted by GRAK for I k and k. k' E N. In traditional RL, the random return is decomposed into the one-step re- ward and the rewards obtained later: Go.so Ro + G1
With (z) 2 Gu: the Markov property of the MDP
allows writing this decomposition as GH GT(()) 2 Ro + GG (G))),), = x. In the extended MDP, क E TM also
From a theoretical perspective, for SRM, if we extend the state space to include every quantile required to define h,
has the Markov property, therefore we have the flexibility to
5

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making. in Distributional RL
break down the overall return into the 1-step reward and the rewards acquired after time i, Gr:so Gaa-1 + 7 G and write
0.6
0.4
s St, Ct).
(14)
0,3
Since G represents the average of 8t + Gr across all states, for any state (Iz, St, 02) and any quantile level a, we can determine the quantile level 3 for G2 such that Fe (a) = St + ct FF (8). The following theorem shows that { is, in fact, the ratio of these quantile levels (B/a), allowing us to define the agent's risk preference at future time steps with respect to Gt. The proof of this theorem can be found in Appendix G.
+
2
+
2
16
20
8
20
Figure I: A Markov process with the transition probabilities and rewards denoted on the edges and nodes. This process can also be considered as an MDP with a deterministic policy æ. In this way, the number in cach node denotes the r(x, r(x)).
Theorem 5.1. For any SRM defined with probability mea- sure μ., if gt is the optimal dual variable to compute the
(zo, a}, has the reward of 2 0.5 4 0.5² 20 and the probability of 0.6 0.2 0.12.
CVaR at level o, ie. E [gg = CVaRa (G), Xa = FG (a) and Fa, is the CDF of G2, we can calculate
Suppose the risk measure has the following form:
5r = =55 Fil with:
((G) 0.7 CVaRo.. + 0.3 CVaRo.s(G)
= Aa 1a
(15)
Here, a direct calculation of the risk measure shows that CVaRo..(G) = 5.25 and CVaRo..(GG) 6.375, therefore
Ct
we have
and derive the risk level and the weight of CVaRs, at a later time step with aff and &f /( For the general distributions with discontinuities, calculating se requires an additional term:
GG) = 0.7 5.25 + 0.3 6.375 = 5.5875.
We now apply the Decomposition theorem to compute this value using conditional risk measures and clarify the nota- tion used in this theorem. The first step involves analyzing how the risk levels a and the weights of each CVaRR, evolve at t 1. Table 1 presents these calculations, where çd is the optimal dual variable to compute the CVaR at level a satisfying E[e] = 1 and 0 < 5" < 1/a, i.e. E [5 CVaR, Additionally, we have & = IE [ga Fi] and = 5 ((d(()) 0.7 0.7- +0.3 -. and the updated risk levels and weights are computed using of, and & /(da)/E.
St EE = =F= s1) 1)//aaaaa aa- aa- (FF((a)-a)) (16)
Ct
where p pc, (2 (2= (poo)a)
a
Note that with the Quantile representation of return distribu- tions, we have Fa()a)) I 1/N, therefore the error of omitting the additional term in Equation 16 becomes neg- ligible as the number of quantiles increases. The intuition behind Theorem 5.1 and especially Equation 15 is to find & for the return distribution of future states (G,) and use the Decomposition Theorem in Section 3.4 to convert this value into the risk levels and weights of CVaRs and its associated risk measure at future states. To further elaborate on the components of the Decomposition theorem and Theorem 5. we discuss an example in detail later in this section. We also provide two additional examples in Appendix H, where we use Theorem 5.1 to demonstrate the change in the risk preferences and also analyze a single trajectory in more practical context within one of our experiments. These amples give a clear intuition behind the temporal adaptation of the risk measure.
Table 1: Details used to compute the conditional risk measures.
po(s)
G
g0
glla
X1
pc,(s)
G,
E
30%
5
2.5
1.25
50%
6
18%
0
1.25
x]
30%
1.25
1.0833
1.2
12%
9
0
0.41667
20%
14
16%
6
1.5625
1.25
40%
12%
7
0
1,25
30%
10
0.625
0.875
0.7
12%
30%
For instance, in state 2/. we have c0.4 1.25 and g0.8
1.0833. With & 0.7 1.25 + 0.3 1.0833 1.2, the conditional risk measure at this state is calculated as
(G X,- = x+)
Example 1. To illustrate the calculations of the conditional risk measures, consider the Markov process in Figure 1, where the number on the edges and nodes represent the transition probabilities and rewards, respectively. In this example, we use 7 = 0.5. For instance, the trajectory
1.25
CVaRR..(GG) + 1.0833
-CVaRo.soGi
1.2
1.2
0.73 CVaRo..(G.) + 0.27 CVaRo
0.73 6 0.27 8.69 6.73
6

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
Similarly for si. we have
6. Experimental Results
Pe [Xo = x
In this section, we study our model's performance with four examples. First, we start with the American Option Trading environment, commonly used in the RSRL literature (Tamar et al., 2017: Chow & Ghavamzadeh, 2014; Lim & Malik, 2022), followed by the Mean-reversion trading environment as outlined in the work by Coache & Jaimungal (2023), Finally, we tackle the more challenging Windy Lunar Lander enviromment. Ultimately, in Appendix I, we also examine the effect of the number of quantiles on performance. The details of these environments are available in Appendix J.
= -0.625 CV (G+) + 0.375 CVaRo.7(G.) = 00.625 + 0.375 8.86 = 8.32.
Recall that P(G) = BIg- P (G) Fi)l. Thus, to compute P(G), the probabilities of reaching #1 or of are reweighted using Ç. Moreover, since the return distribution of the initial state, conditioned on being at t 1, is given by 8, + c,G,, with st 2 and Ct- 0.5, we obtain the same value for p(G):
For each experiment, we report the expected value and the risk-adjusted value of the discounted return. Additionally, we employ diverse range of risk spectrums to derive poli- cies in our algorithm. We denote this approach as QR- (RM(Ø), where & represents the risk spectrum, with sub- scripts indicating the functional form. The specific cases are as follows:
P(G) = 0.3 (0.6- 1.2. 6.73 -...66.7--..4.7.8....... = 5.5875
Now, consider the goal of analyzing the evolution of the risk measure over time. With this goal in mind, we only need to calculate the updated risk levels and weights using CP- Theorem 5.1 demonstrates that, rather than directly computing & using E [5t Fi), we can leverage the CDF of return variables, as outlined in Equations 15 and 16, to perform the calculations. For X1 = at. we have:
QR-SRM(a.): CVaR with @a(u) _ (u).
a QR-SRM w: Weighted Sum of CVaRs (WSCVaR)
with da (u) = ES w; + ll (o..... (u).
QR-SRM(@x): Exponential risk measure (ERM) with
Φx (w) de T-e-x
6 2
c Fa, 100.4
- 1.25, 0.5 (9-2)
QR-SRM Dual Power risk measure (DPRM) with (u) = v(1 - uu)-
0.5
2
PG.l
G = Fa.
/00.8
(Fc - 0.8)
0.5
Pc
6.1. American Put Option Trading
0.2
-1/0.8
(0.88
0.8) 1.0833,
0.12
In this enviromment, we assume that the price of the underly- ing asset follows Geometric Brownian Motion and at cach time step, the the option- holder can either exercise or hold the option. For this example, we selected QR-SRM((2)) with a E {0.2, 0.6, 1.0]. The distribution of option pay- off is displayed in Figure 3(a). In this figure, the solid, dashed, and dotted vertical lines depict the CVaR1.0 (G), (0.46 - 0.4)/0.4 0.254 = 0.625, CVaRa.s(G)), and CVaRo.2 for each of these distribu- tions, respectively. We can see that QR-SRM(@ QR- 0.7/0.8 0.875. SRM(@u=0.s). and (R-SRM()==0:)) successfully finds (14)/088 -0000800..7..0
and for X1= = 27. we have:
011 Fa.
6 6-2 2
)/0.4
PG.
6-2)
0.5
(Fo(G) 0.4)/0.4
0.5
0.16
0.4
= 0.4/0.4
0.16
0-2
2
0.
Fa.
1/0.8
Fa.
0.5
the policy with the highest CVaR CVaRg,. and CVaRo.2(G)) respectively. The exercise boundary of each policy, as depicted in Figure 3(b), also shows that as a de- creases from 1.0 to 0.6 and then to 0.2, the policy becomes more conservative and the agent exercises the option sooner, leading to higher CVaRR, but lower CVaR2
08
114
6.2.
Trading Strategy
0.
65
In the algorithmic trading framework, the asset price fol- lows a mean-reverting process and the agent can buy or sell the asset to earn reward. To showcase our model's performance and its versatility to employ a diverse range of risk spectrums for policy derivation, we employ QR- SRM(0x-120). QR-SRM(0r=4.0), and QR-SRM (
(a) The Quantile function
(b) The CDF
Figure 2: The Quantile function and the CDF of the return distributions in states zo (black), oi (green), and 2i (blue) in
Example I.
7

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
measures.
MAR
For ERM4,0. DPRM2.0. and WSCVaR model ds can identify top-performing policies. Furthermore, we train QR-SRM(Os,..t algorithm with āz=[0.2, 1.0) and ( 0.5]. Compared to QR-SRM(@.=..))) and QR- SRM(@.=0.2) models, the results of this policy demonstrate the possibility of increasing the performance w.r.t one risk measure at the expense of decreasing the performance w.r.f to another in model.
(a)
(b)
Figure 3: Figure 3()) illustrates the distrihution of discounted returns for different policies. Figure 3(b) demonstrates the exercise boundary of each policy.
6.3. Windy Lunar Lander
To evaluate our algorithm in a more complex environment, utilize the windy Lunar Lander environment. The com- bination of the larger state and action spaces, along with the stochasticity in the transitions, makes this environment particularly challenging for training. As shown in Table 3, QR-SRM(2a=1.0 performs slightly worse than QR-DQN, but the difference within a standard deviation. This likely due the state augmentation used in our model. We also observed unusually low scores for the QR-CVaR al- gorithm, which were traced back to poor performance in 3 out of 5 seeds. Additionally, QR-iCVaR under-performed compared to QR-SRM at the same risk levels, suggesting that using a fixed risk measure, as in Dabney et al. (2018b), can lead to sub-optimal performance.
with [g=[0011, 0.6, 1.0] and 0.3, In Figure 4(a), the solid, dashed, and dotted vertical lines depict the WSCVaR (G), FRM1,.0(G). and DPRM (G) for each of these distributions, respectively. This figure demon- strates that our model effectively handles more complex risk measures and identifies the policy with the highest (RM
Several factors can contribute these discrepancies be- tween the objective and the evaluated performance. These include the use of function approximation for value func- tions and the inherent stochasticity of the environment. How- ever, a particularly significant factor for CVaR is the fact that these objectives focus exclusively on the left tail the distribution, overlooking valuable information in the right tail. As result, these algorithms are more likely to converge to sub-optimal policies. This limitation, commonly referred to "Blindness to Success" (Greenberg et al., 2022), is well-known issue with CVaR- approaches.
(a)
(b)
Figure 4: Figure 4(a) illustrates the distribution of discounted re- turns for different policies. Figure 4(b) displays the risk spectrums used to derive these policies.
For this example, we also conduct a comparative evaluation of our model against the rest-neutral QR-DQN model, its
risk-sensitive variant introduced in Bellemare et al. (2023) for static CVaR, and a model with risk-sensitive action se- lection similar to the approach proposed by Dabney et al. (2018a). We refer to these two models QR-CVaR and QR-iCVaR. As the QR-iCVaR with a=1 is identical to the QR-DQN model, only the results of one of them are dis- played.
A remedy to this situation, enabled by our model, is as- signing a weight to the expected value. The results for the RR-SRM(0a,. model show that this agent can not only achieve the highest WSCVaR ag but also improve the CVaRp.2 and CVaRo.s without a great impact on the ex- pected return. The results of this experiment show the im- pact of having a model with flexible objective that can adapt to the environment.
The first three columns of Table 2 represent the risk-adjusted values w.r.t CVaR, metric with a E {1.0, 0.5, 0.2). As expected, QR-SRM with CVaR as the risk measure and QR-CVaR exhibit similar performances. On the contrary, QR-iCVaR shows sub-optimal results for a=0.5. Even for values of 0.2 and 1.0, where the average risk-adjusted val- ues of all three models are close, QR-iCVaR achieves lower risk-adjusted value w.n. other risk measures. Also, a com- parison between QR- (R-SRM(2x=1.0) and QR-DQN shows that our model discovers superior policies w.r.t various risk
7. Conclusion
In this paper, we introduced a novel DRL algorithm with convergence guarantees designed to optimize the static SRM. Our empirical evaluations demonstrate the algorithm's 's abil- ity to learn policies aligned with SRM objectives, achieving superior performance compared to existing methods in
8

---

Beyond CVaR:
Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
Table 2: The performance of our model against the QR-DQN, QR- and QR-iCVaR models. Bold numbers represent the highest average score with respect to risk measure. The + symbol indicates the standard deviation across seeds.
Model
CV
CVaRo.s
CVa
ERM4.
DPRM2.
WSCVaR WSCVaR
aia
QR-SRM( 2..
1.43++0.33
0.03+:0.04
-1.36=#0.09
-0.37++0.05
0.43+0.02
0.35=0.02
0.04+00.0
QR-CVaR(a=1.0)
1.08
-0.02:0.10
-1.42:0.21
-0.110.113
0.40:0007
0.35
0,03+10110
QR-DQN
1.40:10.09
-0.24:0.17
-1.76:10.27
-0.67:1.19
0.21
0.1610.14
-0.1810.16
(R-SRM((00u.5)
0.7811002
0.2740.03
-0.4:10.05
0.00:0004
0.3810002
0.29:0003
0.17:0.03
QR- (R-CCVR(a=0.5)
0.79:10.08
0.2810.02
0,4110.11
0.02:+0.05
0.40
0.31
0.19
QR (.)iiCaaR(=0..))
0.82:10.17
0.14:00.44
0.36110.09
-0.00:10.02
0.32+10.77
0.33
0.23
OR-SRM(@.-0.2)
0.5611.08
0.21+0.03
-0.1110.4
0.05:10.01
0.29:10.05
0.2410.04
0.17:0.02
(a==CV)
0.64
0.2410.03
0..7710.03
0.05:10.01
0.33+10.03
0.26
0.19
QR-iCVaR(a=0.2)
0.40=#0.04
0.04+
0.01
-0.00+0..11
0.13+00.2
0.16=
0.11
(RRSSRM(O... (1))
0.84:0.05
0.28:1.02
0037:10077
0.05:10.03
0.42:00.02
0.35+0.02
0.23:00.03
QR (R-SSR(@v==2)
1.01
0.27
-0.2210.14
-0.03+0.06
0,46:00.2
0.3610001
0.19
QR-SRM ())
1.06
0.26++003
-0.57
-0.01=0.05
0.47+00.33
0.40
0.24
QR-SRM(2ds
1.11=0.08
0.26++0.04
-0.68+0.19
-0.05++0.08
0.48++.002
0.40=00.33
0.22+00.06
Table 3: The performance of our model against the QR-DQN, QR ER-C, and QR-iCVaR models. Bold numbers represent the highest average score with respect risk measure. The + symbol indicates the standard deviation across seeds.
the objective. In section 6.2, we empirically observed that our algorithm converges to policies similar to QR-CVaR, which has stronger convergence guarantees. However, an algorithm with stronger guarantees for convergence the optimal function h can enhance understanding of static SRM.
Model
CVaRas
CVaRas
WSCVaR
QR- DR-SRM().-1.))
27.86-112.8
10.17+14:07
-807+17.36
9.2++14.75
QR -.-
14.52=191
-1.1133.73
-15.5845.73
-0.55:3.24
DR-DON
32.73
3.79
-24.99123.39
4.23115.99
Impact Statement
QR SRM)
23.25=12288
4.1241244
11226116677
5.67=1205
QR- 5V
10.7725.34
6.2618.44
-2107711.00
-5.8819.13
QR (.V
21.72:22409
-1.33422.33
22.37338.35
-0002:30.97
This work advances Machine Learning by providing a princi- pled approach handling worst-case scenarios in sequential decision-making. We do not foresee any negative societal impacts. On the contrary, and reliability of AI systems in high-stakes domains such as finance and healthcare, where effective risk management is critical. We believe this contribution supports the devel- opment of safer and more trustworthy machine learning applications.
QR aRa)
22.281112.66
3.37213.09
1351111451
1.57213.53
QR-C CVaR(ozl)
11.8526.73
4.7816611
-19.9144199
-3,335532
QR-ICVaR(=-D.)
18.53-22.07
-2.00-27.34
-1882431668
004+22680
QR (..().
31,70E11.21
14.17:1346
-285:206.3
14.57=15.222
method can enhance the safety
variety of risk-sensitive scenarios. The advantage of using static SRM extends beyond performance; it also enhances interpretability. We showed that by applying the Decompo- sition Theorem of coherent risk measures and leveraging the return distribution available in the DRL framework, we can identify the specific objective that the optimal policy is optimizing for. This allows for monitoring the policy's behavior and risk sensitivity, and adjusting it if necessary.
References
Acerbi, C. Spectral measures of risk: A coherent represen- tation of subjective risk aversion. Journal of Banking and Finance, 26(7):1505-1511 2002.
A few limitations of our work that can pave the way for future research are as follows: i) Our value-based method is suitable for environments with discrete action spaces. The extension of our algorithm to actor-critic methods can make our approach available in environments with continuous action spaces. ii) In this work, we parameterized the re- turn distribution with the quantile representation. Using other parametric approximations of the distribution (Dab- ney et al., 201 8a; Yang et al., 2019) improvements that have been introduced for the quantile representation (Zhou et al., 2020; 2021) can potentially improve the performance of our risk-sensitive algorithm. iii) The algorithm to up- date the function h, or cquivalently the estimation of the initial state's s return distribution, provides lower bound for
Bastani, O., Ma, J. Y., Shen, E., and Xu, W. Regret Bounds for Risk-Sensitive Reinforcement Learning. Advances in Neural Information Processing Systems, 35:36259 36269, 2022.
Bäuerle, N. and Glauner, A. Minimizing spectral risk mea- sures applied to Markov decision processes. Mathemati- cal Methods of Operations Research, 94(1):35-6) 2021.
Bäuerle, N. and Ott, J. Markov Decision Processes with Average- Value-at-Rise criteria. Mathematicall Methods of Operations Research, 74(3):361-379, 201

---

Beyond CVaR:
Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
Bäuerle, N. and Rieder, U. More Risk-Sensitive Markov Decision Processes. Mathematics of Operations Research, 39()):00--120, 2014.
Hau, L.,, Delage, E., Ghavamzadeh, M., and Petrik, M. On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processes. In Advances in Neural Information Processing Systems, 2023.
Bellemare, M. G., Dabney, W., and Munos, R. A Dis- tributional Perspective on Reinforcement Learning. In Proceedings of the 34th International Conference on Ma- chine Learning, pp. 449-458. PMLR, 2017.
Huang, S., Dossa, R. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Araújo, J. G. M. CleanRL: High- quality Single- file Implementations of Deep Reinforce- ment Learning Algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
Bellemare. M. G., Dabney, W., and Rowland, M. Distri- butional Reinforcement Learning. The MIT Press, 2023. ISBN 978-0-262-37402--4
Huber, P. J. Robust Estimation of Location Parameter. In Kotz, S. and Johnson, N. L. (eds.), Breakthroughs in Statistics: Methodology and Distribution, pp. 492-518. Springer, New York, NY, 1992. ISBN 978-1-4612-4380- 9.
Chen, Y., Zhang, X., Wang, S., and Huang, L. Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation. In Proceedings of the 41st International Conference on Machine Learning, pp. 17911. PMLR, 2024.
Keramati, R., Dann, C., Tamkin, A., and Brunskill, E. Being
Optimistic to Be Conservative: Quickly Learning CVaR
Policy. Proceedings of the AAAI Conference on Artificial Intelligence, 44(04):44364444) 2020.
Chow, Y. and Ghavamzadeh, M. Algorithms for CVaR Op- timization in MDPs. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Kim, D., Cho, T., Han, S., Chung, H., Lee, K., and Oh, S. Spectral-Risk Safe Reinforcement Learning with Conver- gence Guarantees. In The ehirth- Annual Confer- ence on Neural Information Processing Systems, 2024.
Chow, Y. and Pavone, M. Stochastic optimal control with dynamic, time-consistent risk constraints. In 2013 Ameri- can Control Conference, pp. 390-395, 2013.
Kupper, M. and Schachermayer, W. Representation results for law invariant time consistent functions. Mathematics and Financial Economics, 2(3):189 180--199,, 2009.
Chow, Y., Tamar, A., Mannor, S., and Pavone, M. Risk- Sensitive and Robust Decision-Making: A CVaR Opti- mization Approach. In Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015.
Kusuoka, On law invariant coherent risk measures. In
Advances in Mathematical Economics, volume 3, pp. 83- 95. Springer Japan, Tokyo, 2001. ISBN 978-4-431-659377 2 Liang. H. and Luo, Z.-Q. Bridging Distributional and Risk-
Coache, A. and Jaimungal, S. Reinforcement learning with dynamic convex risk measures. Mathematical Finance, 2023.
sensitive Reinforcement Learning with Provable Regret Bounds. Journal of Machine Learning Research, 25(221): 1-56, 2024.
Dabney, W., Ostrovski, G., Silver, D., and Munos, R. Im- plicit Quantile Networks for Distributional Reinforce- ment Learning. In Proceedings of the 35th Interna- tional Conference on Machine Learning, pp. 1096-1105. PMLR, 2018a.
Lim, S. H. and Malik, I. Distributional Reinforcement Learning for Risk-Sensitive Policies. In Advances in Neural Information Processing Systems, 2022.
Majumdar, A. and Pavone, M. How Should Robot As- sess Risk? Towards an Axiomatic Theory of Risk in Robotics. In Amato, N. M., Hager, G., Thomas, S., and Torres-Torriti, M. (eds.), Robotics Research, Springer Pro- ceedings Advanced Robotics, pp. 75-84, Cham, 2020. ISBN 978-3-030-286194
Dabney, W. Rowland, M., Bellemare, M., and Munos, R. Distributional Reinforcement Learning With Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018b.
Gagne, C. and Dayan, P. Peril, prudence and planning as risk, avoidance and worry. Journal of Mathematical Psychology, 106: 102617, 2022.
Morimura, T., Sugiyama, M., Kashima, H., Hachiya, H.. and Tanaka, T. Nonparametric return distribution ap- proximation for reinforcement learning. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, pp. 799- 806, Madison, WI, USA, 2010. Omnipress. ISBN 978-1- 60558-907-7.
Greenberg, I., Chow, Y., Ghavamzadeh, M., and Mannor, S. Efficient Risk-Aver Averse Reinforcement Learning. In Ad-
vances Neural Information Processing Systems, 2022.
10

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
Ni, X. and Lai, L. Risk-Sensitive Reinforcement Learning Via Entropic- VaR Optimization. In 2022 56th Asilomar Conference on Signals, Systems, and Computers, pp. 953- 959, 2022.
Tamar, A., Chow, Y., Ghavamzadeh, M., and Mannor, S. Sequential Decision Making With Coherent Risk. IEEE Transactions Automatic Control, 62(7):3323-3338, 2017.
Pfuug G. C. and Pichler, A. Time-Consistent Decisions and Temporal Decomposition of Coherent Risk Func- tional. Mathematics of Operations Research, 41(2):682- 699, 2016.
Towers, M., Terry, J. K., Kwiatkowski, A., Balis, U., de Cola, G., Deleu, T., Goulão, M., Kallinteris, A., KG, A., Krimmel, M., Perez-Vicente R., Pierré, A., Schulhoff, S., J. Shen, A. T. J., and Younis, O. G. Gymnasium. Zenodo, 2023.
Pichler, A. Premiums and reserves, adjusted by distortions. Scandinavian Actuarial Journal, 2015(4):332-351 2015.
Yang, D., Zhao, L.,, Lin, Z., Qin, T., Bian, J., and Liu, T.-Y. Fully Parameterized Quantile Function for Distributional Reinforcement Learning. In Advances in Neural Informa- tion Processing Systems, volume 32. Curran Associates, Inc., 2019.
Rieder, U. and Bäuerle, N. Markov Decision Processes with Applications to Finance. 2011.
Rockafellar, R. T. and Uryasev, S. Optimization of con- ditional value-at-risk. The Journal of Risk, 2(3):21-41, 2000.
Zhou, F., Wang, J., and Feng, X. Non-Crossing Quantile Regression for Distributional Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 33, pp. 159199 Curran Associates, Inc., 2020.
Ruszczyúski, A. Risk-averse dynamic programming for Markov decision processes. Mathematical Programming, 125(2):235-26) 2611, 2010.
Zhou, F., Zhu, Z., Kuang, Q., and Zhang, L. Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning. In Twenty-Nitth International Joint Conference on Artificial Intelligence, volume pp. 3455-346 2021.
Shapiro, A. and Pichler, A. Time and Dynamic Consisteney of Risk Averse Stochastic Programs. Optimization Online, 2016.
Shapiro, A. and Ugurlu, K. Decomposability and time con- sistency of risk averse multistage programs. Operations Research Letters, 44(5):663-66 2016.
Shapiro, A., Dentcheva, D., and Ruszczyíski, A. P. Lec- tures on Stochastic Programming: Modeling and Theory. NOS-SIAM Series Optimization. Society for Indus- trial and Applied Mathematics Mathematical Optimiza- tion Society, Philadelphia, Pennsylvania, second edition edition, 2014. ISBN 978- 1-6---11199 197-342-6.
Stanko, S. and Macek, K. Risk-averse Distributional Rein- forcement Learning: A CVaR Optimization Approach:. In Proceedings of the 11th International Joint Confer- ence on Computational Intelligence, pp. 412-423, Vienna, Austria, 2019. SCITEPRESS Science and Technology Publications. ISBN 11888999-77888844
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Leam- ing Series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. ISBN 978-0-262-03924-6.
Tamar, A., Di Castro, D., and Mannor, S. Policy gradients with variance related risk criteria. In Proceedings of the 29th International Coference on International Confer- ence on Machine Learning, ICML' 12, pp. 1651-1655 Madison, WI, USA, 2012. Omnipress. ISBN 978-1-4503- 1285-1.

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
A. Property of the Closed-form solution
Using the SRM definition from Equation 4, we have
(Z)
CVaRa (Z)u(da)
[ F (a) - - [ (z 1116-117'00 (2) (a)) p(da)
(a)
(b) E
Fz (a) + (z Fz (a)) u(da)
= E (2 (2))
where step (a) utilizes the CVaR representation provided in Rockafellar Uryasev (2000), and step (b) applies Fubini's Theorem. Next, we that ha defined in Equation differentiable almost everywhere, with its derivative given by
[ Amassz/ton (013 a _
1z,2(2)=
= - ( (F2(2)))
Additionally, the infimum in the concave conjugate ( inf= (d(u) = - hg))(=)) achieved at any where é(u) (z) (F2(2)), which corresponds to (u). Therefore, we obtain
S. haz( = /// p(u). Fz (u) - ho.z (FZ du =z((1)
b(u) F (u)du S/ 10. (F (1)) du
SRM (Z) - E [ho,2
B. Proof of Convergence for Inner Optimization
In this section, we aim to demonstrate the convergence of the inner optimization algorithm to the optimal policy associated with a fixed function ht. Before discussing the main theorem, we must introduce several intermediate results about partial returns. Recall that J(T, hy) = E hi (G*)) + fo hi(d(u))du. Since fo hu()(u))du = 0, we define the mapping Vv.1 X xxxxxxxxx as follows:
Vv, E [1i (s (G,((1,,,,,a))
(17)
Similarly for policy m E MM. we define
(=,,,,,, E [h (s (x,,s,,,
(18)
The goal is to find an optimal deterministic policy क. E TM in the sense that
: (x, max V* (z,,,,,)
(19)
NIERM
With (x, ag, (x, s, c), we have the following recursive property for Vk,1 and V*:
Lemma B.l. For each (x, a) E X x S x A. we have
Ve41, s, a) Exaca [Vi, (X', S',"", and (X',
(20)
Additionally, for a policy E TM we have
4dh (I, Es, (V ('''''',,
(21)
12

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
Proof. The proof for both equations follows similar steps, we present the proof only for VK+1.- Consider partial trajectory that starts with the sample transition (X, C, A, R 8', C", A') and continues with (Xt, St Ct, At, Re) in which Au = A' = 1 (X', S C") and A, ~ Th-2,1 (: X2, St CJ) t 2 1. Since S' = + CR and C" yC, we can write*
E Exsca Ve. S'' A')
E Eyseaa
E T1 (5' (S' + CC (X,,',,",
Euaca
E
h1
S' + Σ
Xo== = x = S" = Ao = =A |
t=0
Ezaca
E
h1
8 s cR + 7
Σ
1 S0 C² AA- A'
1=0
k+1
Egaca
E
h1
s
CR
7" YRR
X 41 =
k+1
= E
h1
8 s c
Σ
R'R:
Xg = I, = s Ao =
t=
-V,,1,(1,,8,,,,)
Lemma B.2. For each (,,), X x S C x A and return-distribution Th, defined by Equation 1 the associated Va,la,,, 8, (, a) indicates the value of the optimal policy for partial return, i.e.:
(2,
max RI RIEMM
hy
8
Re
(22)
Proof. We establish the validity of this lemma through induction on The statement holds true for with Go,(z, s, a) = O. Assuming the statement is true for ViL we leverage the results of Lemma B.1 and the fact that the policy Gi (n,,) selects the action maximizing Vk, conclude the validity of the statement for VK+1,1-
Lemma B.3, For each (I,,,,,,)) a) E X xx x A, it holds that
(*i - & V,,((,,, c., V (ii
(23)
where lim Ex 8, C, a) = 0. It also holds that Vi, v (z,,,,,,
Proof. Let (Re)220 be sequence of rewards in [RMIN, RMAX! for any policy n. Since hy is non-decreasing function, we have
x0
hi + En S hi s + Σ 7',,
(24)
1=
t=0
Also, we can use the ((0))Lipschir) property of h1, i.c ho(u1 uz) - -h(uu)= I (00)uz, to write
2
hi s +
72 Re
h4 + > Be Re s <00)c
Σ 1=k+1
/Re S (00) ++ G
(25)
Combining these results yields the following inequality
V (z, 8, a) Ez 8, c, a) < V/ (2, < (z
(26)
"Note that C. is degenerate random variable that only takes the value 7 therefore multiplying random variable Z by C. scales each realization by 7
13

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
where (x, 8, C, a) (0))er++G This shows that limgg-ee (, a) 0. Taking the supremum over all policies and applying Lemma B.2 results in Inequality 23. By setting Go, 8, c, a) 0 and considering the non- negativity of rewards, we ensure that (e, 8, C, a) is increasing with respect to k and therefore we have VI, 8, C, a) v s,,,,,,a)
Theorem B.4. With T Gi (n..),, it holds that limg V Vmi
Proof. Given that the function hy is non-decreasing and considering the definitions of (2, and V 8, 8, c,a),
we can write:
0 S (1) [he (s + CGR, hy
(27)
clE (G,, 8, a) GR,,,
and
0 0<)(1 E [h (s + cQ s, a)) (s (G, cE GT 8, C, CR,( 8, G, a,a)]
(28)
Since (,, is increasing w.r. k the numerator in Equation 27 is positive and we can conclude that E [ s,e, a)] 2 E E(G,-,,(E,,,,,,) a)]. Utilizing Equation 11, we can also infer that
Bysoa [E (G.,I S', a (X S", C'))]] 2 Exzso [G- 1. (X', S aa (X', S (X))]
(29)
Now in Equation 28, in order to show that V (z. 8, C, a) - 8,,(2,,,,,, 2 0 in every state-action, we need to show that the denominator in this equation is also always positive. With &, EI GR,(z,, 8, c, a)], we
have
Ek (z, s, c,
E E (z, S, S,,
Execa
[ [ IR* S', - ( ("','' " (X))
Gi-I, [E [G (X', "'','' an - (X,
2 7 Busca
E [ (X'S', ,',''',,,,, and (X','''C) C')) - Gx, Cr au (X ,',""'"))))
> E Exsca (cx (X', S' C² ant (X' C')),,
(30)
where we use Equation 29 for (a). Given that c (z. s, c, a) is bounded from below, its infimum G == inf (2.s...)) C* (z, s; c, exists, so we can take infimum from both sides of Equation 30 and replace ex (Ir 8, c,a) with eg. This leads
E 2 7ck =
E
0,
(31)
demonstrating that both denominator and numerator in Equation 28 are positive. Therefore, we can prove the theorem using the Squeeze Theorem and Lemma B.3.
C. Lower Bound for the Objective
In Appendix B, we showed that the fixed point of each distributional Bellman operator T denoted by of and instantiated
as G can be found and we were able to provide the error bound for each Tk. Using the fact that fo (u()()))du 0 for l N, the update rule for function h in Equation 12 shows
E [heii (ari (Xo,0,1,,0 M141 (Xa, 1))] E (n (aTl O, 0,1,87 (Xo,,XX,,, (Xo, 0, 1))]]
where (z (z. 8, c) = a (x, c) denotes the optimal action when the same function h; is used to estimate G and calculate E [.). Remember that the returl-vrribll of the optimal policy derived with the fixed function by and h441 is denoted by G and G Therefore, we have
E h
(GFLL O, 1, 1,441 (Xo, 0, 1))}] E
0,1, M4t (Xo.0. 0, 11)] 2 E hi (G ( O, 1. (Xo, 0, 1))
hhe (a (Xa,
E hi41 G (Xo. 0, 1, 1,4 (Xo. 0, 1))]
14

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
and relative our objective, we can write:
S & (G*1)
sup E h ( (Xo, 0, (Xo, 0, 1))]
LEH
E
hu G (Xo, 0, 0X,,0, (X,, 0.1))
Since the rewards are bounded, both G and function h are bounded. Therefore, the monotonic increase of J(nj, hi) V (Xo, O, a7 (Xo,0, 1)) as oo provides lower bound for the objective.
D. Details of Algorithm 2
The quantile regression loss function used in this algorithm helps estimate the quantiles by penalizing both overestimation and underestimation with weights т and 1 - T, respectively. The quantile Huber loss function (Huber, 1992) uses the squared regression loss in an interval [-x, x|] to prevent the gradient from becoming constant when u + 0+:
pF(u)= = |T - - (u(
(32)
where the Huber loss (u) is given by
fu²,
if [u] S 6 otherwise
(u) { x (lu1 -
(33)
Furthermore, since function hi is approximated with the quantile representation of G ;= G and Equation 6, we need to show how E [hx(6% + c40 (2kk s dg a')) is calculated. With 2j :- 5/e + ch0 S/ C,. a') and 02 = Fa (ft). we can write
h((2)) = Ju ( (a) - (3, FF)((a)) u(da)
Σ
Pa ! (3) -aa((a) m(de))
Σ ( Fo (a)a(da))) f" (3)-1(((a)) u(da)
Ti 1 0 p(da) + (z)- 1 J au(daa)
(34)
In this calculation, the integration interval [0, 1] is divided into N intervals [To, T1), [T, T2), ......,iTss [TN-2, TN- [IN-1,77 Therefore, the integrals J",_, ((dex) is calculated on [T,-1, T,), including the lower limit 7:-1 and excluding the upper limit
Ti. In (a), we used the fact that 0, is constant in [T--1, T.). Also, the first term in the summation can be omitttd since it is constant for all actions.
In this algorithm, it's also important to highlight the direct relationship between the number of quantiles and the expressive- ness of SRM. For example, when the return distribution is approximated with N quantiles, the expectation can be estimated with p.N = -' ud(da = and را = 0 for 1 < j < N. Similarly, CVaR for a < 1 can be approximated by setting ūj = 1/a for [aN] + 1 and رة 0 otherwise.
E. Convergence of Algorithm 2
As described in Section 3.3, we parameterize the return distribution using a quantile representation. Specifically, we employ quantile projection operator, llq, to map any return distribution 17 onto its quantile representation with respect to the
1- Wasserstein distance (w1). Therefore, IQ = f = + Σm do with 02 = F (ft). f (T--1 + + S i S N
corresponds to the solution of the following minimization problem:
minimize w1 n/) subject to n' E &Q.N
15

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
where F Q.N is the space of quantile representations with N quantiles. Using this definition, Algorithm 2 can be expressed as iteratively updating
1, 1lQ T
As previously noted, this process is analogous to the iteration in the QR-DQN algorithm, with two key differences: the incorporation of risk-sensitive greedy action selection and the use of an extended state space. Consequently, we can leverage the steps outlined in Bellemare et al. (2023, Section 7.3) to establish the convergence of IT
To begin, we will demonstrate that 7 is a contraction mapping. That the sequence of iterates defined by T T Ts, converges s7t with respect the supremum p- Wasserstein distance, wp. for p [1, oo]. Here, we assume the existence of unique optimal policy xÿ. With this assumption, we leverage the fact that the action gap, GAP(Q) defined as the smallest difference between the higled and second-highest-vvlueed actions across all states for given Q-function is strictly positive. By setting E = GAP(V*)) )/2 and using Lemma B.3, we can see that after K₂ E N iterations where Kg == [in( projaMx ( )/, the greedy action in state (z, 8, c) becomes the optimal action a", and for any a + a", we
have:
V,,'', v (z,,,, -
2 (*i + - - V,,(I,,,c,,)+ + GAAPP. -22 V,,(r,
Thus, after Kg iterations, the policy induced by the return distribution becomes the optimal policy. Beyond this point, the distributional optimality operator transitions to the distributional Bellman operator for the optimal policy, which is known y-contraction with respect to wy. Using this result, we conclude that the combined operator ITg T is contraction with respect Woo as established in Dabney et al. (2018b, Proposition 2).
F. Additional discussion on Time and Dynamic Consistency
In this section, we need to introduce new notations to discuss the flow of information. Suppose we have a sequence of real-valued random variable spaces denoted as Zo C .... C Zr, Z, == Lp Ft. Here, Ze : Ω2 - R represents an element of the space Z,.
Moreover, let us define the preference system {PeTT TT-1 as the family of preference mappings Pt, ZT + Z,, O,...,T T - 1. The conditional expectation denoted by E Fi] is an example of such mappings. With these notations, optimization problem defined in Equation I can be written with p po, and 2T 4ū, where ZT Z7 denotes the cumulative reward starting from time t. The following definitions help us discuss the connection between the risk measure p and the policy π:
Definition F.1 (Time-consistency). An optimal policy T² = (aj...., ....) is time-consistent if for any t == 1,..., T, the shifted policy 7x = (ai,..., 27,) is optimal for
max Pt. (Ztrr).
(35)
N
Definition F22 (Dynamic-consisteney)) The preference system (PTT) T-11 1 is said to exhibit dynamic consistency if the following implication holds for all 0 S t
< t2 TT-1
PI, (Z) Z Pt..T (Z')= Pt. T (2) = (Z) Z,Z' E 2r, 05 < -
(36)
Additionally, this preference system is said to exhibit strict dynamic consistency if the following implication holds:
(Z) Pta.T (Z') Pt., (Z) Pri. (Z') Z, E 2r. <t TT-1.
(37)
Note that dynamic-consistency is property of a preference system and time-consistency is property of a policy w.r. preference system. Although some authors (e.g. Ruszczyúski (2010)) have used the term "time-consistency" for preference
Forr cases with multiple optimal policies in the risk-neutral setting, refer Bellemare et al. (2023, Section 7.5). Extending this result the risk-sensitive case straightforward
16

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
systems, in this context, we maintain the distinction between these two terms. The primary rationale behind this distinction is that using& dynamically consistent preference system implies a time-consistent policy only when the optimal policy is unique. In scenarios with multiple optimal policies, additional conditions must be satisfied (Shapiro et al., 2014), Nonetheless, employing these preference systems has been widely adopted approach in the RSRL literature to ensure the time-consistency of the optimal policy". To understand the necessary properties of a dynamically consistent preference system, we require additional definitions:
T-1 is said to be recursive if Definition F.3 (Recursivity). The preference system {pr.r t=
Pt. (Pt,,r (Z)) = (Z) Z Zr,05 11 < IT
(38)
For instance, Kupper& & Schachermayer (2009) show that the only law invariant convex risk measure that has the recursion property P (p(- ()) = p(.) for G C F F is the Entropic risk measure:
z( 1 log Blexp(12)) [0,
(39)
Therefore, using the above-mentioned p yields a recursive preference system. The Entropic risk measure is monotone, translation-ivvariant, and convex. However, it does not have the positive homogeneity property, so not suitable for applicationns in which this property is essential. Nevertheless, the risk measures p(-) = E(-) and p(+) = ess are the boundary cases of the Entropic risk measure with ? and 7 oc, have the positive homogeneity property and therefore are coherent risk measures.
ess sup(.), which
Definition F.4 (Decomposability) The preference mappings Pt.r are considered to be decomposable via family of one-step mappings Pa: Ze+1 - Z, if they can be expressed as compositions
Pr. (Z) = Pe (Pe+1 (.. (r-Pr-(())) 2 2T.
(40)
It is easily seen that the preference mappings of a recursive preference system, such as the one with the Entropic risk measure, are also decomposable. The inverse, however, is not always true and a set of decomposable preference mappings constitute a recursive preference system only if their corresponding one-step mappings are translation-invarinn and pr (c) = 0 for t .,...,, T. For instance, a convex conditional risk measure such as Pt C.aR, FF)) can be used as the one-step mapping and establish a decomposable and recursive preference system?. At last, in both of these cases, whether the preference mappings of a recursive preference system or the one-step mappings of decomposable preference mapping are monotone, the preference system is dynamically consistent.
As mentioned before, the dynamic consistency of the preference system only implies the time-consistency of a unique optimal policy. To guarantee the time-consistency of all optimal policies, Shapiro & Ugurlu (2016) shows that the preference system has to be strictly dynamically consistent. This requires the preference mappings of a recursive preference system or the one-step mappings of decomposable preference mapping to be strictly monotone, i.e, the following implication must hold:
2 Z' Z - Pe,((Z) PC, 2 Z' Zr.
The Spectral risk measure is an example of strictly monotone preference mappings only if the risk spectrum ((u) is positive on the interval (0, 1). Consequently, CVaRR is not strictly monotone for (0, 1) and cannot deduce that, for the preference system characterized by nested Conditional Value-at-Risk, every optimal solution of the corresponding reference problem is time-consistent. An easy way to ensure that ((u) = f p(da)= for (0, 1) is check whether ((da)) (am1 is non-zero or not. In other words, if the risk measure assigns non-zero weight on the expectation (CVaR,), the resulting SRM is strictly monotone.
The decomposition theorem shows that the preference mappings Pr, can also be provided for SRM. It also shows that these preference mappings are strictly monotone if the initial risk measure is a strictly monotone SRM. This property is evident since for a = 1, ed and consequently of is also 1. Therefore, the weight of CVaR, in the preference mappings would
'In these works, a set of dynamic programming equations are defined and the optimal policies serve as a solution to these equations, which ensure the time-consistency. Additional details on this topic can be explored in the work of Shapiro & Pichler (2016). These risk measures are also called Nested Risk Mcasures in the literature
17

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
also be non-zero. Intuitively, if the risk measure takes into account the entire distribution calculate the risk-adjusted value, Le. has non-zero weight for the expected value, the resulting preference mappings also have this property.
Additionally, the goal of analyzing the evolution of risk preferences over time can be achieved with the preference mappings, without the need for deriving the one-step mappings. The intermediate random variable εt,e-1 in the one-step mappings shows how the risk preference at time f changes compared to the previous risk preference at time t - 1, however, 58 in the decomposition theorem shows how the risk preference at time / changes compared to the initial risk preference. For example, the CVaR risk level at time t, Cq can be written as both a;- 1 or att. Similarly, in the CVaR case, the risk parameter B can be written as both (B, R or (Bo- St)/Cr.
G. Proof of Theorem 5.1
For general distributions, we have (=) = 1/a for < Aa and <=(z) = 0 for > Aa- To discuss the value of {z(2) when 2 = Aa, let us consider two cases based on the continuity of Fo (2) at & = Aa. i., whether pa(Na) = 0 or Pc()a) > 0.
For the first case, we simply have
1//a
if = S Xa
(z)
(41)
0
i > Xa
Since G' is convex combination 8, + c,G,, we know that pc (Na) implies pc, ((Aa - st)/a)) = Therefore, we
have:
c = E [5a
Fi
E (1stt
a
Fa. Aa )1
a
Ct
da - 81
al = Fa.(
(42)
For the second case where pa()a) > 0, we use the fact that E [Sa] = to write ("(Aa) as a function of Fa (Aa) and Pa(Aa):
1/a
if < Xa
(x) = (1 i (Fa (Aa) - po(Na)))/PG(Aa)
if = Xa
(43)
if Aa
Note that the set (z < } can be empty, especially for small a. In this case, 5d (Aa) = 1/po(An) would be the only go (z). Using this information, we can calculate tt::
non-zero
Esaa Fil
1
1 1 (Fo (Na) po(aa)))
E.
+
E. [1(0+tt)G==aa))
PG Paa)
1
(Fa.
Aa 8t
PG.
81 -
Aa
+
1- 1 (Fa (Ma) - DG po()a))
PCG Na A
a
4
Ct
po()a)
C4
- Fai
0
1
Aa
- 81
)
=
PG.
da -5).
1 1- a - (Fo(da) - pa (aa))
Po(Da)
1
da -81 Pc.
da -t) Fo(Aa)-o -a
Fai
a
Po()a)
da -8) - Pc..
da At). Fo(da) -
- at = Fo.
(44)
Ct
Pa()a)
Notice that this can also be simplified to
da 88 agt Fo.
(45)
44
18

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
if FG, (z) does not have discontinuity at Au In the DRL framework, only estimates of the return return-distributions are available. When estimating the distribution with the Quantile representation, it's easy to see that A, 8; for T;-1 S < so Pa()..) = 1/N. If (>,, st)/c, c, is equal to any of the estimated i,,5, we have Po. ((>a - st)/c) 1/N and af can be estimated with
64
Aa -
agi Fc.
(Fa(Aa) -
Otherwise, we have pc, ((>a 8x)/ce) = 0 and af be estimated with Equation 45.
H. Examples for Calculating the Intermediate Risk Preferences
Example 2. Suppose that the quantiles G and G,, denoted by 8 and 8,, are given as in the table below. Also suppose that
8t 5, C = 0.8, and we are interested the following risk measure at the initial state:
GG)) = 0.6 2V + 0.4 CVaRo..sG))
Table 4: The quantiles of G and G., and the dual variables 50.25 and 5 to calculate p(G)
0
50.25
ç0.8
0,
5%
1.25
5
15%
9
4
1.25
6
25%
12
2
1.25
8
35%
20
0
1.25
14
45%
21
0
1.25
15
55%
27
0
1.25
17
65%
30
1.25
21
75%
32
1.25
25
85%
39
28
95%
46
35
For 0.25, CVaR., and da are 8.8 and 12. For 0.8, these values are 19.75 and 39. Now use Equation 45 to calculate the off values.
FG. 12 - 5,
0.25g0.25
FG, (8.75) 0.3,
0.8
39 -
0.8 Fo.
FG, FG, (42.5) 1.0.
0.8
With 20 0.3/0.25 1.2 and co 1.0/0.8 1.25, we can see that time i, the risk measure changes to
0e (C.) 0.59 CVaRo.3 (G.) + 0.41 CVaR1. (G.)).
where g = 0.6 1.2 0.4 1.25 1.22.
Example 3. In this example, we provide an intuition behind the computation of intermediate risk preferences in the Mean-reversion Trading environment. start, let's examine a sample trajectory within our environment, where we employ our model optimized for CVaRo.5. The details of this trajectory can be found in Table 5. In Figure 6(a), we also present the return distribution G(æ,, St, Ct2 a4) each time step t. These distributions characterize the agent's future reward when it begins in the state-action pair (ær, St, Ct, (e) at time / and follows the optimal policy.
In our model, the risk preference of the agent chosen at time 0, which is associated with function h, remains constant throughout the trajectory. For instance, in our specific scenario with Ao.5 = 0.874, the agent's action selection is based on the average return below this value, corresponding to the 0.5-quantile of the return distribution at the initial state. order to apply this risk preference in all subsequent states, we need to align the return distribution in those states with the agent's
19

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
40
0.8
30
0.6
10
0.2
0
0
0.2
0.4
0.6
0.8
10
20
30
40
50
(a) The Quantile function
(b) The CDF
Figure 5: The Quantile function and the CDF of G (black) and G. (blue) in Example 2
Table 5: The states and actions of single trajectory algorithmic trading environment
0
2
3
7
8
9
10
P.
1.000
0.606
0.768
1.053
0.796
0.934
0.569
0.636
0.238
0.698
0.870
4t
0.000
-0.400
1.600
1.800
-0.200
0.200
0.600
1.000
0,400
0.200
0.000
0.399
-0.820
-0.971
1.053
0.746
0.390
0.175
0.529
0.440
0.962
Cy
1.000
0.990
0.980
0.970
0.961
0.951
0.941
0.932
0.923
0.914
0.904
-0.400
2.000
0.200
-2.000
0.400
0,400
0.400
-0.600
0.400
-0.600
0.000
F1
0.399
-1,232
-0.154
2.086
-0.319
-0.374
-0.228
0.380
-0.096
0.571
0.000
0.500
0.168
0.355
0.083
0.089
0.147
0.544
0.702
0.819
0.084
0.000
perspective at the initial time. This alignment is achieved by scaling the return distribution by and adding st, as illustrated in Figure 6(b).
Now we can see that the value Ao.5 0.874 corresponds to a different quantile of the return distribution in subsequent states. For instance, action selection w.r.t the 0.5-quantile the return distribution time 0 shifts 0.168-quantile the return distribution time 1. This mechanism enables to observe how the agent's risk preference evolves over time. Here, demonstrated the process for single but more complicated SRMs follow similar steps. The only additional step would be the calculation of the weight of each component of the risk measure, similar to the example in Appendix 2.
1,0
[+0
1:1
112
02
0.8
04
Ind
mg
0.0
1.0
200
30
0.2
0.00
0.28
0.50
0.75
1.00
1/3
150
1.75
Discounted Future Rewards
Discounted Future Rewards
(a)
(b)
Figure 6: Figure 6(a) and 6(b) illustrate the CDF of G(v,, 8t, Ct, a.) and 8t + CaG((xs 88, Ct, ar) for each state-action pair in a trajectory.
20

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
I. Experiments with number of Quantiles
Due to the approximation of the probability measure 14, an important question arises: Can our model find a policy that maximizes the expected return, the primary objective of the risk-neutral QR-DQN algorithm? To address this question, we conducted a comparison of the expected return produced by our model under varying quantile numbers (N), with jiy to in the meea-reeerssion trading example. The results of this experiment, presented in Figure 7(a), demonstrate that the number of quantiles increases, our model not only matches the performance of the risk-neutral algorithm but surpasses it, yielding superior expected returns. Furthermore, in Figure 7(b), we observe that the improvement extends beyond expected returns. The policy derived from our algorithm consistently attains higher CVaBa values for all E (0, 1].
0.8
22-03NN--20)
1.5-
GR 2RM2N-200
0.7
GR 1200
0.0
CR-SRMNN-73
08-
3R-SRMN--5)
-15
SRMNN-30)
08-
CB-SRMNN-10)
Dendit
04
02-
-0.0
C1--200/N-0000
CR-0DNNN-200
0 Dacouried Futue Rewards
0.0 0.
02
03
0.4
0.5
0.6
07
0.0
0.9
10
8
(a)
(b)
Figure 7: Figure 7(a) displays the distribution of Cumulative Discounted Rewards for policies with different number of quantiles. The solid lines this figure represent E [G]. Figure 7(b) compares the performance QR-SRM(a=1) against QR-DQN, both with 200 quantiles, w.r. CVaRa for all E (0, 1].
However, this enhanced performance comes at cost. When normalizing all models' scores with the QR-DQN(NV550) score, as depicted in Figure 8(a), all models reach an equivalent performance level within the same number of steps Yet, this figure can be misleading since the time of action selection at each step increases quadratically with the number of quantiles. Specifically, for an action space size denoted A, the QR-DQN model's action selection requires O(AN) operations, in contrast our model, which requires ((AN2) operations. Figure 8(b) presents the score plotted against the training time normalized to the training time of the QR-DQN(N=50), revealing that transitioning from 50 quantiles to 200 has a less pronounced impact on the QR-DQN model compared to our model.
200+
-00
-00%
25
10.
100 207h 20th 30N
20
10
(a)
(b)
(c)
Figure 8: Figure 8(a) and 8(b) displays the moving expected reward of our model and the QR-DQN model with different numbers of quantiles, plotted against the number of steps and time. Figure 8(c) shows the mean absolute error between consecutive estimations of the return distribution G(zo, 0, a6)-
In our algorithm, the estimation of function is updated periodically. This estimation directly linked to the estimation of
Fach step corresponds a single interaction the agent with the environment.
21

---

Beyond CVaR:
Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
G( 0, 1, a)), making the convergence of this return distribution a useful indicator for the convergence of the function h. Figure 8(c) visualizes this convergence, presenting the mean absolute error between consecutive estimations of the return distribution.
J. Details of the Environments
J.1. American Put Option Trading
In this environment, we assume that the price of the underlying asset follows a Geometric Brownian Motion, characterized by the differential equation d.:- = (P,dt a P,dW,, where Ç = -0.3 is the drift, 0.3 the volatility, the initial price is Po and W, is standard Brownian motion. The strike price of the put option is assumed be K 1. At each time step, the agent can either exercise the option and receive T max (0, K P.} hold the option to receive reward at future steps. At maturity, if the option hasn't been exercised yet, the agent automatically receives PT max K Pr).
J.2. Mean-reversion Trading Strategy
In this algorithmic trading framework, the asset price follows Omsttein MrieeinnUUheebeekk process, characterized by the differential equation d'';- x(ς- Pt)dt + odW, where & 1 is the long-term mean level, 2 determines the speed of reversion to mean. At each time step t = 0,... . 1, the agent takes an action E (- [-0aax- Gmax), corresponding to trading quantities of the asset and changes its inventory ft E (-qmaxx (max).. ). The reward is defined as = -a Pr - (p(x)² for S f T' 2 and rT-1 = -0 -1 ((ar-1)² + Pr 2 for the final time step. Here, &p = 0,005 represents the transaction cost and & 0.5 signifies the terminal penalty. our setup, the agent faces penalties for holding any assets the final time step Consequently, the reward at time step
has an additional term for the agent's 's inventory at time
step T. In our example, we consider T 10, Qmax
5, Ornax = 2, 7 0.99, and discretize the action space into 21 actions.
J.3. Windy Lunar Lander
The Lunar Lander environment is classic rocket trajectory optimization problem, involving an 8-dimensional state space and four actions: firing the left right orientation engines, firing the main engine, doing nothing. To introduce stochasticity, enable the wind option. The objective is to guide the lander from the top of the screen to the landing pad. Successful landings yield around 100-140 points. If the lander moves away from the pad, it loses points, while crash results in an additional penalty of -100 points. Landing safely adds a bonus of + +100 points, and each leg that makes contact with the ground carns +10 points. Firing the main engine incurs a penalty of -0.3 points per frame, while firing the side engines costs -0.03 points per frame.
K. Implementation details
This section provides an overview of the implementation details of our model. We adopt the single-file implementation of RL algorithms from CleanRL (Huang et al., 2022) for clarity. In this approach, the model and its training are encapsulated within a single file. The code for the project is available at https: // hub.. com htte/Mehraaaae dadMoghimi /QRSRM.
Table 6: Default hyperparameters in different models
Hyperparameter
Value
Learning Rate
2.5c-4
Discount Factor (Y)
0.99
Batch Size
256
Number of Quantiles
50
The repository contains four Python files for each algorithm discussed in section 6 and Appendix I. The qrsrm. py file defines the state-action value function with a feed-forward network that takes (X, C') as input and outputs a N x A dimensional vector representing the quantile function of all actions. This neural network comprises three hidden layers, each with 128 neurons. The value function is similar across other files, with the only difference being their input value, which can be (X) in qrdon. py and qr evar. .py or (X, B) in grovar.
22

---

Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional RL
In grevar py and gricvar .eaarpp Py, the variable pha determines the risk preference of the agent. In qrarm. the user choose between different risk measures with the risk-measur variable. The value of CVaR, WSCVaR, Dual, and Exp for this variable is associated with the CVaR, Weighted sum of CVaRs, Dual Power and Exponential risk measures.
The number of timesteps to train each algorithm is determined by total-timesteps variable. A fraction of these timesteps is allocated to é-greedy exploration and the rest allocated to learning the value function accurately. Also, in qreem. py and grevar py, the estimation of function & and target value needs frequent updating. The value of variables h frequency and frequency in these files determine the update frequency for these estimations. Lastly, techniques such as Replay Buffers and Target Networks are employed to stabilize the training process for all of the algorithms.
The custom environments used in our experiments are available in the custom envs Py file. We implemented the American Option Trading and Mean-reversion Trading environments using the Gymnasium (formerly OpenAI Gym) package (Towers et al., 2023). This package allows for the definition of the state space, action space, and environment dynamics with simple functions. The primary function is the step function, which takes an action input and outputs the reward and the next state based on the current state.
The state space augmentation for QR-SRM and QR-CVaR models is also defined using two environment wrappers. These wrappers automatically store the target value B or the accumulated discounted reward S and the discount factor C' for trajectory. The key advantage of these wrappers is their compatibility with any environment available in the Gymnasium package.
23


Reviews:
[{"review_id": "TaMP0o31vG", "reviewer": "Reviewer_ymbN", "summary": "This work focuses on the risk-sensitive RL, i.e., maximizing the return and managing worst-case scenarios. As distribution RL considers the return distribution, there are several works applying it to the risk-sensitive RL. Extending the widely used risk metric CVaR to Spectral Risk Measures (SRM), this work proposes a novel safe reinforcement learning method with convergence guarantee. Experiments show that the proposed method outperforms existing risk-neutral and risk-sensitive DRL models in various settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and for recognizing the strengths of our work. We are glad that you found our proposed method novel. We address your questions below.\n\n**Missing References:**\n\nThank you for highlighting the missing references. The first reference is already discussed in the paper, but we will ensure the remaining references are added in the updated version.\n\n**Strengths and Weaknesses:**\n\nThank you for recognizing the strengths of our work. We would like to emphasize that extending static CVaR to the more general static SRM is just one of our contributions. Another significant contribution is the interpretation of the learned policy through the decomposition of static coherent risk measures and the distribution of returns in our algorithm, as discussed in Section 5. Unlike existing works in the risk-sensitive RL literature, our approach introduces a decomposition of risk preference that enables tracking the agent's intermediate, state-dependent risk preferences. This added interpretability is a unique feature of our method, allowing for continuous monitoring of the policy's behavior. If the policy ever diverges from the user's preferences, a new policy can be trained to realign with those preferences.\n\n**Q1:**\n\nWhile different methods in Table 2 excel on different indicators, our proposed algorithm (QR-SRM) consistently ranks as the top-performing or within one standard deviation of the top-performing algorithms across all objectives. These minor variations can be attributed to factors such as the inherent stochasticity of the environment and the use of function approximation for value functions. We will highlight the performance of our model more clearly in the revised manuscript to further demonstrate its strong performance.\n\n**Q2:**\n\nAs mentioned in the conclusion section, our value-based method is designed for environments with discrete action spaces. Extending our algorithm to actor-critic methods would enable its application in environments with continuous action spaces. In fact, addressing the continuous control problem is the focus of our upcoming work.\n\n**Q3:**\n\nThank you for bringing this to our attention. We will address this issue in the revised draft.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "roughly", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "This work is established on previous risk-sensitive RL methods based on CVaR and introduces Spectral Risk Measures (SRM) as a more general metric.", "essential_references_not_discussed": "As this work mainly focuses on risk-sensitive RL, several works closely related to this work are missed, like CVaR [1-4], EVaR [5], and so on.\n\nReference:\n\n[1] Risk-sensitive and robust decision-making: a cvar optimization approach\n\n[2] Towards safe reinforcement learning via constraining conditional value-at-risk\n\n[3] CVaR-constrained policy optimization for safe reinforcement learning\n\n[4] Risk-sensitive reward-free reinforcement learning with cvar\n\n[5] Risk-sensitive reinforcement learning via Entropic-VaR optimization", "other_strengths_and_weaknesses": "Strength:\n\n- Extending CVaR to more general metric Spectral Risk Measures (SRM) is novel in risk-sensitive RL\n\n- The proposed method is novel with solid proof.\n\nWeakness:\n\n- It seems that different methods in Table 2 perform better on different indicators. Is there any insights for this phenomenon?\n\n- The experimental scenario is relatively simple. Can the proposed method be applied to more complex tasks, such as continuous robot control?\n\n- Minors: The spacing between rows in some places looks very small and needs to be adjusted", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See weaknesses above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f3MOs2y7tV", "reviewer": "Reviewer_pk66", "summary": "This paper proposes a distributional RL algorithm for optimizing the Spectral Risk Measures (SRM, with CVar as a special case) of return in discounted MDPs. \nSpecifically, this paper makes use of a variational representation of SRM proposed by Kusuoka, thus transforming the optimization of SRM into a two-layer optimization problem. The inner-layer optimization problem is almost the same as the problem encountered when optimizing CVaR. It is a problem of solving the optimal risk-neutral strategy for an MDP with augmented states using the current cumulative reward (stock) and discount rate (while also requiring the estimation of the return distribution). As for the outer-layer optimization problem (i.e., the variational representation), it has an explicit solution expressed in terms of the quantiles of the return distribution.\nThe paper proves the theoretical convergence of the algorithm (assuming that each step can be solved precisely) and characterizes the temporal decomposition of SRMs for the interpretation of the learned policy.\nThis paper verifies through extensive experiments that this algorithm can optimize the SRM and outperform the previous risk-neutral and risk-sensitive algorithms.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and for recognizing the strengths of our work. We are glad that you found our paper well-written. We address your questions below.\n\n**Q1:**\n\nThank you for pointing this out. Theorem 2 of [1] and Theorem 3.1 of [2] provide a detailed discussion on the equivalence of these two supremums. We will clarify this in the revised manuscript.  \n\n**Q2:** \n\nWe appreciate the reviewer’s careful observation. Indeed, the convergence proof of the inner optimization (Line 649) assumes that $h_l$ is $\\phi(0)$-Lipschitz. While Algorithm 1 states that $h_0​$ is randomly initialized, we clarify that the required Lipschitz property holds from the beginning.\n\nSpecifically, in Algorithm 1, $h_0$ takes the form of Eqn. (6), which utilizes the quantiles of a return distribution and the risk spectrum $\\phi$. As detailed in Appendix D, this equation is by construction $\\phi(0)$-Lipschitz. Therefore, using the quantiles of a random return distribution ensures that the Lipschitz condition is satisfied for $h_0$.\n\nWe acknowledge that this point was not clearly stated in the current version of the paper. In the revision, we will explicitly clarify this in Section 4 and in the algorithm description to avoid confusion.\n\n**Q3:**\n\nAs highlighted in our paper’s contributions, SRM provides valuable flexibility for practitioners. For example, Mean-CVaR is widely used in finance, particularly in portfolio management and insurance pricing. While CVaR, as a subclass of SRMs, offers some flexibility through the alpha parameter for adjusting policies, SRM provides even greater flexibility in defining objectives. For instance, WSCVaR allows practitioners to combine multiple CVaR objectives with arbitrary weights.\n\nA key advantage of SRM over CVaR is its adaptability in cases where the ideal risk-sensitive objective is not immediately clear. In certain environments, incorporating risk sensitivity leads to policies that improve both worst-case return and average return, demonstrating that optimizing for risk-sensitive performance does not always come at the cost of lower average returns. By treating the parameters of the risk-sensitive objective as hyperparameters, users can tune them and compare the resulting policies. This flexibility is especially useful in environments where reward models are arbitrarily designed and lack a clear real-world interpretation. For example, while a portfolio manager may have a well-defined objective in a trading environment, such clarity is often absent in environments like Lunar Lander. In these cases, tuning risk objectives as hyperparameters provides a practical solution.\n\nIn the experimental section, Table 2 shows that our algorithm with a CVaR objective matches the performance of QR-CVaR from [3], which benefits from stronger convergence guarantees. Furthermore, our algorithm successfully optimizes various risk measures, such as the dual power risk measure and Mean-CVaR, consistently identifying top-performing policies (or policies within one standard deviation of the best-performing ones).\n\nAnother major advantage of SRM over CVaR was observed in the Windy Lunar Lander experiment, where our algorithm demonstrated significantly more stable training. Finding CVaR-optimal policies is known to be challenging in risk-sensitive RL literature ([4]), and we observed this difficulty in our experiments. However, our algorithm, especially with a Mean-CVaR objective, exhibited much more stable training and resulted in the top-performing policy with respect to risk-sensitive metrics.\n\nFinally, as noted in our response to reviewer ymbN, this general risk measure was achieved without sacrificing the interpretability of the optimal policy, which was available for static CVaR-optimal policies. Our approach leverages the decomposition of static coherent risk and the distributional return distribution (Section 5), allowing the policy’s risk sensitivity to be continuously monitored when deployed in real-world settings.\n\nThe combination of SRM’s flexibility, the convergence guarantees established in our manuscript, and the interpretability tools presented in Section 5 makes SRM an excellent choice for risk-sensitive policy optimization.\n\n**Other:**\n\nWe appreciate the reviewer's feedback on the clarity of our notation. To address these concerns, we will revise the notation in the revised manuscript as follows:\n - We will use $\\boldsymbol{\\pi}$ instead of $\\boldsymbol{\\pi}_{\\mathrm{H}}$  to represent the general class of history-dependent policies.\n - We will use $\\mathcal{Y}$ to denote the interval $[G_{\\min}, G_{\\max}]$, consistent with the notation used in [1].\n\n**References:**\n\n[1] Bäuerle and Rieder. More Risk-Sensitive Markov Decision Processes \n\n[2] Bastani et al. Regret Bounds for Risk-Sensitive Reinforcement Learning \n\n[3] Bellemare et al. Distributional Reinforcement Learning\n\n[4] Greenberg et al. Efficient Risk-Averse Reinforcement Learning", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "This paper extends the optimizing the CVaR [1] of return of discounted MDPs with distributional RL to optimizing SRM. (Optimizing SRM in MDP was also considered in [2] without using distributional RL)\n\n[1] Bellemare, M. G., Dabney, W., and Rowland, M. Distributional Reinforcement Learning\n[2] Nicole Bauerle and Alexander Glauner. Minimizing spectral risk measures applied to Markov decision processes", "essential_references_not_discussed": "As far as I know, no key references are missed.", "other_strengths_and_weaknesses": "Strength:\n1. The generalization from CVaR to SRM is meaningful, and using distributional RL to obtain the explicit solution of the variational representation of SRM is a nice and natural idea.\n2. The paper is well-written and the technical parts are not hard to follow.\n3. The paper also has extensive experiments to verify the effectiveness of the proposed algorithm.\n\nWeakness:\n1. In the right part of line 117-132, the paper introduced non-stationary Markov policy $\\Pi_M\\subset\\Pi_H$ (with stock augmentation), but did not state that $\\sup_{\\pi\\in\\Pi_H}J(\\pi, h)=\\sup_{\\pi\\in\\Pi_M}J(\\pi, h)$, and in the following, the paper only focused on $\\sup_{\\pi\\in\\Pi_M}J(\\pi, h)$. I know that $\\sup_{\\pi\\in\\Pi_H}J(\\pi, h)=\\sup_{\\pi\\in\\Pi_M}J(\\pi, h)$ is absolutely correct, but it would be better to clarify it in the paper.\n2. Why in Algorithm 1 h_0 is randomly initialized, but in the proof of the convergence of inner problem (Line 649) $h_l$ is $\\phi(0)$-Lipschitz, which is not stated in any other places. As far as I know, for such distributional value iteration algorithm (inner optimization problem), the Lipschitz property of the function is necessary to obtain the $\\gamma^k$ convergence rate. I think it should be clarified in the paper by e.g. further explaining Eqn.(6).\n3. A minor point: This paper pays too much attention to the technical part. I think it would be better to clarify why  generalizing from CVaR to SRM is significant, such as providing and explaining simple and intuitive examples of SRM other than CVaR (line 330-337 present some examples of SRM without enough explanation).", "other_comments_or_suggestions": "Line 110 reward->return (or cumulative reward)\nLine 165 rho_{\\tilde{\\xi}} is used before definition.\n\nThe notations are misleading, for example, in Eqn (7), H in pi_H is history; but h\\in\\mathcal{H} is concave function, and pi_h is also used. It would be better to use different letters. And using \\mathcal{S} to denote [G_min, G_max] is also strange in RL literature.", "questions_for_authors": "I have no question", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o8cntEUD4m", "reviewer": "Reviewer_eG5t", "summary": "This paper studies how to optimize a single static Spectral Risk Measure (SRM) for an RL agent, leveraging a distributional RL framework. By extending the state to track discounted returns and employing quantile-based updates, the authors propose a two-tier algorithmic approach:\n\nInner Optimization: Fix an approximate function $h$ related to the SRM, then learn a greedy policy in the extended MDP.\nOuter Optimization: Update $h$ using closed form solution to better match the (new) return distribution of the initial state.\n\nThe authors highlight that using a single SRM generalizes beyond basic CVaR-based methods, potentially offering more flexibility in risk profiles. Experiments on algorithmic trading tasks and a Windy Lunar Lander environment suggest that their approach can produce policies that better align with static SRMs compared to baselines such as standard CVaR-based methods or purely risk-neutral approaches.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the strengths of our work. We address your questions below.\n\n**Q1:**\n\nWe thank the reviewer for this insightful comment. We agree that an explicit comparison with [1] would improve the clarity of our contribution. While both works share methodological components, such as the use of SRMs, a bilevel structure, and state augmentation, they address fundamentally different problems and employ distinct technical approaches.\n\n[1] focuses on risk-constrained RL, where SRMs are used to define constraints. Their method uses a primal-dual optimization approach, requiring gradient-based updates for both the policy and the dual variables associated with the risk constraint. In contrast, our paper addresses risk-sensitive RL, where the SRM is used as the objective function. Our method leverages the SRM’s supremum representation to derive a closed-form outer update (Eq. 6), avoiding a computationally expensive optimization step.\n\nPractically, this leads to three key distinctions:\n- **Computational Efficiency:** Our method eliminates costly outer-loop optimization over dual variables. Instead, the outer update step uses the learned return distribution in the closed-form solution.\n\n- **Theoretical Differences:** Unlike the actor-critic framework in [1], our approach is value-based, requiring a different convergence analysis.\n\n- **Interpretability:** We provide insight into the agent’s intermediate risk preferences through SRM decomposition, an aspect not addressed in [1].\n\nWe will include a dedicated subsection summarizing these differences and explicitly reference [1] in the revised manuscript.\n\n**Q2:**\n\nWe appreciate this thoughtful comment. Our convergence analysis focuses on asymptotic behavior, demonstrating that QR-SRM monotonically improves a lower bound on the SRM objective (Theorem 4.1). While our work emphasizes practicality and interpretability, other studies have explored the theoretical foundations of spectral risk measures. For instance, [2,3] provide regret and sample complexity bounds for SRM in distributional RL. Additionally, the approximation error for quantile TD algorithms, discussed in Proposition 19 of [4], is directly relevant to our inner optimization step, as we employ the same distributional Bellman operator in the augmented MDP.\n \n**Q3:**\n\nThank you for raising this important point. Our method indeed optimizes the policy with respect to a single, user-specified SRM, and the final policy at convergence is aligned with that specific risk preference. Thus, after convergence, the agent is strictly tied to that SRM. We do not claim that the learned policy generalizes across all risk criteria, and we agree that a policy optimized for one SRM (e.g., CVaR) may underperform under others (e.g., expected value).\n \nHowever, the mention of changing risk preferences over time (not during training) refers to a key interpretability contribution of our work: through the decomposition of SRM (Section 5), we are able to analyze the agent's intermediate, state-dependent risk preferences as the return distribution evolves. While the objective remains fixed, the conditional risk levels and weights (e.g., combinations of CVaRs) change across states and over time, providing insight into how the policy’s effective behavior adapts over time. This decomposition does not imply that the algorithm optimizes for multiple SRMs, but rather enhances interpretability within a single SRM objective.\n \nTo address multiple risk criteria or stakeholder objectives, we agree this is an interesting and practically relevant direction. One possible extension would involve optimizing for a mixture of SRMs, for example, via multi-objective RL. Some relevant works, such as [5], have explored these directions. Another direction could involve conditioning the policy on the risk parameter. While we do not explore these extensions in this work, we believe our framework provides a strong foundation for such adaptations, and we appreciate the reviewer highlighting this point.\n \n**Q4:**\n\nThank you for the suggestion. Our work focuses on developing a general RL algorithm based on SRMs, rather than a financial trading strategy. While metrics like mean-variance, Sharpe ratio, and maximum drawdown are important in finance, they are not directly applicable to our benchmark environments. Nonetheless, applying our method to financial data and evaluating it under these metrics is a promising direction for future work.\n \n**References:**\n\n[1] Kim et al. Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees\n\n[2] Bastani et al. Regret Bounds for Risk-Sensitive Reinforcement Learning \n\n[3] Chen et al. Provable Risk-Sensitive Distributional Reinforcement Learning  with General Function Approximation\n\n[4] Rowland et al. An Analysis of Quantile Temporal-Difference Learning\n\n[5] Moffaert et al. Risk-sensitivity through multi-objective reinforcement learning", "claims_and_evidence": "1. \n- Claim: The paper argued that it can directly optimize a spectral risk measure (broader than CVaR) through a distributional approach, thereby obtaining a single “best” policy according to that measure.\n- Evidence: Empirical demonstrations on trading tasks show that the learned distribution of returns indeed improves under the selected SRM.\n\n2. \n- Claim: The method achieves a “monotonic” or “non-decreasing lower bound” style improvement in the outer loop.\n- Evidence: The authors prove a type of asymptotic convergence result for the policy improvement step (inner optimization). However, they do not provide finite-step performance bounds or sample-complexity analysis.", "methods_and_evaluation_criteria": "Methods:\n- The extended MDP approach (tracking discounted returns in the state) is standard for risk-based RL.\n- The inner optimization uses quantile RL updates (in the style of QR-DQN).\n- The outer optimization updates the SRM parameter $h$ in closed form (from the Kusuoka representation).\n\nEvaluation:\nExperiments focus on the resulting return distributions of policy, both visually and via risk metrics (CVaR, exponential risk, etc.). The main shortcoming is that tasks are relatively toy-scale or moderately sized, and the baselines are fairly limited (primarily QR-DQN or variations of CVaR). For a final measure of practical performance, more rigorous tasks and additional metrics (especially in trading, e.g., Sharpe ratio, drawdown) would strengthen credibility.", "theoretical_claims": "1. Asymptotic Convergence of Inner Loop: The paper shows that for the inner loop (policy learning under a fixed $h$), the linearization of the sub-risk measure allows for a standard policy-improvement argument in distributional RL. This is reminiscent of known results (e.g., Kim et al. 2023, which obtains a similar linearization in a constrained RL setting). Furthermore, the paper only proves a monotonic lower bound in the tabular or exact distribution scenario. Under function approximation, or in finite-sample contexts, rigorous finite-time or non-asymptotic analyses are not provided.\n\n2. Monotonic Outer Updates: By iteratively re-estimating $h$ from the newly observed initial-state return distribution, the algorithm is said to climb toward an optimal measure. To the best of my knowledge, this part is novel in risk-aware RL with SRM.", "experimental_designs_or_analyses": "1. Tasks:\n- Two trading-like environments (American option, mean-reversion) demonstrate the algorithm’s ability to favor certain parts of the distribution (e.g., different quantile levels).\n- Windy Lunar Lander adds complexity but remains a somewhat “toy” environment compared to many large-scale or safety-critical domains.\n\n2. Comparisons:\n- Against risk-neutral QR-DQN, a fixed-CVaR version, and “iCVaR” from prior work. Results show each approach yields distinct return distributions, with the SRM-based approach often outdoing purely CVaR or risk-neutral baselines in the chosen metric.\n\n3. Comments:\n- The tasks could be considered simple or proof-of-concept. For a deep distributional RL method, it would be insightful to test more safety-critical or complex environments (e.g., from Safety Gymnasium: https://safety-gymnasium.readthedocs.io/en/latest/index.html).\n- In the trading tasks, standard finance metrics (mean-variance, Sharpe ratio, maximum drawdown) are absent, making it less clear how these policies compare to typical trading strategy benchmarks.", "relation_to_broader_scientific_literature": "- Spectral Risk Measures: The approach is consistent with the Kusuoka or supremum representation of coherent risk measures.\n- Distributional RL: The method leverages standard quantile approximation (QR-DQN) and is part of a growing body of risk-oriented distributional methods.", "essential_references_not_discussed": "1. Kim, Dohyeong, et al. \"Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees.\" Advances in neural information processing systems 2024.\n- While Kim et al. use SRM in a constraint setting (safe RL) and this paper uses SRM as a single-objective optimization, the overall methods (bilevel structure, state augmentation) are very similar. Yet the paper does not explicitly discuss or compare its approach to Kim et al.\n- Add a dedicated subsection contrasting this method with Kim et al.\n- Emphasize where the approaches are methodologically parallel (e.g., the idea of rewriting the SRM using supremum representation; the linearization of risk measure in the inner optimization).\n- Pinpoint novel aspects in this paper, such as the closed-form solution in Eq. (6) vs. the optimization-based approach for the supremum in Kim et al. (Eq. 5 in this work).\n\n2. Advanced Distributional RL: If your approach relies on or extends any of the standard distributional algorithms (C51, QR-DQN, IQN, etc.), referencing these helps readers see how your design decisions compare or could be integrated with more sophisticated distributional frameworks (e.g., FQF).\n\n2-1. Dabney, Will, et al. \"Implicit quantile networks for distributional reinforcement learning.\" International conference on machine learning. PMLR, 2018.\n- Extends quantile ideas via implicit quantile networks (IQN).\n\n2-2. Yang, Derek, et al. \"Fully parameterized quantile function for distributional reinforcement learning.\" Advances in neural information processing systems 32 (2019).\n- Improves upon quantile-based DRL by parameterizing quantiles themselves as learnable functions.", "other_strengths_and_weaknesses": "1. Strengths:\n- Demonstrates clear expansions beyond pure CVaR to a flexible set of SRMs.\n- Good interpretability argument for static risk measures, using a decomposition approach.\n- Shows how distributional RL can incorporate these measures with (arguably) minimal overhead.\n\n2. Weaknesses:\n- Limited comparison to Kim et al. despite using very similar core techniques.\n- Convergence analysis does not address finite-sample complexities.\n- Single-objective viewpoint: The final converged measure is still a single SRM, which might not serve multiple stakeholder or multi-risk contexts.\n- Experiments remain modest and do not thoroughly test the approach in safety-critical or large-scale RL scenarios.", "other_comments_or_suggestions": "- Multi-risk or multi-constraint extension: Address the possibility of combining multiple SRMs or constraints to handle different aspects of risk.\n- Deeper experimental demonstration (including standard trading metrics, more complicated tasks) would enhance confidence in the method’s real-world applicability.", "questions_for_authors": "1. Could you explicitly discuss how your approach (bilevel with a closed-form outer step) differs in practice from Kim et al.’s primal-dual approach, which also uses state augmentation and spectral risk?\n\n[Kim et al.] Kim, Dohyeong, et al. \"Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees.\" Advances in neural information processing systems 2024.\n\n2. Is there a straightforward way to bound approximation errors or obtain a sample-complexity result for the inner–outer loops?\n\n3. You mention that the risk measure $h$ can “change” mid-training (mentioned in Introduction), but eventually converges to a single SRM. Could you clarify whether, at convergence, the agent is strictly tied to one final risk measure? In other words, do you expect any meaningful “variety” in the agent’s risk preference by the end of training, or is it effectively just a single risk measure?\n\n3-1. If a policy is optimized for one particular SRM, it may perform poorly under different risk criteria (e.g., a strong CVaR policy but weak expected return). Have you considered how users might adapt your method if they wish to account for multiple forms of risk or multiple stakeholder objectives simultaneously?\n\n4.  How does this policy compare when measured by classical trading benchmarks such as mean–variance (Markowitz), the Sharpe ratio, or maximum drawdown? Could you include these metrics to show practical relevance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the paper presents a valuable contribution to risk-averse RL. The paper presents new insights into a complex problem and new general algorithms. The results in the paper are not surprising, and the paper builds on several recent papers that introduce and study similar ideas. In fact, the reviewers point out recent work that introduced similar results for spectral risk measures. The authors, however, convincingly argue that the contribution compared with this prior work is significant. In addition, the reviewers uncovered a few relatively minor technical issues. The author's response indicates that these problems are quite easy to address in the final version of the paper.
Decision: Accept (poster)