Return Capping: Sample Efficient CVaR Policy Gradient Optimisation
Authors: Harry Mead
Clarissa Costen
Bruno Lacerda
Nick Hawes
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Machine Learning
CVaR
Risk-Averse
Primary area: reinforcement_learning->deep_rl
TL;DR: We present a sample-efficient method for policy gradient CVaR optimisation by capping trajectory returns, rather than discarding trajectories.

Abstract: When optimising for conditional value at risk (CVaR) using policy gradients (PG), current methods rely on discarding a large proportion of trajectories, resulting in poor sample efficiency.  We propose a reformulation of the CVaR optimisation problem by capping the total return of trajectories used in training, rather than simply discarding them, and show that this is equivalent to the original problem if the cap is set appropriately. We show, with empirical results in an number of environments, that this reformulation of the problem results in consistently improved performance compared to baselines. We have made all our code available here: \url{https://github.com/HarryMJMead/cvar-return-capping}.

Return Capping: Sample-Efficien CVaR Policy Gradient Optimisation
Harry Mead Clarissa Costen
Bruno Lacerda' Nick Hawes
Abstract
some applications. it presents number of issues. Primarily, it results in very poor sample efficiency, especially when a is low, as a large proportion of sampled trajectories are discarded. Figure ((a) illustrates how existing CVaR PG methods are inefficient in terms of trajectory usage. In addi- tion, since is the highest return samples that are discarded, the policy is unable to learn from the best performing trajec- tories, which can present further issues for policy training. In order mitigate these issues, we propose Return Cap- ping. Rather than discarding trajectories, we propose that all trajectories are used in training. but the return of those above a certain threshold is truncated. We prove in Section 4 that if this threshold, or cap, is set correctly, optimising CVaR under Return Capping is equivalent to optimising CVaR without the cap. Since Return Capping allows all trajecto- ries to be used, sample efficiency is greatly improved, and learning performance improvements from high-performing trajectories is possible. Our main contributions are:
When optimising for conditional value at risk (CVaR) using policy gradients (PG), current meth- ods rely on discarding a large proportion of tra- jectories, resulting in poor sample efficiency. We propose reformulation of the CVaR optimisation problem by capping the total return of trajecto- ries used in training, rather than simply discard- ing them, and show that this is equivalent to the original problem if the cap is set appropriately. We show, with empirical results in an number of environments, that this reformulation of the prob- lem results in consistently improved performance compared to baselines. We have made all our code available here: https: gi thub. com/ HarryMJMead/ cvar return return-capping.
1. Introduction
Proposing the Return Capping reformulation of the CVaR optimisation objective, with proof that the two are equivalent if the cap is set correctly.
In applications that are safety critical, or where catastrophic failure is a possibility, it can be beneficial to minimise the worst case outcomes of decisions, rather than simply per- forming well on average. Risk-averse reinforcement learn- ing (RL) addresses this problem by maximising the perfor- mance of policy using a risk metric as the objective, rather than expected value. We focus here on conditional value at risk (CVaR). CVaR represents an intuitive, coherent (Artzner et al., 1999) risk metric that assesses the expected value of the worst a proportion of runs. Note that in this work we focus on static CVaR, where static CVaR refers to the CVaR of full episode returns, due its ease of interpretability for setting level of risk aversion.
Presenting a method for approximating this target cap.
Performing empirical evaluations of this method in established risk-aware benchmarks, where Return Cap- ping has better performance than state-of-the-ar base- lines.
We also introduce a new CVaR policy gradient baseline based on PPO, which we show performs better than standard baselines in some environments.
A common approach to maximising CVaR in RL is to use
2. Related Work
policy gradient (PG) methods (Tamar et al., 2015). In the
case of CVaR, PG methods sample a set of trajectories, then discard all but the bottom a-proportion based on total re- turns, and then aim to maximise the expected return of this remaining subset of trajectories. Whilst this is sufficient for
Training for risk-averse behaviour has been explored in many existing works. Direct reward shaping has been used to explicitly discourage risky behaviours (Wu et al., 2023). Work in robust adversarial RL (Pinto et al., 2017: Pan et al., 2019) involves simultaneously training both the primary agent and an adversary that perturbs environment dynamics in order to train more robust policies. However, in both of these approaches, it is challenging to quantify the level of risk-aversion that results from either the adversary or reward shaping.
'University of Oxford. Correspondence to: Harry Mead <hmead ionecad@robotsoo wx..oosoox..c..k....
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Return Capping: Sample-Eficient CVaR Policy Gradient Optimisation
Trajectory Returns
Tralectory Returns
Sampled Trajectories
Sampled Trajectories
(a) Standard CVaR policy gradient
(b) Return Capping
Figure 1. Comparison of trajectory usage of an example batch of sampled trajectories between standard CVaR Policy Gradient methods and Return Capping
method of quantifying risk aversion to use risk met- rics. Possible metrics include mean variance (Sobel, 1982; Sato et al., 2001; La & Ghavamzadeh, 2013; Prashanth & Ghavamzadeh, 2016) or Value at Risk (VaR) (Filar et al., 1995; Chow et al., 2018), however these metrics are not coherent (Artzner et al., 1999), which can present issues when using them for decision making. Examples of co- herent risk metrics include entropic value at risk (Ahmadi- Javid, 2012), the Wang risk measure (Wang, 2000), or, the focus of this work, conditional value risk (CVaR) (Rock- afellar et al., 2000). Optimising CVaR using policy gradi- ents has been explored in a number of works (Tamar et al., 2015; Tang et al., 2019; Keramati et al., 2020; Schneider et al., 2024; Kim & Min, 2024). However, much of this work is based Distributional-RA (DRL. (Bellemare et al., 2017), and generally focuses on dynamic CVaR (Dabney et al., 2018; Ma et al., 2020; Lim & Malik, 2022; Schneider et al., 2024). Unlike static CVaR, dynamic CVaR is Marko- vian (Ruszczyúski, 2010), and thus can be locally optimised in policy. However, is difficult to interpret and can lead to overly conservative or overly optimistic behaviours (Lim & Malik, 2022). (Lim Malik, 2022) does suggest mod- ification DRL to optimise for static CVaR, but results in (Luo et al., 2024) show poor performance in relation to policy gradient methods.
objective, before annealing the risk-sensitivity to the target value. Alternatively, (Luo al., 2024) presents a policy- mixing approach, where the final policy is mixture of risk-neutral and risk-averse policy. However, the bene- fits of this method are greatest only in environments where there are large regions where the optimal risk-neutral and risk-averse policy are the same, which cannot generally be assumed to be the case.
An alternate approach for risk-averse RL is Constrained RL. Constrained RL presents problem formulation for settings where distinct reward and cost functions exist, where the objective is to maximise reward subject to constraints on the cost (Stooke et al., 2020; Yang et al., 2021; Wu et al., 2024). However, explicitly separate reward and cost func- tions limits the applicability of these methods to certain environments, and our work instead focuses risk over single optimisation metric.
3. Background
3.1. Conditional Value at Risk (CVaR)
Let Z be a bounded random variable with a cumulative distribution function F()) = P(Z I 2). In the case of this paper, the random variable 2 will be the return of trajectory obtained from agent acting in an environment. For Z, the value risk (VaR) confidence level a is defined as
The most similar works to ours, with focus on efficient optimisation of static CVaR are (Greenberg et al., 2022) and (Luo et al., 2024). (Greenberg et al., 2022) shows sig- nificant efficiicff improvements over baselines but requires the environment be formulated as Context- MDP (Hallak et al., 2015), where the context captures part, or all, of the environment's 's randomness. To account for vanishing gradients, (Greenberg et al., 2022) proposes Soft Risk, where the policy is initially optimised for a risk-neutral
VaR. = 2 a).
(1)
The conditional value at risk (CVaR) at confidence level is then defined
CVaR (Z) 1
(2) dz.
(2)
2

---

Return Capping: Sample-Efficient CVaR Policy Gradient Optimisation
Alternatively, if Z is continuous, the CVaR can be expressed
Algorithm Return Capping
Input:
risk level a, batch size N, training steps M, mini-
CVaR, (Z) = 2 < (Z)]
(3)
mum cap CM, cap step size 1j
Initialise: policy Th, value function Vo.. C CM for E 1 M do
Therefore, the CVaR, (Z) can be interpreted as the expected value of the variable Z in the bottom a proportion of the tail. In our case, the C (It) is the expected total return R of the worst performing a proportion of trajectories.
// Sample Trajectories
{Tili=ı + sample I/ Cap Trajectories
{r f=)==1 + cap.trajectories((; (Ti)i=1)
3.2. CVaR Reinforcement Learning
Update 0,, O2 using PPO (B1 Vegi [,;) 1=1
Consider a Markov Decision Process (MDP) defined by (S, A. P. R), corresponding to the state space, the action space, the state transition probabilities and the state transi- tion rewards respectively. Optimal static CVaR policies may be non-Markovian, as static CVaR is dependant on the tra- jectory history. To account for this, we instead solve for an augmented MDP with state space 5+ = (S, r), where Er r is the sum of all rewards up to the current time-step in the trajectory (Lim & Malik, 2022). From this MDP, a policy nh. parametrised by 0, maps state input to a dis- can
VaR < calculate VaR(a, (T))==)) C + C + m)( - -C)
C + max(C,
end for
a 0.05, 95% of all trajectories will be discarded, and much greater number samples are required to get an equivalent batch size to when optimising for expected value. Another potential issue with standard CVaR-PG optimisa- tion is blindness to success (Greenberg et al., 2022). Given a batch of sampled trajectories, the standard CVaR policy gradient does not differentiate between high trajectory re- turns due effective actions being sampled or environment stochasticity benefitting the agent. However it is likely, es- pecially early in training, that proportion of these high returns are due to better policy actions sampled in those specific trajectories. Due to the nature of CVaR-PG, if these trajectories fall outside of the bottom-a proportion of runs, the policy training is effectively blind to these successes. A related issue is a vanishing gradient when the tail of the distribution of trajectory returns is sufficiently flat. In envi- ronments with discrete rewards, it is possible that the bottom a proportion of rewards are all equally bad, resulting in the CVaR-PG gradient vanishing entirely. Similarly to the blind- ness to success, gradient vanishing is most likely to be an issue early in training. The solution presented in (Greenberg et al., 2022) for both issues is to initially optimise for a risk-neutral policy and then slowly anneal a to the target value, but this can present issues when the risk-neutral and the CVaR optimal policies are sufficiently distinct policy may become stuck in the local optimal of the risk-neutral policy.
tribution over possible actions. With this policy, sample trajectories = {(8i; d;, r=]][= and compute the total discounted return R(() Ei I' using a discount factor y We denote the expected return of policy as
J(xo) = E [R())]
(4)
Similarly, we denote the policy CVaR confidence level as
(TO) = Ey [R(T) & VaR.. (5)
Given set of trajectories [Ti))= the CVaR policy gradient (CVaR-PG) method aims to maximise Ja (ma) by perform- ing gradient ascent with respect to the policy parameters 0. The CVaR gradient estimation is given by (Tamar et al., 2015)
Voja (TTi))"ma
=
N
1
1 1
(A(TT) (R(Ts)
N
1=0
VaR.)
Ve log To To(a (Lt)
(6)
L=0
In this formulation, the indicator function means that only samples from the bottom a-proportion of runs are used to compute the gradient.
4. Return Capping
Rather than discard all trajectories outside of the bottom proportion, instead we propose that we keep all trajectories but truncate all returns above a cap C. Algorithm 1 outlines the implementation of this method, and Figure illustrates the difference in how the trajectories are handled compared to standard CVaR-PG. Below, we show that, if this cap C set correctly, the optimal policy for this problem
3.3. Limitations of CVaR-PG
The primary issue with the standard method for CVaR-PG optimisation is the poor sample efficiency, especially in cases where the target small, For an example of where
3

---

Return Capping:
Sample-Efficien CVaR Policy Gradient Optimisation
equivalent to the optimal & policy.
and so now we have (1 - az) aa as upper bound the right integral. For the right integral, for all so VaRx 2 VaRa(=*) for all æ. Therefore, the optimal CVaRa policy x results in
Proposition 4.1. Suppose x² is the optimal CVaRa policy, and (=²)( is the VaR of this policy. Any policy that satisfies πT = arg m By~x (ri=smmin(R)) ((()) C*] where C' = VaRa(=*) will
will also be aVaRa optimal.
min( V VaRa dr
Proof. Below is an expression for the expectation of the capped trajectories.
.. (T) da = (a-a)) (14)
J (TU; C) Er [min( C)
(7)
If we denote (T as the VaR of confidence level æ of the returns of trajectories sampled using the policy x. we can rewrite this equation as
and so is also equal the right integral upper bound. There- fore,
max JO - (ar',
(15)
J (TT; C) - %/- mii (TaR C) dz.
C)=
(8)
and so the optimal CVaR, policy is also the optimal policy to maximise the expected return of the truncated trajectories. As this policy satisfies the upper bound defined by Equa- tions 11, and 13, all optimal policies must satisfy this upper bound. Therefore, from Equation 11, the optimal policy to maximise the expected return on the truncated trajectories must also be the optimal CVaR, policy, when the cap is set to VaR..
Suppose now we set C' VaRa and split the integral into two regions separated by a
(TTT VaRa(T)))
% min( VaR, (TT), VaRa(=²)) () de
S. mimmmaa VaRa(=*)) dx. (9)
4.1. Cap Approximation
Whilst we have shown equivalence between the two prob- lems when the cap is set to VaR, in most environments, this value will be unknown and must be approximated. In order to approximate this value, we use the set of trajectories we have previously sampled.
Consider the first half of the equation.
min VaR (TT), (Tr,("")) dx
< % VaRm (To) de- = aJa((T) (10)
For the trajectories Th where h denotes the number of policy gradient up- dates, a possible approximation is C = (T How- ever, this is likely to be a high-variance approximation of VaR, (TT).. especially with smaller batch sizes. In order to reduce variance, we used the following update rule to update the approximation after each policy update step.
N [(r))]]]]] &.
sampled using the policy
Therefore, the upper bound of the first half of the integral is
max (T) a:J (T*).
(11)
As VaR,,nt (r) is monotonically increasing with respect to x, when x S a, VaR=(TT) < VaRa(=*), and in the left
integral x < a for all x. Therefore, the optimal CVaR, policy results in
C2 = Ck-1 + 17 (VaRo(TTL_.)) - Ck-1)
(16)
0 min VaR (TT), VaR,(T") dx
where v parametrises the size of the cap update step.
In order to reduce the effect of vanishing gradients, we also introduced a minimum cap value C.M In many environ- ments, it is relatively simple to determine the CVaR, of an overly conservative policy that does nothing, e.g. betting 0 at each turn in the betting game outlined below in Sec- tion 5.3. We suggest that any environment where finding a risk-averse policy is compelling should have a policy that performs better than this highly conservative baseline. By setting the minimum cap value above the performance of this baseline, we reduce the likelihood of vanishing gra- dients. However, necessary to ensure that C. is a
/ VaR=(T") do aJa(TT) (12)
and so the left integral is equal to its upper bound. Now considering the right integral
min( (TaR VaR,, (T="("T)) (T*)) dx
2 $ (a)o = -1 (1-a) VaR, (13)

---

Return Capping: Sample-Efiiciet CVaR Policy Gradient Optimisation
valid minimum, where we define valid minimum cap CM I (**). Given,
CVaR-PPO: A CVaR-PG implementation using PPO (Outlined in Appendix A).
CV)Ra S VaR. (T), VT
(17) When evaluating Return Capping against these baselines, our focus is on CVaR relation to total number of agent steps in the environment, As the CVaR-PG methods involve discarding a proportion of trajectories, either each gradient update step will have smaller batch size compared to Re- turn Capping, or the total number of gradient update steps will be reduced. In order to fairly assess the performance of the CVaR-PG baselines, we will compare Return Cap- ping the better performing of these two options. We have included details on all hyperparameters used for baselines and Return Capping in the Appendix.
and
CV)Ra(TT) S CVaRa( VTT
(18)
the CVaR, of any policy valid minimum cap. There- fore, we know that the CVaR, of the conservative baseline is valid. In order to reduce the likelihood of vanishing gra- dients, we assume a cap marginally higher is also valid, but further increasing of the minimum increases the proba- bility of the cap being invalid. Alternative minimum caps that would guarantee validity include the CVaR, of a ran- policy,
dom policy, or the CVaR, of the optimal although the latter may be non-trivial determine.
5.2. Environments
We use range of environments to compare the reward- capping algorithm the baseline methods. In selecting illustrative environments, we focus on environments where the CVaR optimal policy is significantly different from the optimal expected value policy. We have outlined these envi- ronments below, with further detail in the Appendix.
4.2. Reward Adjustment
Given the objective function in Equation the environment rewards needs to be adjusted as sum to the capped reward. Given reward rt at timestep f, let
R
T1
(19)
Goal
To calculate the adjusted reward ri
of = mimminnee C) min(RR-1s
(20)
Using this adjusted reward, use reward-to-go to compute the policy gradients, rather that just using total episode return.
Figure 2. Autonomous Vehicle navigation environment from (Rigter et al., 2021). Colours indicate the road type as follows - black: highway, red: main road, blue: street, green: lane
5. Experiments
5.1. Baselines
Although (Tamar al., 2015) derives Equation the policy gradient for CVaR optimisation, empirical evidence suggests this baseline performs poorly in practice (Green- berg et al., 2022; Luo et al., 2024). This outcome is not unexpected, given it both does not incorporate a state-value function baseline (Sutton & Barto, 1998), and also bases the gradient update on total episode return, rather than return- to-go, making credit assignment more difficult. In order to present more representative baselines, choose com- pare return capping the following:
5.3. Betting Game
The betting game environment (Bäuerle & Ott, 2011) repre- sents a useful domain to assess agents aiming to optimise CVaR due to the substantial and obvious difference between how the agent should act depending on aiming to optimise either expected value or CVaR. the environment, the agent begins with 16 tokens and is able to wager proportion of these tokens with p(win) r. If the agent wins, will win the number of tokens wagered, but will lose the same amount on loss. The game is sequential, such that after each bet, the subsequent bet is based on the current total of tokens the agent has. The game continues for a fixed number of steps; in our experimentation, six. For x > 0.5, the optimal policy to maximise expected is to always bet all
Expected Value: PPO (Schulman et al., 2017) maximis- ing the expected value of return;
CVaR-PG: Unmodified CVaR-PG from Equation 6; and
5

---

Return Capping: Sample-Efficiet CVaR Policy Gradient Optimisation
in the environment, however in (Greenberg et al., 2022), this penalty is limited only the first 32 environment steps, This results in the CVaR of a policy that does not reach the goal but also does not ever cross the guard being higher than the CVaR of policy that optimally takes the shortest path passing through the guard. This reduces the likelihood that a policy optimising CVaR would converge to this shortest path. This is because with this limit on step penalties is longer optimal compared random exploration. In our experimentation, we remove this limit and demonstrate that Return Capping still converges to the CVaR-opting policy,
A discrete state space variant of the Guarded Maze was introduced by (Luo et al., 2023). However, the maze used in (Luo et al., 2023) moved the goal such that the differ- ence between the shortest path (9 steps) and the optimal path avoiding the guard (11 steps) was much smaller. This reduces the challenge of policy optimisation since it reduces the likelihood of policies converging to the local optimal of the risk-neutral policy. In our implementation, the goal is positioned as shown in the continuous example shown in Figure 3, resulting in a much greater difference between the shortest (6 steps) and guard avoiding (I4 steps) paths.
Figure 3. Continuous Guarded Maze Environment. Two trajecto- ries shown here- one taking the shortest path passing through the guard whereas the other takes longer path that avoids the guard
available tokens at each step. Such policy results in high return trajectories infrequently the majority of trajectories will result negative return. A CVaR policy should aim for even the worst performing runs to result in positive return. For our implementation. 0.8.
5.6. Lunar Lander
5.4. Autonomous Vehicle
We also evaluate on Lunar Lander from OpenAl Gym (Brockman et al., 2016; Towers al., 2024). Whilst the other benchmarks have relatively simple environment dynamics, Lunar Lander represents more complex physics- based environment where the optimal risk-neutral policy is non-trivial. In order to induce a difference between optimal risk neutral and risk-averse policies, we follow the example of (Luo et al., 2023) where the environment is modified so that landing on the right-hand side results in an additional high-variance reward. (Luo al., 2023) uses a zero-mean reward sampled from // 1) 100. However, reward such as this does not necessitate a difference between the neutral and CVaR optimal policies, as a policy that landed on the left-hand side of the landing pad could be optimal both for CVaR and expected value of return. To fix this, we modify the environment such that in addition to the zero-mean, high- random reward, landing on the right-hand side also results in an additional 70 reward.
The second environment used is adapted from (Rigter et al., 2021). An autonomous car must navigate along different types of roads to reach the goal state, represented shown in Figure 2. The cost of traversing a given road is sampled from distribution that depends on the type of road it is, being either highway, main road, street or lane. The exact distributions sampled from are outlined in Appendix C.
graph
5.5. Guarded Maze
Pictured in Figure 3, the agent in the Guarded Maze aims to reach the green goal state, receiving a reward for doing so. However, should the agent pass through the red guarded zone, it will receive cost penalty sampled from random distribution. We adapt two versions of the Guarded Maze from (Greenberg et al., 2022) and (Luo et al., 2023; 2024), which vary in respect to state space and the guard cost dis- tribution. The guard cost distribution is parameterised such that an optimal resk-neutral policy would ignore the guard and always take the shortest path to the goal, whereas a CVaR optimal policy instead should take the longer path to avoid the guard. We have modified both these environ- ments in order to make them more challenging, with these modifications outlined below.
6. Results
Figures 4, 5, and 6, show the performance of Return Cap- ping against the baselines outlined above. We also plot the performance of Return Capping depending on the initiali- sation of C.M, using either the VaR of the optimal CVaR policy, the CVaR of the optimal expected value policy, and either slightly above the CVaR of the conservative 'do noth- ing' policy or the CVaR of random policy when the former does not exist.
The Guarded Maze, shown in Figure 3, from (Greenberg et al., 2022) consists of a continuous state space but discrete action space. The agent is penalised for each step takes
6

---

Return Capping: Sample-Efiiiien CVaR Policy Gradient Optimisation
-2.
-2.
Return Capping
-5.0
-5.0
Expected bed
-7.
CVaR-PG
-75
10.0
10.0
-12
-22
Value (DVaf)
CM Conservative (CVaR)
15.00
15.0
Optimal
0.4
0.0
0.6
Eny Steps
Eny Steps
1e6
(a) Baselines
(b) Minimum Return Cap
Figure 4. Betting Game -V performance, Comparing to Baselines and showing effect different minimum return caps
Betting Game: In the betting game, the optimisation ob- jective was CVaRa= For comparing to the baselines, we initialised CM based on the CVaR of the conservative policy betting nothing at each turn. We see in Figure 4(a) that neither CVaR-PPO or CVaR-PG match the performance of Return Capping. As the probability of winning was suf- ficiently high, the optimal CVaR policy is not trivially to bet nothing each turn. However, we see that when using CVaR-PG, training converges to this overly conservative policy, resulting in CVaR close to zero. From Figure 4(b), see that performance was affected by C. initial- isation. However, all three initialisations result in better performance than the baselines.
ting, we see in Figure 5(c) that although CVaR-PPO does consistently converge to the optimal policy, Return Cap- ping converges in approximately half many environment steps. Interestingly. Figures 5(c) and 5(f) show the in both guarded maze domains, the final policy is highly sensitive to C Unlike in the two previous domains, where setting CM (TR produced the best results, here we see the opposite. This behaviour is likely due to the difference in complexity of the neutrall and risk-averse policies. As mentioned in Section 5.5, the optimal risk-averse path approximately twice as long as the risk-neutral path. When the cap is high, ing to the local, less complex risk neutral policy, reducing policy exploration, and resulting in the risk-averse policy never being discovered. However, when the cap is set suf- ficiently low, the policy is initially agnostic to trajectory length, given the trajectory return is above the cap. This allows for sufficient exploration for the optimal risk-averse policy to be found.
this results in the policy converg-
Autonomous Vehicle:
In the autonomous vehicle domain, the optimisation objective was CVaR., C. was ini- tialised using a random policy. In this domain, CVaR-PPO performs very similarly Return Capping, whereas CVaR- PG takes significantly longer to converge. However, of the eight seeds we ran in this domain, two of the CVaR-PPO runs, did not converge to goal-reaching policies. We have chosen to exclude these two seeds from the plot in Figure C to present a more representative baseline (see Appendix C for the plot without outliers removed). However, the pres- ence of these outliers suggests the Return Capping may be more robust method compared CVaR-PPO. Figure 5(d) shows that that the MM initialisation had minimal effect the training performance.
In the lunar lander environment, unlike the other environments, initialising the cap minimum using the conservative policy did not result in consistent conver- gence to the optimal CVaR policy. Figure 6(b) illustrates the relative performance of the three cap minimums. For comparisons to the baselines, we have used C equal to the CVaR of the risk-neutral policy. However, as this policy is non-trivial to determine, we have also included an offset Return Capping plot, accounting for the 4e5 environment steps required to learn the risk-neutral policy. Figure 6(a) shows the performance of Return Capping against two of the baselines. We have not included the CVaR-PG baseline here was unable discover any successful policy. The Lunar Lander environment illustrates the issues present with standard CVaR-PG methods. The CVaR-PPO baseline con- verges to very conservative policy that does land the left but is subsequently unable to improve, which can likely be attributed to the blindness to success issue. However, using
Lunar Lander:
Guarded Maze: In the continuous state space setting, Fig- ure 5(b) shows that Return Capping outperforms all base- lines. For each method, we ran six seeds. None CVaR-PG runs converged to a policy that reached the goal, and of the CVaR-PPO seeds, two converged to the CVaR-optimal pol- icy, one converged to the risk-neutral policy, and three did not converge to goal-reaching policies. All six of the Re- turn Capping runs converged to the -Vat-ootimal policy, again indicating greater training robustness compared to other CVaR-PG methods. In the discrete state space set-

---

Return Capping: Sample-Efiiciet CVaR Policy Gradient Optimisation
Besm Copplng
20
25
30
CUUB 0.15
0.0
Rebur Casping
-40
Expected Value
-100
-100
Gupcag
-120
-1.5
-240
-120
-330
br Sugs
365
Er Stggs
162
165
(a) AV: Basclines
(b) C-GM: Baselines
(c) D-GM: Baselines
50
20
20
CAA 0:2
Clu 8:23
CO0002
-50
-0
-40
-100
Dpectedd Vaice (OVAAJ
Expected Ville K(64
Experted Valir (hiss)
Rundor 1CWU
Cetceralivve (CG
Consersattee (Crufl
Cf Optimal CAR (vah)
Optival CAR (ra))
Opdimal CVSB (1a))
-120
-120
-100
Dne Stpt
Step
Jeh
Dry Stspe
165
(d) AV: Minimum Return Cap
(c) C-GM: Minimum Return Cap
(f) D-GM: Minimum Return Cap
Figure 5. Autonomous Vehicle (AV) CVaRa==.os Continuous Guarded Maze (C-GM) CVaR.=o.os and Discrete Guarded Maze (D-GM) CVa performance. Comparing to Baselines and showing effect of different minimum return caps.
Return Capping, we are able to train policy that both lands on the left-hand side but also is able to land without being overly conservative. Additionally, we have included the MIX baseline from (Luo et al., 2024). Given we require the risk-neutral policy to set CMM, we included the MIX baseline mixing this policy with a learnt risk-avers policy. However, see from this baseline that this simply converges to the of the
such as those proposed in (Greenberg al., 2022; Lim & Malik, 2022; Luo et al., 2024). Although these combination algorithms fell outside the scope of this research, examining whether performance improvements can be made here may an interesting area to explore for future work.
A Acknowledgements
rest-neutral policy. This environment highlights issues with the MIX baseline, being that if the risk-neutral and risk-averse policies are in direct opposition in some regions, learning the risk-averse policy may be made more difficult by the interference of the risk-neutral policy, and increases the probability of converging to this local optimal.
This work was supported by the EPSRC Centre for Doc- toral Training in Autonomous Intelligent Machines and Sys- tems [EP/S024050/1]. Lacerda and Hawes have received EPSRC funding via the "From Sensing to Collaboration" " programme grant [EP//V0004881].
7. Conclusion
Impact Statement
This paper proposes a reformulation of the CVaR optimi- sation problem by capping the total return of trajectories used training, rather than simply discarding them. We have shown empirically that this method shows significant improvement to baseline CVaR-PG methods in number of environments, although there is still space for additional experimentation in broader range of environments. As we have shown, some environments are sensitive to the mini- mum cap value, and further environment testing may help to better identify environment characteristics that lead to this.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
References
Ahmadi-Javid, A. Entropic value-at-risk: A new coher- ent risk measure. Journal of Optimization Theory and Applications, 155:1105-1122 2012
Further work could be done examining how Return Capping can be combined with other CVaR optimisation methods
Artzner, P., Delbaen, F., Eber, J.-M., and Heath, Coherent
8

---

Return Capping: Sample-Eficiien CVaR Policy Gradient Optimisation
100
Clal:: 0.2
CVAR: 0.2
1000
-100
Capping (offsee)
CVaR.PPO
C Expected Value
-200
C Conservative (CVaR)
MIX (offset)
Optimal CVaR (vaR)
-300
-300
0.0
0.2
0.4
0.8
0.8
1.0
12
1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Env Steps
166
Env Steps
le6
(a) Baselines
(b) Minimum Return Cap
Figure 6. Lunar Lander aVaRa=0.2 performance. Comparing to Baselines and showing effect of different minimum return caps. We have included MIX (Luo al.. 2024) as an additional baseline. (offsett plots offistt by the training steps required to learning the nestrnl policy.
measures of risk. Mathematical finance, 9(3):203-25) 228. 1999.
in Neural Information Processing Systems, 35:32639 32652, 2022.
Bäuerle, N. and Ott, J. Markov decision processes with vverage-value-atriis criteria. Mathematical Methods of Operations Research, 74:361-379, 2011.
Hallak, A., Di Castro, D., and Mannor, S. Con-
textual markov decision processes. arXiv preprint urXiv:/502.000 2015.
Bellemare, M. G., Dabney, W., and Munos, R. A distribu- tional perspective on reinforcement learning. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th Inter- national Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 459-
Keramati, R., Dann, C., Tamkin, A., and Brunskill, E. Being optimistic be conservative: Quickly learning evar policy. Proceedings of the AAAI Conference Artificial Intelligence, 34(04):4436 4443, Apr. 2020. doi: 10.1609/ aaai. mmii.v34004.5800. URL / /orgg// i index.php/ https//pppA.ataaattcclle//ee//e/
PMLR, 06-11 Aug 2017.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J.,
Kim, J.-H. and Min, S. Risk-sensitive policy optimization via predictive evar policy gradient. In Forty-first Interna- tional Conference on Machine Learning. 2024.
Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint 1rXiv:1606.01550 2016.
Chow, Y., Ghavamzadeh, M., Janson, L., and Pavone, M. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (167):1-51, 2018.
La, P. and Ghavamzadeh, M. Actor-critic algorithms for risk-sensitive mdps. Advances in neural information pro- cessing systems, 26, 2013.
Lim, S. H. and Malik, I. Distributional reinforcement learn- ing for risk-sensitive policies. Advances in Neural Infor- mation Processing Systems, 35:30977-30983 2022.
Dabney, W., Ostrovski, G., Silver, D., and Munos, R. Im- plicit quantile networks for distributional reinforcement learning. In Dy, J. and Krause, A. (eds.), Proceed- ings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learn- ing Research, pp. 1096-1 105. PMLR, 10-15 Jul 2018. https:: :tattts::/procee http///rrceeensss..eee mlr. dabney 8a. html.
Luo, Y., Liu, G., Poupart, P., and Pan, Y. An alternative to variance: Gini deviation for risk-averse policy gradient. Advances in Neural Information Processing Systems, 36: 2023.
wwtp:sswww.......ee.eemm pww.ss/v0/
Luo, Y., Pan. Y., Wang, H., Torr, P., and Poupart, P. A simple mixture policy parameterization for improving sample efficiency evar optimization. In RLC, pp. 573-592, 2024.
Filar, J. A., Krass, D., and Ross, K. W. Percentile per- formance criteria for limiting average markov decision processes. IEEE Transactions on Automatic Control, 40 (1):2-10, 1995.
Ma, X., Xia, L., Zhou, Z., Yang, J., and Zhao Q. Dsac: Distributional soft actor critic for risk-sensitive reinforce- ment learning. arXiv preprint arXiv:2004. /4547, 2020.
Greenberg, I., Chow, Y., Ghavamzadeh, M., and Mannor, S. Efficient risk-averse reinforcement learning. Advances

---

Return Capping: Sample-Efiicien CVaR Policy Gradient Optimisation
Pan, X., Seita, D., Gao, Y., and Canny, J. Risk averse robust adversarial reinforcement learning. In 2019 Inter- national Conference Robotics and Automation (ICRA), pp. 1522-8528 IEEE, 2019.
Tang, Y. C., Zhang, J., and Salakhutdinov, R. Worst cases policy gradients. arXiv preprint /9t1:::91110000.0 2019.
Towers, M., Kwiatkowski, A., Terry, J., Balis, U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.1703. 2024.
Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. Ro- bust adversarial reinforcement learning. In Precup. D. and Teh, Y. W. (eds.), Proceedings of the 34th Interna- tional Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2817- 2826. PMLR, 06-11 Aug 2017.
Wang. S. S. A class of distortion operators for pricing finan- cial and insurance risks. Journal of risk and insurance, pp. 15-36, 2000.
Prashanth, L. and Ghavamzadeh, M. Variaane--onssraineed actor-critic algorithms for discounted and average reward mdps. Machine Learning, 105:367-417, 2016.
Wu, L.-C., Zhang, Z., Haesaert, S., Ma, Z., and Sun, Z. Risk- aware reward shaping of reinforcement learning agents for autonomous driving. In IECON 2023- 49th Annual Conference of the IEEE Industrial Electronics Society, pp. 1-6, 2023.
Rigter, M., Lacerda, B., and Hawes, N. Risk-avers bayes- adaptive reinforcement learning. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, W. (eds.), Ad- vances in Neural Information Processing Systems, 2021.
Wu, Z., Tang, B., Lin, Q. Yu, C., Mao, S., Xie, Q., Wang. X., and Wang, D. Off-policy primal-dual safe reinforcement learning. arXiv preprint arXiv:2401. 400.:4448.,4 2024.
Rockafellar, R. T., Uryasev, S., et al. Optimization of condi- tional aalue-at-risk. Journal of risk, 2:21 421, 2000.
Yang, Q. Simão, T. D., Tindemans, S. H., and Spaan, M. T. Wesac: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pp. 10639 10646, 2021.
Ruszczyúski, A. Risk-averse dynamic programming for markov decision processes. Mathematical programming, 125:235-200 261, 2010.
Sato, M., Kimura, H., and Kobayashi, S. Td algorithm for the variance of return and mean-variance reinforce- ment learning. Transactions the Japanese Society for Artificial Intelligence, 16(3):353-362 2001.
Schneider, L., Frey, J., Miki, T., and Hutter, M. Learning risk-aware quadrupedal locomotion using distributional reinforcement learning. In 2024 IEEE International Con- ference Robotics and Automation (ICRA), pp- 11451- 11458, 2024. doi: 10. 11009ICRA771447222.2.6666
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017.
Sobel, M. J. The variance of discounted markov decision processes. Journal of Applied Probability, 19(4):794-802 1982.
Stooke, A., Achiam, J., and Abbeel, P. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133- 9143. PMLR, 2020.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, 1998.
Tamar, A., Glassner, Y., and Mannor, S. Optimizing the evar via sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
10

---

Return Capping: Sample-Efiicien CVaR Policy Gradient Optimisation
A. CVaR PPO Given the Standard PPO loss as defined by
(a(a))) Tua (als)
r(0)
(21)
L CLP (0) = ETTWT
+)
(22)
and the standard CVaR objective
(T)) = -ETNTO [R(T))R(T) & (RRa(
(23)
We introduce the CVaR-PPO loss
I. C. (0) By 10
< (R()
(24)
Effectively, only doing the PPO update using the worst a proportion of trajectories. Similarly, the value function update also done only on this proportion of trajectories. We show that this baseline works better than the standard CVaR Policy Gradient in some environments.

---

Return Capping: Sample-Efficient CVaR Policy Gradient Optimisation
Table 1. Betting Game Hyperparameters
PARAMTER
EXPECTED VALUE PPO
RETURN CAPPING
CVAR-PPO
CVAR-PG
CVAR
1
0.2
0.2
0.2
NUM UPDATES
200
NUM STEPS PER UPDATE
5000
EPOCHS PER BATCH
5
SUB BATCH SIZE
50
1000
0.99
IB-3
PPO
CLIP
0.2
GAE A
0.95
ENTROPYC
IB-S
RETURN CAPPING
0.2
OPTIMAL CVAR (VAR) MINIMUM CAP
16
EXPECTED VALUE (CVAR) MINIMUM CAP
-16
CONSERVATIVE (CVAR) MINIMUM CAP
B. Betting Game
State Space: The agent's state its current number of tokens (does need to be integer number), and its current turn
Action Space:
The agent's actions are to bet (0%, 12.5%, 25%, 37.5%, 50%, 62.5%, 75%, 87.5% of its current tokens.
Reward:
The agent receives reward after each turn based on the total number of tokens gained/lost.
Environment Dynamics: Episode terminates after 6 bets or losing all tokens
12

---

Return Capping: Sample-Efficient CVaR Policy Gradient Optimisation
Table 2. Autonomous Vehicle Road Time Cost Distributions
ROAD TYPE
SMALL COST
MEDIUM CoST
LARGE COST
LANE
STREET
4
5
11
MAIN ROAD
2
4
13
HIGHWAY
2
18
Table Autonomous Vehicle Hyperparameterss
PARAMTER
EXPECTED VALUE PPO
RETURN CAPPING
CVAR-PPO
CVAR-PG
CVAR
1
0.05
0.05
0.05
NUM UPDATES
400
200
66
NUM ENV STEPS PER UPDATE
1000
2000
6000
EPOCHS PER BATCH
1
SUB BATCH SIZE
50
300
0.99
IB-3
PPO
CLIP E
0.2
GAE À
0.95
ENTROPY
IB-5
RETURN CAPPING
CAP
0.6
OPTIMAL CVAR (VAR) MINIMUM CAP
27
EXPECTED VALUE (CVAR) MINIMUM CAP
6
RANDOM (CVAR) MINIMUM CAP
-256
C. Autonomous Vehicle
State Space: Discrete and position of agent
Action Space: Move {Up, Down, Left, Right}
Reward: Table outlines the distribution of costs in the Autonomous Vehicle environment. When the agent traverses a road, the cost is sampled from the distribution outlined, with 0.4, 0.3, 0.3 probability of receiving the Small, Medium or Large cost respectively. Reward of for reaching the goal.
Environment Dynamics: Max number of env steps 32
Figure 7 illustrates the performance in the Autonomous Vehicle without the outliers removed. These plots show the 95% confidence interval. Due to the outlier runs, the mean performance of CVaR-PPO is substantially worse than the other methods.
13

---

Return Capping: Sample-Effiiiet CVaR Policy Gradient Optimisation
50
-50
CVaR: 0.05
-100
-150
-200
Return Capping CVaR-PPO
-50
Expected Value
CVaR-PG
-300
0.0
0.5
1.0
2.0
2.5
3.0
3.5
4.0
Env Steps
1e5
Figure Autonomous Vehicle with outliers cVa performance
Table 4. Guarded Maze (Continuous) Hyperparameters
PARAMTER
EXPECTED VALUE PPO
RETURN CAPPING
CVAR-PPO
CVAR-PG
CVAR
1
0.05
0.05
0.05
NUM UPDATES
100
200
66
NUM STEPS PER UPDATE
10000
EPOCHS PER BATCH
6
SUB BATCH SIZE
50
500
0.99
IB-3
PPO
CLIP E
0.2
GAE A
0.95
ENTROPYC
IB-5
RETURN CAPPING
C)P
0.2
OPTIMAL CVAR (VAR) MINIMUM CAP
-3.7
EXPECTED VALUE (CVAR) MINIMUM CAP
-54
CONSERVATIVE (CVAR) MINIMUM CAP
-151
D. Guarded Maze (Continuous)
State Space: Continuous and position of agent
Action Space: Move (Up, Down, Left, Right]
Reward: for each step in the environment. Reward of 16 for reaching the target zone. Guard is present with 20%. If the Guard is present, cost is sampled from an exponential distribution with a mean of 32.
Environment Dynamics: Max number of env steps 161. The movement actions are all affected by added noise.
14

---

Return Capping: Sample-Efficient CVaR Policy Gradient Optimisation
Table 5. Guarded Maze (Discrete) Hyperparameters
PARAMTER
EXPECTED VALUE PPO
RETURN CAPPING
CVAR-PPO
CVAR-PG
CVAR
1
0.2
0.2
0.2
NUM UPDATES
100
40
40
NUM STEPS PER UPDATE
1000
5000
EPOCHS PER BATCH
6
SUB BATCH SIZE
50
1000
0.99
IB-3
PPO
CLIP
0.2
GAR A
0.95
ENTROPYC
IB-S
RETURN CAPPING
0.2
OPTIMAL CVAR (VAR) MINIMUM CAP
-4
EXPECTED VALUE (CVAR) MINIMUM CAP
-40
CONSERVATIVE (CVAR) MINIMUM CAP
-90
E. Guarded Maze (Discrete)
State Space: One-hot encoding of discrete state
Action Space: Move (Up, Down, Left, Right]
Reward: -1 for each step in the environment. Reward of 10 for reaching the target zone. Guard cost is sampled from a normal distribution / ((, 1) x 30.
Environment Dynamics: Max number of env steps 100.
15

---

Return Capping: Sample-Efficient CVaR Policy Gradient Optimisation
Table 6. Lunar Lander Hyperparameters
PARAMTER
EXPECTED VALUE PPO
RETURN CAPPING
CVAR-PPO
MIX
CVAR
0.2
0.2
0.2
NUM UPDATES
150
NUM ENV STEPS PER UPDATE
10000
EPOCHS PER BATCH
SUB BATCH SIZE
50
2000
0.99
1E-3
PPO
CLIP E
0.2
GAE A
0.95
ENTROPY
1E-5
RETURN CAPPING
CA
0.8
OPTIMAL CVAR (VAR) MINIMUM CAP
270
EXPECTED VALUE (CVAR) MINIMUM CAP
200
CONSERVATIVE (CVAR) MINIMUM CAP
-128
F. Lunar Lander
State Space, Action Space, Environment Dynamics: As presented in OpenAl gym box2D (Brockman al., 2016).
Reward: As presented in OpenAl gym box2D, but with additional 70 +N(00, 1) 100 reward for landing on right size. The left and right regions are shown in Figure 8
Left
Right
Figure 8. Lunar Lander environment from (Luo al., 2023) showing divide between left and right landing zones
16


Reviews:
[{"review_id": "fBv3xL70z9", "reviewer": "Reviewer_iMqv", "summary": "This paper proposes a novel method for CVaR optimization in Reinforcement Learning. The proposed method caps trajectory returns by a certain value and maximize its expected value with respect to a policy. It is theoretically shown that, the maximizer of the proposed objective matches the conventional optimal CVaR policy if the capping threshold value $C$ is set to the VaR of the optimal CVaR policy. In practice, the capping threshold $C$ is approximated by a moving average of VaR of the learning policy. The proposed method requires to set the minimum capping threshold $C^M$ appropriately. The proposed method does not need to discard sample trajectories unlike naive baselines. The effectiveness of the proposed method is validated several numerical experiments.\n\n## update after rebuttal\nI keep my score from the following reasons.\n- The authors answered adequately to my question that \"$C^M$ seems more difficult to set in larger domains\". However, I believe that ultimately the practical applicability meeds to be evaluated experimentally.\n- Though the paper makes solid contribution, the absence of convergence analysis slightly hurts the quality of the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for referencing [3], we will include it in Related Work\n\n**Setting Cap Minimum**\n\nAddressing the question on setting $C^M$ in larger environments, it is very possible to set a suitable $C^M$, irrespective of the complexity of the environment.\n\nWe show in Proposition 1 that the Return Capping optimisation objective is equivalent to optimising for CVaR if $C$ is set to $VaR_\\alpha(\\pi^*)$, so we need to ensure that $C^M$, is less than or the equal to $VaR_\\alpha(\\pi^*)$.\n\nGiven Eqn (17)\n$$CVaR_\\alpha(\\pi) \\leq VaR_\\alpha(\\pi), \\quad \\forall \\pi$$\nand Eqn (18)\n$$CVaR_\\alpha(\\pi) \\leq CVaR_\\alpha(\\pi^*), \\quad \\forall \\pi$$\nwe know that the $CVaR_\\alpha$ of any policy is necessarily less than the $VaR_\\alpha(\\pi^*)$. So we know that if we set $C^M$ to  $CVaR_\\alpha$ of any policy, it will be less than $VaR_\\alpha(\\pi^*)$. This means we can take any policy and sample a set of trajectories and use this sampled $CVaR_\\alpha$ to set $C^M$. This could be a random policy, or it could be a policy that maximises Expected Value.\n\nEven if an environment was sufficiently complex that it was not immediately apparent what an appropriate value of $C^M$ was, it would always be possible to use either the  $\\text{CVaR}_\\alpha$ of a random policy, or a policy trained in any manner to set $C^M$.\n\nRequiring a previously trained policy to set $C^M$ does increase the samples required for training. However, for all but the AV environment, Return Capping would outperform all baselines even accounting for the additional training required to learn an Expected Value optimal policy (shown as Return Capping (offset) in Lunar Lander as this was the only environment that required the cap to be set according to the Expected Value policy, but we could include this in all other environments as well).\n\n**Environment Simplicity**\n\nWe have discussed issues with more complex environments further in our rebuttal to reviewer r3Le, but these issues arise more from challenges with optimising for CVaR in general, rather than any specific characteristics of Return Capping.\n\n**Additional Baselines**\n\nAddressing the lack of inclusion of [1, 2] as baselines, the reason we have not included [2] is because it requires the environment to be formulated as a Context-MDP where the context encapsulates the environment randomness. This limits the applications of the baseline as not all problems can be formulated as such. The reason we have only included [1] in Lunar Lander is that this baseline requires an optimal Expected Value policy. Lunar Lander was the only environment where Return Capping required the Expected Value policy to set $C^M$. However we could include this baseline in other environments - it generally just converged to the optimal Expected Value policy, similarly to in Lunar Lander.\n\n[1] Luo, Yudong, et al. \"A simple mixture policy parameterization for improving sample efficiency of cvar optimization.\"\n\n[2] Greenberg, Ido, et al. \"Efficient risk-averse reinforcement learning.\"\n\n[3] Kim & Min, “Risk-Sensitive Policy Optimization via Predictive CVaR Policy Gradient”", "claims_and_evidence": "In my view, the contributions of this paper are mainly twofold; (1) the proposal of the novel objective Eq. (7) and the establishment of the equivalence with the conventional objective (Proposition 4.1) and (2) the proposal of the practical method to optimize the proposed objective and its numerical validation. The first contribution is supported by the proof of Proposition 4.1 and the second contribution is supported in Section 5. The proposed method consistently performs better than baselines.", "methods_and_evaluation_criteria": "The proposed method seems sound and the evaluation criteria makes sense to validate the aforementioned two claims, though the environments are relatively small and simple, and baselines are limited to Expected PG, CVaR-PG and CVaR-PPO.", "theoretical_claims": "I checked the proof of Proposition 4.1 and it seems correct.", "experimental_designs_or_analyses": "The experimental design seems to appropriate to validate the aforementioned two claims. However, since the simplicity of the environments, it is not fully convincing that the proposed method scales to the larger environments, where the $C^M$ is more difficult to set appropriately, i.e., Expected Value (CVaR) and Optimal CVaR (VaR). In addition, comparison with stronger baselines (e.g. [Greenberg et al., 2022] and [Luo et al., 2024]) is absent.", "relation_to_broader_scientific_literature": "The aforementioned contribution (1) is an interesting reformulation of CVaR optimization in RL. Broad class of RL methods could be applied to this formulation.", "essential_references_not_discussed": "The following paper [1] is not referred, which does not naively discard the sampled trajectories but assigns \"weights\" to them and optimize CVaR for RL.\n[1] Kim & Min, Risk-Sensitive Policy Optimization via Predictive CVaR Policy Gradient, ICML, https://proceedings.mlr.press/v235/kim24x.html.", "other_strengths_and_weaknesses": "- Weakness  \nThe absence of the convergence analysis of Algorithm 1.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Do you think that the proposed method works well in the larger environments, such as Atari, where the $C^M$ seems more difficult to set appropriately? It would be grateful if you could discuss with some evidence.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q1XsT5yhhE", "reviewer": "Reviewer_r3Le", "summary": "The authors address the problem of risk-sensitive policy optimization via policy gradient methods. They note several issues with the standard formulation of PG+CVaR which cause catastrophic losses in performance: specifically, because it discards the best trajectories by design, it is extremely difficult for an RL algorithm to encounter \"lucky\" transitions early in training from which it can learn a useful policy, resulting in extremely poor sample efficiency.\n\nThe paper proposes an alternative formulation for computing risk-sensitive policy gradients, by _capping_ the returns rather than subsampling the worst-case trajectories. They show that optimizing this capped-return formulation is equivalent to the standard formulation of CVaR under the condition that the cap is set at the VaR, and devise a practical algorithm to compute it by estimating the VaR online.\n\nThey show strong results with a practical implementation of this algorithm on top of PPO in risk-sensitive control baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time spent reviewing this paper and we appreciate your feedback\n\n**Environment Complexity**\n\nThe main issue raised in this review is the limited complexity of the environments used. Whilst we agree that more complex environments would benefit the paper, as far as we are aware, there is very limited work that has presented promising results for optimising static CVaR for episode return in more complex environments.\n\nSome related work does explore more complex environments. In Safe RL, such as [4], the main benchmarks are modified MuJoCo environments [7]. However, unlike in our work, in Safe RL, the objective is to maximise expected return subject to constraints on cost, where environments have a distinct cost and reward function. Some distributional RL work has been done in MuJoCo [5] and Atari [8]. However these works focus on dynamic CVaR, rather than static CVaR.\n\nThe work we are aware of optimising for static CVaR referenced in the paper [1, 2, 6] as well as a similar work [3] suggested by Reviewer iMqv all focus primarily on comparatively similar scale environments to the examples presented in our paper. One of the exceptions are some modified MuJoCo environments in [1]. However, in practice, when optimising policies in these environments we found no distinction between optimal Expected Value and optimal CVaR policies. The other exception is the Atari game Asterix in [6]. This paper presents a modification to distributional DQN to optimise for static CVaR and shows promising results. However, results from [8] show that optimising for dynamic CVaR using distributional DQN in Asterix results in better Expected Value performance that a policy optimising for Expected Value, so it is ambiguous whether the improved CVaR performance in [6] was due to a distinct CVaR optimal policy being found, or due to this aforementioned result shown in [8].\n\nThe main issue we have found in scaling to more complex environments is an issue inherent to return CVaR optimisation rather than specifically our method which is that for CVaR optimisation techniques to be relevant, the environment has to have distinct CVaR-optimal and Expected-Value-optimal policies. Generally, what we have observed is that one of these policies will be less complex to learn, and so in more complex environments, either the Expected Value optimal policy converges to the CVaR-optimal policy, or all CVaR optimisation methods converge to the Expected Value optimal policy.\n\n**Off Policy Methods**\n\nThank you for pointing out the two papers [4, 5] as both are relevant to risk-sensitive RL and we will include them in reference to Related Work. However, as mentioned above, they are both optimising for different objectives compared to our work.\n\n\n\n\n\n\n**Other Strengths and Weaknesses**\n\nAddressing the two points the review raised in the strengths and weaknesses section:\n- The reason we have included (offset) In the Lunar Lander environment, is we found better performance by optimising for Expected Value and then using this policy to set $C^M$ as outlined in Section 4.1, even accounting for the additional environment steps required to train the optimal Expected Value policy (see Return Capping (offset)). In all other environments, we set $C^M$ based on a trivial policy that took no actions.\n- Conditioning the cap on the initial state is a sensible suggestion if the desired goal is to optimise for CVaR conditioned on the initial state.\n\n\n\n\n\n[1] Luo, Yudong, et al. \"A simple mixture policy parameterization for improving sample efficiency of cvar optimization.\"\n\n[2] Greenberg, Ido, et al. \"Efficient risk-averse reinforcement learning.\"\n\n[3] Kim & Min, “Risk-Sensitive Policy Optimization via Predictive CVaR Policy Gradient”\n\n[4] Yang, Qisong, et al. \"WCSAC: Worst-case soft actor critic for safety-constrained reinforcement learning.\" \n\n[5] Ma, Xiaoteng, et al. \"Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning.\"\n\n[6] Lim, Shiau Hong, and Ilyas Malik. \"Distributional reinforcement learning for risk-sensitive policies.”\n\n[7] Ji, Jiaming, et al. \"Safety gymnasium: A unified safe reinforcement learning benchmark.\"\n\n[8] Dabney, Will, et al. \"Implicit quantile networks for distributional reinforcement learning.\" International conference on machine learning.", "claims_and_evidence": "The authors claim that their method is better able to converge to a risk-sensitive policy. This does seem to be true in practice on the environments studied (where the CVaR of their method is substantially better than both the risk-neutral policy and prior work in risk-sensitive RL).", "methods_and_evaluation_criteria": "The method is evaluated in several risk-sensitive RL settings, although a few (particularly Lunar Lander) are somewhat contrived in order to force a difference between risk-sensitive and risk-neutral policies. Evaluations are fair, although given the focus on sample complexity it also might have been nice to include an off-policy method (where CVaR-based RL algorithms have also been studied, see [1, 2]).\n\nGenerally, my main complaint with the evaluation is that the environments studied are very simple; considering more complex environments would greatly strengthen the paper's claims.\n\n[1] Yang, Qisong, et al. \"WCSAC: Worst-case soft actor critic for safety-constrained reinforcement learning.\"\n[2] Ma, Xiaoteng, et al. \"Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning.\"", "theoretical_claims": "I looked over the proof for Proposition 4.1 and it seems reasonable, though I did not check it in detail.", "experimental_designs_or_analyses": "The experimental results are strong if somewhat limited; the most complex task studied is Lunar Lander which has some odd design choices in the experimental study:\n> we modify the environment such that in addition to the zero-mean, high-variance random reward, landing on the right-hand side also results in an additional 70 reward.\n\nI would suggest the authors find some more realistic or complex environments in which CVaR is helpful.", "relation_to_broader_scientific_literature": "The paper is well-positioned in the risk-sensitive RL setting, proposing a relatively simple theoretical modification to the risk-sensitive RL setting that induces a fairly large improvement in practice.", "essential_references_not_discussed": "As mentioned previously I would recommend comparing against works that consider risk-sensitive control with off-policy approaches [1, 2].\n\n[1] Yang, Qisong, et al. \"WCSAC: Worst-case soft actor critic for safety-constrained reinforcement learning.\"\n[2] Ma, Xiaoteng, et al. \"Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning.\"", "other_strengths_and_weaknesses": "- Learns a risk-neutral policy first (at least in Lunar Lander, see: (offset)), but it's not clear how this fits into the overall algorithm (maybe adding another algorithm block would be helpful for clarity).\n - The cap value is fixed over all trajectories, but it should probably be conditioned on initial state for any environments in which initial state has a large impact on returns (many practical environments).", "other_comments_or_suggestions": "- The decision to exclude two outlier results from the AV results for the PPO baseline is odd; I would suggest using the original plot (and plot the median between seeds if there is a concern of outliers).\n - The results section writes out how many training runs of each baseline converged to which policies:\n>For each method, we ran six seeds. None of CVaR-PG runs converged to a policy that reached the goal, and of the CVaR-PPO seeds, two converged to the CVaR-optimal policy, one converged to the risk-neutral policy, and three did not converge to goal-reaching policies\n\nI would suggest conveying this information visually for all environments if it's important.", "questions_for_authors": "No additional questions; my main concerns are on the complexity of the environments studied.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zzAJ2mjCjz", "reviewer": "Reviewer_7ZeW", "summary": "The authors consider risk sensitive reinforcement learning where the goal is to optimize the $\\alpha$-parameterized tail of the return distribution (on the lower end). Prior work proposed an approach known as conditional value at risk (CVaR) policy gradient, where the algorithm filters out all trajectories except the worst $\\alpha$ fraction and does policy gradient on the remaining trajectories. \n\nThe authors note that this results in wasting a lot of samples, and propose an alternative equivalent formulation with a parameterized threshold for capping the return instead.  The authors show that for an appropriately chosen (only approximable) threshold for the return cap, the two approaches are equivalent. The paper evaluates the proposed approach against prior baselines in numerical experiments on toy domains and shows benefits to the new proposal.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In relation to the size of the environments presented, we have discussed this in our rebuttal to reviewer r3Le. \n\n**Sample Efficiency**\n\nFor the specific question on sample efficiency, whilst there are improvements to sample efficiency using Return Capping compared to CVaR PG, it is unlikely to achieve improvements of magnitude $\\frac{1}{\\alpha}$ as by capping returns, we do lose some information about trajectories. However there is still information to be learnt from capped trajectories.\n\nPrimarily, examples of trajectories that reach the cap are present in policy optimisation. In standard CVaR PG, we effectively only have negative examples of trajectories that resulted in low returns. Whilst we lose the relative performance between capped trajectories by capping returns, we still do get a gradient between the uncapped low performing trajectories and these capped trajectories. By training using capped trajectories, the policy is able to learn from positive examples (e.g. actions that result in the trajectory reaching the cap) rather than just having to rely on negative examples.\n\nIn terms of speed of convergence observed in empirical results, in both Guarded Maze environments and the Lunar Lander environment, we see much faster convergence for Return Capping compared to the risk-sensitive baselines. The Expected Value policy does converge more quickly in all environments but this is unsurprising given it is learning from all uncapped trajectories. In the betting game, the CVaR PG policy does converge more quickly, albeit to a less optimal policy, than Return Capping. We suggest that this is likely due to the CVaR PG policy being a much more simplistic, conservative policy compared to the more optimal Return Capping policy. We agree that in the AV environment, there is minimal performance improvement over CVaR PPO*, but we suggest that the better performance of Return Capping in all other presented environments suggests it is a better method overall. \n\n*It should be noted that the plot in Figure 5a excludes CVaR PPO outliers that did not converge to the optimal CVaR policy (see Figure 7 in the Appendix for the unmodified plot), so although median performance is comparable to Return Capping, in this environment, CVaR PPO is less consistent at finding the CVaR optimal policy.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The main claim is in Proposition 4.1, which looks reasonable to me.", "experimental_designs_or_analyses": "Yes, the authors consider an array of small scale toy environments to evaluate the new algorithm against baselines. The authors also modify the lunar lander environment to make it more interesting to see the difference between risk neutral versus risk sensitive optimization, which seems like a reasonable design.", "relation_to_broader_scientific_literature": "The authors show an intuitive and easy to implement alternative that is equivalent to prior work (filtering trajectories), but with better sample efficiency.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths: Clear and concise algorithm proposed based on a novel but simple insight with numerical evidence.\n\nWeaknesses: Most of the experiments are on very small toy domains.", "other_comments_or_suggestions": "The relation between Eq (2) and (3) appears elementary but might be worth spelling out clearly as it may not be obvious when encountering for the first time.", "questions_for_authors": "While the claim about sample efficiency sounds reasonable, it is not clear how that materializes in practice given the lack of sensitivity of the policy to the outcome beyond the threshold. Furthermore, the graphs appear to show a benefit for the final risk sensitive return, but not necessarily the speed of convergence (as one may expect for a drastic reduction in sample efficiency).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N16ibAMkVv", "reviewer": "Reviewer_saRX", "summary": "This paper presents a new method for optimizing Conditional Value at Risk (CVaR) in reinforcement learning using policy gradients. Traditional CVaR policy gradient methods suffer from poor sample efficiency because they discard a large proportion of trajectories. The authors propose Return Capping, a novel approach that retains all trajectories but caps high returns at a threshold, ensuring better sample efficiency. Empirical results across multiple risk-sensitive environments show that Return Capping outperforms existing CVaR methods in both learning efficiency and robustness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The main issue suggested in this review is flaws with the theoretical proof of the equivalence of Return Capping and standard CVaR PG optimisation. We would summarise the two points the reviewer raises as:\n- Given Eqn (7), when $C$ is selected in the $\\min(\\cdot)$, this trajectory will have no effect on the optimisation of $\\pi_\\theta$\n- Eqn (7) is not being applied to the actual return capping algorithm\n\nHowever, the above points are not the case, as we can show below.\n\n**Addressing the first point:**\n\nGiven Equation 7\n$$J^C (\\pi_\\theta; C) = E_{\\tau \\sim \\pi_\\theta} [\\min(R(\\tau), C)].$$\n\nTo compute policy gradients, we need to compute the gradient with respect to $\\theta$\n$$\\nabla_\\theta J^C (\\pi_\\theta; C) = \\nabla_\\theta E_{\\tau \\sim \\pi_\\theta} [\\min(R(\\tau), C)].$$\nThis can be expanded to the integral\n$$\\nabla_\\theta J^C (\\pi_\\theta; C) = \\nabla_\\theta \\int_\\tau P(\\tau|\\theta) \\min(R(\\tau), C).$$\nThis integral highlights the flaw in the first point, as when either $R(\\tau)$ or $C$ is selected in the $min(\\cdot)$, the probability of that given trajectory is still dependent on $\\theta$, and thus will have an effect on the policy optimisation. This addresses the first point made.\n\n**Addressing the second point:**\n\nFrom the equation above, we can move the gradient inside the integral and use a log-derivative trick to put the integral in the form\n$$\\nabla_\\theta J^C (\\pi_\\theta; C) = \\int_\\tau P(\\tau|\\theta) \\nabla_\\theta \\log P(\\tau|\\theta) \\min(R(\\tau), C).$$\n\nWe can then return this back to an expectation\n$$\\nabla_\\theta J^C (\\pi_\\theta; C) = E_{\\tau \\sim \\pi_\\theta} [\\nabla_\\theta \\log P(\\tau|\\theta) \\min(R(\\tau), C)],$$\nand then we can reformulate the trajectory probability in terms of action probability\n$$\\nabla_\\theta J^C (\\pi_\\theta; C) = E_{\\tau \\sim \\pi_\\theta} [\\sum_{t=0}^T \\nabla_\\theta \\log \\pi_\\theta (a_t|s_t) \\min(R(\\tau), C)].$$\nFrom here, we can apply commonplace RL techniques such as using return-to-go, rather than using full episode returns, and using a Value function baseline.\nNote that return-to-go is computed as:\n$$\\hat{R}^C_t =  \\min(\\sum_{t'=0}^T R(s_{t'}, a_{t'}, s_{t'+1}), C) - \\min(\\sum_{t'=0}^t R(s_{t'}, a_{t'}, s_{t'+1}), C).$$\nWhen incorporating both of these techniques, the gradient update becomes:\n$$\\nabla_\\theta J^C (\\pi_\\theta; C) = E_{\\tau \\sim \\pi_\\theta} [\\sum_{t=0}^T \\nabla_\\theta \\log \\pi_\\theta (a_t|s_t) (\\hat{R}^C_t - V(s_t)].$$\nIn practice, we used a Generalised Advantage Estimator to compute advantage. This gradient update can then be clipped, using the PPO algorithm, and then this is the exact gradient update used in Return Capping. As such the Return Capping gradient update is derived directly from Eqn (7), addressing the review’s second point.\n\nWe are happy to provide any further clarification about the proof, or how the derivations above show that the reviewer’s main issues are unfounded.\n\n**Other Issues Raised**\n\n- In relation to the scale of the environments, we have discussed this thoroughly in our review to r3Le\n- The reason we have only included MIX [1] in Lunar Lander is that this baseline requires an optimal Expected Value policy. Lunar Lander was the only environment where Return Capping required the Expected Value policy to set $C^M$. However we could include this baseline in other environments - it generally just converged to the optimal Expected Value policy, similarly to in Lunar Lander. As explained in the paper, the Return Capping (offset) example was included to demonstrate the relative performance of Return Capping accounting for the environment steps required to train an optimal Expected Value policy.\n\n[1] Luo, Yudong, et al. \"A simple mixture policy parameterization for improving sample efficiency of cvar optimization.\"", "claims_and_evidence": "1. It is claimed that the solution of return capping objective results in the $CVaR_\\alpha$ optimal policy if the cap is set correctly. I believe there are major technical flaws in the proofs of this; for example, the author claimed that (7) is the objective of return capping. However, in my perspective, if $C$ is independent of $\\tau$, when $C<R(\\tau)$ such that $C$ is chosen in the $min(\\cdot)$ function, it will have no effect of optimizing $\\pi_\\theta$. Therefore, (7) is actually equivalent to (5), and the provided theoretical result is trivial, not being applied to the actual return capping algorithm authors proposed.\n\n2. Using $VaR_\\alpha(\\pi_{\\theta_{k-1}})$ at a starting of a training does seem to be a sensible choice.", "methods_and_evaluation_criteria": "Proposed method does make sense, but its not theoretically grounded. The algorithms are evaluated with CVaR, which is a trivial choice.", "theoretical_claims": "I have checked the correctness of the proof. The proof itself seems fine, although the result seems trivial and irrelevant to the proposed method.", "experimental_designs_or_analyses": "The experiments suggest superiority of the proposed method compared to baselines. However, \n1. The experiments are only performed in relatively low-dimensional environments. Considering lack of theoretical contributions, I would expect more experiments and empirical analyses compared to previous works.\n2. Different comparisons are conducted in different environments, e.g., offset, mix(offset) are only experimented in Lunar Lander. This lowers the statistical significance of the comparisons.", "relation_to_broader_scientific_literature": "This paper proposes a practical extension to previous CVar Policy optimization algorithms.", "essential_references_not_discussed": "I am not aware of essential references not discussed here.", "other_strengths_and_weaknesses": "- overall the paper is less dense compared to other top conference level papers. long, not important proof is included in the main paper, and environment describing figures, and result figures are organized in a less dense way.", "other_comments_or_suggestions": ".", "questions_for_authors": "- What do you think about the issue above regarding the theoretical result?\n- Is there any other way to formulate and justify the proposed algorithm?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a clear and meaningful contribution to sample-efficient CVaR optimization in reinforcement learning. The authors introduce a novel approach that caps returns based on an estimated VaR, in contrast to conventional methods that discard high returns. This modification is shown to improve sample complexity. The method is partially supported both theoretically—demonstrating that, under an accurate VaR estimate, the optimal policy under the proposed objective aligns with the CVaR-optimal policy—and empirically in relatively simple domains.

A notable weakness lies in the treatment of the parameter $C^M$. While the paper offers some guidelines for selecting $C^M$, its theoretical justification remains limited, and the method’s performance is sensitive to this choice. Moreover, the appropriate value (strategy to set it) appears to vary significantly across domains (e.g., Figure 5(f) vs. Figure 6(b)). Nonetheless, the paper introduces a novel idea that is intuitively appealing and opens promising directions for future research in CVaR optimization.
Decision: Accept (poster)