Predictive CVaR Q-learning
Authors: Ju-Hyun Kim
Seungki Min
Venue: ICLR 2026 Poster (2026)
Keywords: CVaR optmization
Risk-sensitive RL
Q-learning
Bellman equation
Policy improvemen
Primary area: reinforcement learning
TL;DR: We introduce a new Bellman equation tailored for the CVaR objective, and develop an efficient Q-learning algorithm accompanied by a policy improvement theorem.

Abstract: We propose a sample-efficient Q-learning algorithm for reinforcement learning with the Conditional Value-at-Risk (CVaR) objective. Our algorithm is built upon predictive tail value function, a novel formulation of risk-sensitive action value, that admits a recursive structure as in the conventional risk-neutral Bellman equation. This structure enables the Q-learning algorithm to utilize the entire set of sample trajectories rather than relying only on worst-case outcomes, enhancing the sample efficiency. We further derive a Bellman optimality equation and a policy improvement theorem, which provide theoretical foundations of our algorithm and remedy inconsistencies that have existed in the literature. Empirical results demonstrate that our method consistently improves CVaR performance while maintaining stable and interpretable learning dynamics.

Published as a conference paper at ICLR 2026
PREDICTIVE CVAR Q-LEARNING
Ju-Hyun Kim
Advanced Instrumentation Institute Korea Research Institute of Standards and Science (KRISS) Yuseong-gu, Daejeon 34113, Republic of Korea jhk mm@ krsss.........................ss re. kr
Seungki Min
Business School
Seoul National University
Dongjak-gu, Seoul 08826, Republic of Korea mskyt@sun.aad
ABSTRACT
We propose sample-efficient Q-learning algorithm for reinforcement learning with the Conditional Value-at-Risk (CVaR) objective. Our method introduces two key innovations. First, we propose the predictive tail value function, novel for- mulation of risk-sensitive action value, admits recursive structure as in the con- ventional risk-neutral Bellman equation. This novel formulation addresses the problem of noisy policy evaluation originating from the non-decomposable ob- jective. Second, we introduce Two-way exploration strategy that explores the agent's risk-sensitivity level in addition to its actions. This technique mitigates the "blindness to success" phenomenon by preventing premature convergence to overly conservative policies. We establish rigorous theoretical foundation for this framework, including a new Bellman optimality equation and a policy im- provement theorem. Empirical results demonstrate that our algorithm significantly improves both CVaR performance and learning stability.
INTRODUCTION
In high-stakes sequential decision-making tasks, the consequences of rare but catastrophic out- comes cannot be ignored. Standard reinforcement learning (RL), which assumes risk neutrality and optimizes expected returns, is insufficient in such settings. Indeed, risk-sensitive approaches have demonstrated superior performance in variety of safety-critich domains, such autonomous driving (Wen et al.] 2020). robotic surgery (Pore et al. (202)) and finance (Greenberg et al., 2022). Among the various risk measures, Conditional Value-at-Risk (CVaR) has emerged prominent objective, valued for its mathematical tractability (Rockafellar et al. 2000) and its direct focus on worst-case outcomes.
CVaR quantifies the expected loss within the worst-case quantile of return distribution, making a natural fit for agents designed be averse to catastrophic losses. Mirroring the development of risk-neutral RL, methodological advancements to optimize the CVaR objective have primarily evolved along two main avenues: policy-gradien methods (Tamar al.) 2015 Rajeswaran al.) Tamar et al. 20166] Queeney et al. [02 Urpí al., 2021 Greenberg et al. 2022 Markowitz
2016
al. 2023 Kim Min 2024 Mead et al. 2025) and value-based approaches (Bäuerle 2011
Chow et al. 2015 Pflug & Pichler 2016 Stanko & Macek 2019 [Singh et al. 20200] Zhang & Weng,
2021 Lim & Malik] 2022 Li et al. 2022 Wang et al. 2023). Despite this progress, applying these
methods introduces significant challenges.
CVaR RL notoriously smmple-nnfffccient a problem often attributed to its focus on a narrow subset of worst-case trajectories. In this work, argue that this inefficiency stems from two more fun- damental issues: noisy policy evaluation due to a lack of temporal decomposition, and ineffective exploration caused by ignoring successful outcomes.

---

Published as a conference paper at ICLR 2026
The first fundamental issue stems from the difficulty of temporal credit assignment. In many value- based formulations, the CVaR objective is treated as single, non-decomposable reward realized only the end of the episode. This structure prevents the agent from assessing the immediate impact of its actions, as the learning signal from an entire trajectory collapsed into one delayed outcome. This lack of temporal decomposition makes policy evaluation exceptionally noisy, which is a primary driver of the sample inefficiency observed in CVaR RL (Hau et al. 2023 Kim & Min 2024).
Compounding this evaluation problem is a fundamental difficulty with exploration. Because the CVaR objective is driven by the worst-case outcomes, the learning signal is derived almost exclu- sively from "failure" trajectories. High-return trajectories that fall outside the lower-risk quantile are effectively ignored, preventing the agent from learning to improve upon already successful behav- iors. This well-documented phenomenon, known as "blindness to success" (Greenberg et al. [2022] Mead et al. 2025), can cause the learning process to stagnate overly conservative, suboptimal policies.
To address these challenges, we introduce Predictive CVaR Q-learning, a novel value-based al- gorithm built upon new, recursive formulation of the CVaR objective. We provide rigorous theoretical foundation for our method, justifying each component of the algorithm, and demonstrate its superior performance and sample efficiency through experiments.
Our primary contribution is a new pair of value functions- the predictive tail value function and the predictive tail probability function that resolves the temporal credit assignment problem. This approach adapts and extends an idea originally proposed in the volicy-gradient setting (Kim & Min (224) for value-based learning. Together, these functions allow objective into a temporally decomposable structure. We prove that this formulation satisfies a risk- neutral Bellman-style style recursion, allowing the learning signal to be propagated at every step of trajectory. This provides dense, immediate feedback for policy evaluation, drastically reducing esti- mation noise and improving sample efficiency.
reformulate the CVaR
Our second contribution is a two-way randomized exploration strategy designed to mitigate "blindness success." In addition conventional e-greedy for action-level exploration, we in- troduce novel exploration in the augmented state space by randomizing the initial risk budget. This encourages the agent to experience trajectories with varying degrees risk sensitivity-sometiie acting boldly, other times conservatively. This exploration of risk preferences prevents the agent from prematurely converging to overly safe, suboptimal policies and promotes the discovery of more robust strategies.
2 PROBLEM SETUP AND PRELIMINARIES
We consider a Linieehhoizzon Markov decision process (MDP) M = (T is the state space. A is the action space, T is the time horizon, $1 S is the initial state, and P. is the transition kernel at time t. At each time step f 1...., T, the agent observes the current state S, E S. chooses an action A, A, receives a reward R2 E IR, and transitions next state S.+1 E S according the transition kernel Pr. i.e., (Re, Se+1) P. S,, A.).
where
For notational convenience, we define Rast
Σt Rr, and (x)+= max 0}.
CVaR optimization Given risk level q E (0, 1], the CVaR of a random variable is defined as
CVaR, = 1 f" [a],][Xdu,
q o
where [a] sup(n E R|F(X- < 1) < q} denotes the Value-at-Risk the risk level i.e.,
the q-quantile of the distribution of total reward.
Our goal is to find the optimal policy that maximizes the CVaR value of the total reward Rir at the risk level q:
sup {CVaR, [Rir]} sell
where II is the set of non-anticipating policies, including randomized ones. More formally, let H, = (Si, A1, RI,..., St-1, A RG R, St) be the history revealed up time I, and let H, be its
2

---

Published as a conference paper at ICLR 2026
space. Each policy II is defined as sequence of functions (Ty :4 - A such that each Ty specifies the distribution over actions at time t given history, i.e., A, ~ Te(: Hi).
State space augmentation As of its most favored properties, the CVaR measure admits variational representation that provides a more tractable alternative to its original definition that involves non-smooth and non-linear structure. Specifically, for any non-anticipating policy π E II, the CVaR can be expressed in the following variational form:
q CVaR
max nER
{ {qm + E"
Here, the factor q on the left hand side is introduced to simplify expressions in later parts. This vari- ational form allows reinterpret CVaR maximization two-stage optimization with respect to the tail risk budget (outer) and the policy (inner):
97 + sup ET [-(1 R B1x
sup xell
{q CVaR, max BER
sell
This representation naturally leads to the idea of state space augmentation, with an additional state variable representing the tail budget. Formally, we introduce residual tail budget process (Y,") T+1 defined as
Y == 17 R1a-1-
where 7 E IR is an auxiliary variable specifying the (initial) tail budget.
A Markov policy living on the augmented state space chooses the current action A, based on the current state S, and the current residual budget Y," Such a policy is specified by an augmented Markov policy kernel X that is defined sequence of functions (x+) such that Xt S x I A maps the current state and the residual budget to an action distribution. That is, under policy induced by a kernel X with a tail budget 77. the action A is chosen according to
A, Xx( |t, Y.").
Note that a kernel X alone does not define a non-anticipating policy π E II; it must be coupled with specific tail budget and we write Px.0 to denote their corresponding probability measure. We denote by X' the set of augmented Markov policy kernels.
Notably, Bäuerle & (201 show that, for dynamic CVaR optimization, it is sufficient to search over the augmented Markov policies instead of the entire set non-anticipating policies:
sup {q- [R max nER
9m7 sup EX XEX
[-(n RI.)) R1+)
sell
Wang et al. 2023 applies dynamic programming (DP) principles based on this observation. Our approach also builds on this insight, but adopts a different formulation, as detailed below.
A line of work (Bäuerle & Ot 2011 Pflug & Pichler 2016
Dynamic programming on augmented state space Viewed as risk-neutral optimization, the inner optimization, supxex Ex.. -(1 Ri:T) can be solved by applying the DP principles on the augmented state space (Pllug & Pichler 2016).
In particular, aforementioned prior studies postulate risk-neutral decision maker who receives total reward of (n - R1.r)+ at the end of horizon. They introduce the action value function of an augmented Markov policy kernel X as
ux. v. a). a) = EX- (n R (S =,==sY," = y. A. = a
which leads to the following Bellman equation:
(2) with u²=, (s. a. - Conventional risk-neutral Q-learning algorithms be applied to optimize the kernel X so that the optimal action value function ut can be obtained, and then the outer optimization reduces simple one-dimensional optimization, max (qn + max, of (81, a))-
(x a)
(S Re, Att1);
3

---

Published as a conference paper at ICLR 2026
However, this approach suffers from sample inefficiency. Among the sample trajectories collected over the course of Q-learning procedure, only small subset of them will be meaningfully utilized since the effective reward will be zero, i.e., (n-RRrr)*
RI:r)*
0, in the majority of trajectories (roughly,
(1 - 1)-fraction of trajectories). The main issue is that the term (1) - R1:r) is non-separable across time steps, effectively delaying the reward realizations the end of time horizon.
THEORETICAL FOUNDATIONS
Existing CVaR-based dynamic programming approaches often rely on augmented state representa- tions and treat the CVaR term as terminal, non-decomposable objective. This leads to significant sample inefficiency and complicates recursive value estimation. In this work, we resolve this by introducing a novel, recursive formulation for the CVaR objective. Our key idea is to define pair of predictive functions - the predictive tail value function and the predictive tail probability func- tion - that permit temporal decomposition and satisfy risk-neutral Bellman-style recursion. This structure facilitates value propagation and policy improvement a manner analogous to standard Q-learning, forming the theoretical bedrock for sample-efficient Predictive CVaR Q-learning algorithm.
Definition (Predictive tail value/probability functions). Given an augmented Markov policy kernel X. its predictive tail value function fx (f. S R A T+ is defined as
(f y. a) = Ex. I(ReTT S ReT
S. = s, Y, = A.
with ff+ (s,8, y. a) = 0. Additionally, its predictive tail probability function gx (g) : SSX R
A [0, 1) is defined as
ge (s, a) = ppx.1=0 (RIT Sy Se =s= =,=," = y. A, A,).
(3)
with of (s, y, a) = 110 < 3).
Here, the choice 7] = is arbitrary, Above notion of predictive tail values and probabilities are invariant in n,
The predictive tail probability function gi quantifies the likelihood that the remaining return from time É onward falls below specified threshold given the current state, residual budget, and ac- tion. This object effectively encodes the probability of entering the CVaR tail conditioned on the current decision point. The idea of modeling such risk-conditioned probabilities is reminiscent of the Predictive CVaR Policy Gradient framework (Kim & Min 2024). which uses similar quantities inform policy gradient-based policy updates. In contrast, we integrate action-conditioning into the predictive structure and embed it within a Bellman-style recursion. This formulation enables value-based approach such as Q-learning, moving beyond trajectory-Ievel estimation and allowing for action selection that directly maximize the CVaR objective.
In addition the predictive tail probability, we define the predictive tail value function for- mally described by the function action value function (so called Q-function). This function captures the expected cumulative re- turn weighted by the probability that the trajectory remains in the CVaR tail from time t onward. This function reflects both the magnitude and likelihood of tail outcomes. Importantly, the return R&T is recursively decomposable, which allows fx to satisfy a risk-neutral Bellman-style recursion. This recursive structure enables value propagation and policy improvement analogous to standard Q-learning, while preserving sensitivity risk throughout the learning process.
a) - as a novel, risk-sensitive analogue of the standard
Next result shows that the objective of the inner optimization, EX.0 - (n Ry.r)* can be repre- sented in terms of fx and gx, and also that fx can be decomposed across time steps.
Assumption 1. Under any non-anticipating policy II and any time i {1,..., T), the distri- bution of remaining return R&T does not have any probability mass.
Proposition 1 (Temporal decomposition). Under Assumption ! for any E S.y y E R, a E A, and I (I,..., T), ,, the following equations hold:
4

---

Published as a conference paper at ICLR 2026
() Ex. Σ 1 (Sr+1, 1++1,A+11 Rr
St =s Yi" y, Ar
fe (..y,a) = T2x""
Yo41, Act1)
S 8, 8, Y = y, A = a],
(4)
and
Ex," (1- ((n-- (a (f1, 77 A1) 9 (11, 7/ A1) x n/
We next show that the predictive tail value function exhibits a recursive structure that is very analo- gous the standard Bellman equation in the ristrllutral setting.
Theorem (Bellman equation). Given an augmented Markov policy kernel X, under Assumption !
its predictive tail value function fx and predictive tail probability function g² satisfy
(s, ( (ePl)l)))))))))))))) (1
[f (S1+1,3/ 21 Re: AL+1)
+ ge (Sc+1. - Pa, Act1) x Ri],
for all 8 &ES. E TR, a E A, and 1. (1,...,7).
Compared to the Bellman equation (Eq. 2 established in prior work, this Bellman equation (Eq. 5) includes an additional term, g²,1 (S R,, Al+1) x R, corresponding to the immediate reward in the standard Bellman equation. This term reflects that the anticipated contribution of the current reward to the objective, enabling efficient value propagation in our suggested CVaR Q-learning algorithm.
Building on this recursion, we can derive the optimality conditions that the optimal kernel has to satisfy and its implication the CVaR objective. This allows characterize optimality and construct improvement principles, despite the MarMaanov nature of CVaR. We begin by defining the notion greedy kernels for the state space augmentation, mirroring the classic notion of greedy policies in the standard Q-learning approaches.
Definition 2 (Greedy kernel). augmented Markov policy kernel X is said to be greedy with respect to predictive tail value/probability function pair (f.9) if
{a E xi y) 0} S arg max (fe(s, a) x y}.
aEA
for all S, y E RR and {1...., T). Theorem (Bellman optimality equation). Define
ex (s,y) y) == EA.-x((((s.)) (fx y, At) - Ie (s. x yl, (f(s)) := sup (s.
xEX
Then, the following holds under Assumption 7
1. Let 11(x) be the set of augmented Markov policies induced by kernel x across all values of 1) E TR. Then,
sup {q NC-CVaR, [Rrr]] nER
max {um + + ((1,
Tell())
2. With respect to all non-anticipating policies.
sup
{q- Cg--VaR,, (RL:]] max of
well
yER
3. xx o² if and only if x is greedy with respect to (fx,g*).
The functions ex² and i² are analogous to the state value function and the optimal state value func- tion, respectively, in the risk-neutral setting. The above result establishes explicit connections be- tween the CVaR objective and predictive tail value/probability functions, and clarifies the meaning of optimizing the augmented Markov policy kernel x through dynamic programming principles.
We now demonstrate that the state-wise improvement of the kernel (in the augmented state space) indeed improves the CVaR performance of the resulting policies.
5

---

Published as a conference paper at ICLR 2026
Theorem 3 (Policy improvement). Consider augmented Markov policy kernel x along with its predictive tail value function fx and predictive tail probability function g². Let X be the greedy kernel with respect to (fx.g²). Then, under Assumption 1
of (8. v) or (8.1), As
Consequently,
sup CVaR [1:T] mell(x)
sup mell(x))
CVaR (RI:T:
for any q E (0, 1].
This result formally guarantees that taking greedy updates with respect the predictive functions yields monotonic improvement in the CVaR objective over the entire augmented state space. This result generalizes the classical policy improvement theorem to the risk-sensitive setting and justifies our value-based approach to CVaR optimization.
In addition, estimating tail probability and tail value separately offers an important advantage rather than estimating a tail expectation itself. In particular, this advantage becomes most pronounced in the challenging cases (1) where the number of trajectories with non-zero effective reward small, still
and (2) where the rewards beyond the threshold exhibit large variance. The tail probability be estimated in a relatively stable way-e.g. by counting the proportion of trajectories exceeding the threshold. In contrast, estimating tail expectation requires regressing on the magnitude of the tail values, which becomes considerably more unstable when the effective-reward samples are both few and high-variance Our method explicitly separates probability estimation (via (g*)) and value estimation (via (fx)) through the temporal decomposition we introduce. This decoupling allows the probability component to be learned in more stable manner, while the value component can leverage the decomposition structure to reduce variance. Moreover, since g represents a probability bounded in ([O,1]) can benefit from stable learning strategies, as the reviewer noted, such as log- loss or KL-ddivereence based objectives, further mitigating noise.
ALGORITHM
Based on the theoretical framework established above, we now present our Predictive CVaR Q- learning (PCVaR-O) algorithm. The algorithm learns three key components: two function approxi- mators, f and g which estimate the predictive tail value and probability, and risk-budget param- eter 77, which tracks the optimal risk level. The learning process driven by a two-way exploration strategy and periodic updates to these parameters, detailed in Algorithm I
Generating sample trajectories with two-way randomized exploration To generate sample tra- jectories, the agent follows behavior policy built upon a two-way randomized exploration strategy (lines 4 The first component is an e-greedy scheme for action-level exploration. This applies random perturbations to the actions of the greedy policy derived from the current function approxi- mators, fo and g..
The second, more distinctive component of this strategy is the exploration within the augmented state space. In line the initial residual budget Y, is sampled from a normal distribution A (1, o instead of being fixed to a single value. Here, 1) corresponds to current best estimate of the optimal risk budget. By sampling around this central value, the agent is prompted to experience trajectories under varying degrees of risk sensitivity, thereby exploring the space of risk preferences more effectively. Analogous to e; for action-level exploration, the variance of governs the extent of this exploration and is typically annealed as training progresses.
Our approach follows this established practice: risk-level exploration provides a flexible heuristic whose hyperparameters can be tuned through standard experimental procedures, just as is common for classical e-greedy exploration strategies. Similar to e-greedy exploration, its exploration is gov- erned by two design choices: the initial value of a and the schedule by which o decays over time. The initial can be reasonably specified from prior domain knowledge or from basic properties of the environment. The decay schedule, much like the choice e-decay in e-greedy, admits multiple reasonable forms and is ultimately selected based empirical validation.

---

Published as a conference paper at ICLR 2026
Algorithm Predictive CV&RR Q-Learning Algorithm (PCVaR-Q)
1: Initialize parameters (0, é, 11), risk budget grid H. replay buffer D.
2: Pretrain parameters (0, 0, vi) using pre-existing sample trajectories, available.
3: for k to K do
4: Set S1 f- $1, and sample Y1 ~ ((n.o))
5: for t= 1 to T do
6: With probability cx. choose A, ~ Uniform(AA) Otherwise, choose A greedily with respect to (f gr)):
7:
A, 4 arg max (S,, V,, 9. (St, Yt, a)) Yı
8:
Execute Ar, observe Re, S1+1; update Y++1 t Yr--ik Rr,
9:
Store (R1:-1, St, A,, RG, S(+1) into replay buffer D. Sample mini-batch B {R1g-I1. A,. R,. S)+1) R from D.
11:
Update & and á using TD losses, Eq. [and Eq.8
0 - 00. (0:
Φ t Φ a V (o)
12:
end for
13:
Every episodes, update risk budget y/:
f- arg max max YEH aEA { (1) (1-
{
14:
end for
Updating the predictive functions and risk budget The learning process involves updating the
parameters of the function approximators, denoted by and for fer and 3 respectively, along with the risk budget parameter 7 (lines 11 13 11-13. From the sampled trajectories, 0 and & are updated using two distinct temporal-differeee
(TD) loss functions.
The parameter is updated updated by minimizing the following TD loss, (0; B), which is derived from the Bellman equation (Eq. 5 in Theorem
(0; = B
(S,,, R1ij-1, Aj)
N'EH 1=1
- (f
Rugh A/t11) + G5tz (S Rigj A/t11) R2] R2))²
where A',1 is the greedy action at (S)+1, - R1,)) with respect to (fo, go). A key aspect of our method is that the loss for each sample computed H C IR, leveraging the property that the predictive functions are invariant to the initial risk budget level. This encourages the function approximators to generalize across various risk levels.
discrete of candidate risk budgets
Similarly, the parameter Φ is updated by minimizing a second TD loss (o; B), derived from the martingale property of the predictive tail probability function (Eq. 4 in Proposition
B
() = B
ΣΣ (g)) (S - -Riy, A/-1) - (S n/ Rug- Rig- (8)
J=E
Finally, the risk level is updated periodically for stability by solving the outer optimization problem in the variational formulation of CVaR (ef. Theorem 2 This newly identified optimal risk budget then serves as the central point for the agent's risk-level exploration in subsequent episodes.
This entire learning procedure constitutes a form generalized policy iteration (GPI), The TD updates for 0 and Φ act as the kernel evaluation step, driving the learned functions satisfy the Bellman optimality equation (Theorem ( The greedy kernel derived from these updated functions then guaranteed to be superior by Theorem 3 This interplay between evaluation and improvement ensures that our algorithm progressively finds better kernel, converging towards a CVaR Cotimnl solution.

---

Published as a conference paper at ICLR 2026
Starting with pretrained parameters (optional) While our core PCVaR-Q algorithm is inher- ently more robust to the sousscesst--ssscesss" phenomenon than prior methods because it utilizes two-way exploration, this issue still be challenge in the early stages of learning. To further mitigate this risk, we propose an optional warm-start by pre-training the parameters (0. o, 7) (line This is highly practical step, it can leverage any dataset of pre-existing sample trajectories (e.g.- ones obtained from a risk-neutral policy). The pre-training procedure itself is straightforward. as it reuses the update rules fit the predictive functions this data, from which an initial risk budget n) can also be derived. This initialization anchors the agent to promising, high-return behaviors from the outset, ensuring a more stable learning dynamic.
EXPERIMENTS
In this section, we evaluate the performance Predictive CVaR Q-learning algorithm in a controlled setting to investigate (1) whether our method improves CVaR performance over -eutrall policies, and (2) whether achieves stable and sample-efficiet learning compared to baselines. We compare our proposed algorithm PCVaR-Q with the other two baselines, RN and CVaR-Q:
RN: Risk-neutral optimal policy learned through risk-neutral Q-learning.
CVaR-Q: The policy learned through a Q-learningstyyee approach based on the Bellman operator in (Pflug & Pichler 2016), as detailed in Algorithm 2
PCVaR-Q: The policy learned through the Predictive CVaR Q-learnigg algorithm.
FCaa 0
AN
Ra M( 17)
(N(,19)
0
Rs -3
(b) The stochastic grid-world en- vironment.
(a) Sequential decision tree.
Figure 1: Illustration of experiment environments.
5.1 SEQUENTIAL DECISION TREE (TREE-STRUCTUU MDP)
Setup We design a finite-horizon MDP to highlight risk-return trade-offs under CVaR optimiza- tion. The agent starts in State 0 and proceeds through branching structure involving both stochastic and deterministic rewards. Figure Ta illustrates the simulated environment. The dynamics are de- scribed as follows. The agent starts in State 0. At State O, the agent moves deterministically to State 1, receiving a stochastic reward R ~ ~(11 12). At State and State 2, the agent chooses between two actions, up and down. The up action yields stochastic rewards whereas the down action yields deterministic rewards and terminates the MDP process (see Figure Ia).
The total return Ri.a Rí + Rg R3 (with missing rewards treated zero) presents clear risk-return trade-off: the down path offers lower variance and moderate return, while the up path provides higher expected return but greater risk. The rest-nl optimal policy always chooses the up action, yielding an average total return of 5.0 but a CVaR value of only 1.96 at level contrast, the CVaR-optimal policy at the same level (q = 0.1) adopts more cautious approach. A simple calculation shows that it will play down at State if R1 < -0.2 and will do so at State 2 if R, + R2 < 1.1. The CVaR performance of this policy is 2. 2.50, and its corresponding VaR value 3.02.
0.1. In
Results To evaluate risk-sensitive behavior, we compare the distributions of total reward R under the policies obtained by running RN and PCVaR-Q over 100,000 episodes. As shown in
8

---

Published as a conference paper at ICLR 2026
PCVaR-Q yields higher lower-tail return, achieving CVaR value of 2.45 risk level = 0.1. whereas RN achieves 1.96 (the optimal CVaR value is 2.50). This confirms that PCVaR-Q successfully finds out the CVaR-optimal policy.
Figure| 2a
Figure 26 tracks the evolution of CVaR performance (q 0.1) throughout the training process: Each solid line represents the mean performance across 10 independent trials. Policies were eval- uated every 100 iterations, with each point estimated from 100,000 sample runs. Our PCVaR-Q algorithm exhibits stable learning curve that steadily converges the CVaR-Q baseline shows high variance and converges to suboptimal policy. This result con- firms the superior stability and sample efficiency of our method, validating the practical benefits of learning context.
near-optimal value, whereas
our proposed recursive structure in
(a) Histograms of total reward for the final PCVaR-Q Q and risk- neutral (RN) policies.
(b) Actual CVaR performance (g 0.1) the course Q-learning.
Figure 2: Results for sequential decision tree
5.2 GRID-WORLD WITH STOCHASTIC TRANSITION AND OBSTACLES.
Setup We conduct experiments in grid- baded navigation environment designed evaluate decision-making under uncertainty and risk. The environment is two-dimensional grid size (height) 10 (width), where an agent starts the bottom-left corner and aims reach goal located at the bottom-right corner. The agent's ' action space consists of four cardinal movements: up, down, left, and righ.. The transition is stochastic: an intended action is successful with a probability 0.7, but with 0.3 probability, the agent moves in one of the other three directions, chosen uniformly at random. The episode terminates the agent reaches the goal, which yields reward from (500, 1), or hits an obstacle, which incurs penalty from (-550. 1). All other transitions receive a reward of as a step penalty. While the risk-neutral policy tends follow direct path toward the goal, often incurs a higher chance of collision with obstacles. In contrast, the risk-sensitive policy, such as PCVaR-Q, learns a safer path that avoids obstacles, even it results in a longer trajectory. Figure illustrates the experimental environment and the distinct policies learned by the risk-neutral (RN) and P agents.
Results Figure Ba presents the total reward distributions from 50,000 evaluation episodes, showing that PCVaR-Q produces a more robust policy. By mitigating the risk of catastrophic penalties, improves the lower tail of the distribution, achieving a CVaR of -55.84 at the q 0.1 level clear improvement over the RN policy's -8.37 and close to the optimal value of -3.34.
Furthermore, Figure 3b illustrates that this superior outcome is achieved through more efficient and stable learning process. The CVaR performance of PCVaR-Q, measured every 1000 iterations, shows rapid and consistent convergence, while the baseline model is more volatile. This demon- strates that method's improved sample efficiency directly translates to better and more reliable risk-aware policies in complex environments.
CONCLUSION
We introduced Predictive CVaR Q-learning (PCVaR-Q), a novel CVaR Q-learning framework de- signed to optimize CVaR objectives in sample-efficient and theoretically grounded manner. Our key contributions include the predictive tail value function, which enables a recursive Bellman struc- ture tailored to CVaR, and two-way randomized exploration strategy that explores both action and

---

Published as a conference paper at ICLR 2026
NR
of Fi of
CravO
(a) Histograms of total reward for the final PCVaR-Q and risk-neutral (RN) policies.
(b) Actual CVaR performance (g = 0.1) over the course of Q-learning
Figure 3: Results for stochastic grid-world.
risk budget spaces. Together, we provide theoretical guarantees including a Bellman equation, opti- mality condition, and policy improvement theorem specific to the CVaRR setting. These results offer (TD) learning style algorithm, thereby extending clas-
principled foundation of temporal-differene sical Q-learning theory into the domain of CVaR measure. Experimental results further demonstrate the practical effectiveness and stability of our approach.
One limitation of our method the increased model complexity: the introduction of the predictive tail value and probability functions requires learning two separate function approximators, and the residual threshold 17 must also be tracked and updated throughout the learning process. Despite this added complexity, our results show that the benefits in sample efficiency and theoretical rigor justify the overhead in most settings. Future work may extend our framework deep RL environments, and explore integration with model-based based risk-sensitive planning. Our method also opens the door for further research on adaptive risk modeling and cafety-criticee learning in real-world applications.
Reproducibility Statement. We have made significant efforts to ensure the reproducibility of our results. Baseline algorithmic descriptions and detailed experimental settings, including hyperparam- eters, are described in Appendix
ACKNOWLEDGMENTS
This work was supported by the New Faculty Startup Fund from Scoul National University.
REFERENCES
Nicole Bäuerle and Jonathan Ott. Markov decision processes with average-value-at-riskk criteria. Mathematical Methods of Operations Research, 74:361-379, 2011.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: optimization approach. Advances in neural information processing systems, 28, 2015.
Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems, 35:32639-32652, 2022.
JL Hau, E Delage, M Ghavamzadeh, and M Petrik. On dynamic program decompositions of static risk mea-sures. Les Cahiers du GERAD ISSN, 711:2440, 2023.
Ju-Hyun Kim and Seungki Min. Risk-sensitive policy optimization via predictive evar policy gradi- ent. In Forty-first International Conference Machine Learning, 2024.
Xiaocheng Li, Huaiyang Zhong, and Margaret Brandeau. Quantile markov decision processes. Operations research, 70(3):142) 1428-14428-1447 2022.
Shiau Hong Lim and Ilyas Malik. Distributional reinforcement learning for risk-sensitive policies.
Advances in Neural Information Processing Systems, 35:30977-30989, 2022.
10

---

Published as a conference paper at ICLR 2026
Jared Markowitz, Ryan W Gardner, Ashley Llorens, Raman Arora, and I-Jeng Wang. A risk- sensitive approach to policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 15019-15027, 2023.
Harry Mead, Clarissa Costen, Bruno Lacerda, and Nick Hawes. Return capping: Sample-efficient cvar policy gradient optimisation. arXiv preprint arXiv:2504.20887, 2025.
Georg Ch Pflug and Alois Pichler. Time-consistent decisions and temporal decomposition of coher- ent risk functionals. Mathematics of Operations Research, 41 41(2):682-69 6999, 2016.
Ameya Pore, Davide Corsi, Enrico Marchesini, Diego Dall' Alba, Alicia Casals, Alessandro Farinelli, and Paolo Fiorini. Safe reinforcement learning using formal verification for tissue traction autonomous robotic-assisted surgery. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4025 4031. IEEE, 2021.
James Queeney, loannis Ch Paschalidis, and Christos G Cassandras. Uncertainty-aware policy op- timization: A robust, adaptive trust region approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9377-9385, 2021.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv: 1610.0028,, 2016.
R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21 1211-42.22 2000.
Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distri- butional reinforcement learning. In Learning for Dynamics and Control, pp. 958-968. PMLR, 2020.
Silvestr Stanko and Karel Macek. Risk-averse distributional reinforcement learning: A cvar opti- mization approach. In IJCCI, pp. 412 423,4423,200 2019.
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the evar via sampling, In Proceedings of the AAAI Conference Artificial Intelligence, volume 29, 2015.
Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Sequential decision mak- ing with coherent risk. IEEE transactions on automatic control, (2(7):3323-3338, 2016.
Núria Armengol Urpi, Sebastian Curi, and Andreas Krause, Risk-averse offline reinforcement learn- ing. arXiv preprint vrXiv:2102.05371 2021.
Kaiwen Wang, Nathan Kallus, and Wen Sun. caar-minimax-opiine risk-sensitive reinforcement learning with evar. In International Conference on Machine Learning, pp. 35864-35907. PMLR, 2023.
Lu Wen, Jingliang Duan, Shengbo Eben Li, Shaobing Xu, and Huei Peng. Safe reinforcement learning for autonomous vehicles through parallel constrained policy optimization. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (P. 1-7. IEEE, 2020.
Jinyy Zhang and Paul Weng. Safe distributional reinforcement learning. In International Confer- ence on Distributed Artificial Intelligence, pp. 107-128. Springer, 2021.
A EXPERIMENT AND IMPLEMENTATION DETAILS
A.1 BASELINE: CVAR-Q ALGORITHM
We employ CVaR Q-learning to solve the CVaR optimization problem. This approach learns a func- tion approximator d that estimates the CVaR value function. The training procedure is summarized in Algorithm 2 While the method shares similarities with Predictive CVaR Q-learning, the key difference lies in the optimization of the parameter 6, which is updated using temporal-differeme (TD) loss derived from the Bellman optimal equation in (Wang et al. (023) such as: for a mini-batch B ((R.g-1 Sj. Aj. S5,S 8
11

---

Published as a conference paper at ICLR 2026
B (S) -1:5-11 A,) (1 Brg Ai+aa))
1
L0,0:
B
W'EH j=1
where Aj+1 E arg aa 1 (S a).
Algorithm 2 CVaR Q-Learning Algorithm (CVaR-Q)
1: Initialize parameters 0,, ④. risk budget grid H, initial H, replay buffer D
2: for k k= to K do
3: Initialize
f 81. Yi f- 1/.
4: for (or1= 1 to d do
5: With probability choose A, Uniform(AA.
6: Otherwise, choose A, greedily with:
A. arg max {ul (Se, Y,, a).
Execute At, observe Re, S update Y1+1 t Y2 R. end for
8:
9:
Store 1R At, R4, St+1) ) into replay buffer D. Sample mini-batch B {(Rig1 Sn, Ag, R2, {3+1)} from D.
10:
11:
Update using TD loss:
0 + 0 ag VoCu
12:
Update risk budget f):
- arg max T'EH aEA
(f + 7 9).
13:
end for
A.2 SEQUENTIAL DECISION TREE
CVaR parameterization To optimize for Conditional Value Risk (CVaR), we define a discrete grid for the risk budget H as follows:
= (-10.0 9.9, -98, 1.48, 14.9, 15.0):
and the quantile level g 0.1. The Predictive CVaR Q-learning algorithm estimates the predic- tive tail value function fx and the predictive tail probability function gx. The CVaR Q-learning algorithm estimates value function ux. A tabular function approximator is used for each of these functions: _ g" and u², where the values are maintained separately for combinations of states, residual risk budgets, and actions. The CVaR objective at given risk budget 1) is computed as:
6(s1. n) 17 (q (s1, n. a')) f1(s,,,,a)),
or alternatively:
(s1, 1) = q (11,
where a' = arg Ta (81, d') fi a') or ' arg max 7- a').
Learning and optimization We utilize both Predictive CVaR Q-learning and CVaR Q-learning algorithms. The learning process maintains estimates of 1 g and w". These are updated via the Adam optimizer. Key settings include:
Learning rates: ag = 0.01, as 0.0001
Epsilon decay: 0.1 - 0.9 0.91e/200]
Batch size: 8 sampled trajectories per episode
Optimizer: Adam with 31 0.9, 82 = 0.999, 10-8
Episodes: 5,000
12

---

Published as a conference paper at ICLR 2026
Risk-level exploration scheduling:
3, if S k < 500
2, if 500 v & < 1000
1, if 1000 < A < 1500
0, if 2 1500
Both g and u² are updated based on cumulative return trajectories. The risk level parameter updated every 500 iterations. The same hyperparameter configuration is applied to both Predictive CVaR Q-learning and CVaR Q-learning algorithms ensure consistency. Under ideal conditions, the CVaR Q-learning algorithm also converges to the optimal risk-sensitive policy. However, we aim highlight the robustness and stability of the Predictive CVaR Q-learning algorithm under the same conditions.
A.3 GRID-WORLD
CVaR parameterization To optimize for Conditional Value at Risk (CVaR), we define a discrete grid for the risk budget Hl as follows:
H (-150) 149, - ...,..., 98, 99, 100),
and the quantile level g 0.1. A tabular function approximator is used for each of these functions: f go, and u², where the values are maintained separately for combinations states, residual risk budgets, and actions. The CVaR objective at given risk budget r) is computed same method before.
Learning and optimization We utilize both Predictive CVaR Q-Jearning and CVaR Q-learning algorithms. The learning process maintains estimates of / , and u". These are updated via the Adam optimizer. Key settings include:
Learning rates: og = 0.01, 0g 0.01
Epsilon decay: max(1 1
repisode
2000 ), 0.0)
Batch size: 1 sample trajectories per episode
Optimizer: fixed learning rates
Episodes: 15,000
Risk-level exploration scheduling:
45, i < 2000
30, if 2000 < k < 4000
15, if 4000 < k 6000
0, if 6000
Pretrain parameters (0, o,1) with risk-neutral policy
Both fo d g and u are updated based on cumulative return trajectories. The risk level parameter updated every 500 iterations. The same hyperparameter configuration is applied to both Predictive CVaR Q-learning and CVaR Q-learning algorithms to ensure consistency.
A.4 COMPUTATION AND IMPLEMENTATION
All experiments were conducted on a system with an 1tth Gen Intel(R) Core(TM) 17-11700K pro- cessor running 3.60 GHz, 32 GB of RAM, and 8 CPU cores. Each experiment for a given seed typically requires about 20 (tree) and 90 world) minutes to complete. The implementation written in Python 3.8+ using only NumPy and Matplotlib, without relying any external reinforce- ment learning libraries (such as Gym or Stable Baselines). All code and experimental results will be released as ZIP file.
13

---

Published as a conference paper at ICLR 2026
B EFFECTIVENESS OF TWO-WAY EXPLORATION AND PARAMETERS PRETRAINING
In this section, we evaluate the performance of Predictive CVaR Q-learning algorithm in a controlled setting to investigate the effect parameters pretraining and sampling. We compare our proposed algorithm PCVaR-Q Q with or without the two technicque, Two-way and Pretrain:
Two-way (O/X): : exploration with both action-level and risk- (O) or with only action- level (X).
Pretrain (O/X): Start with pretrained parameters (0,0,1) v) (0) or zero parameters
(0, (X)
Result Each row in Figure visualizes the distinet paths discovered during training and the state visit frequencies as heatmaps for each model at specific iterations (left: at 2,000 iterations, right: 12,000 iterations). In Figures 4a 4d where models were initialized with zero value functions and trained without pretraining, we observe that the agents fail to learn optimal trajectories. The case without both two-way exploration and pretraining failed to achieve sufficient exploration, re- sulting in poor performance. Even when two-way exploration the agent still converged to suboptimal local behaviors. This phenomenon reflects a blindness to success, which may bring CVaR learning to local optimum deadlock. In contrast, models were initialized with the pretrained parameters in Figures (e)-(h) successfully identify safer trajectories. Notably, models augmented with two-way exploration converge more quickly and consistently to the desirable path, demonstrating improved exploration and stability during training.
enabled without pretraining,
C PROOFS OF THEORETICAL FOUNDATIONS
To establish the theoretical foundations. we begin by introducing a key technical assumption that simplifies the treatment of conditional distributions over cumulative rewards. Assumption 1. Under any non-anticipating policy II and any time & {I,..., T), the distri- bution of remaining return R&T does not have any probability mass.
Assumption T ensures that the cumulative return Rer has continuous distribution, with point masses (atoms). This condition critical to the theoretical framework presented in this work. Specifically, it guarantees that the conditional probability P(R.T Y, S., Y,, As) is differen- tiable with respect to the threshold Y,, which is a key component for defining the tail probability the
function Sºa and constructing the recursive CVaR objective function. Without Assumption function d could exhibit discontinuities or even be undefined. This is due to the presence of atoms in the conditional distribution of returns. This would invalidate several important results, including Lemma I Proposition] 1 and Theorem | which rely on the smoothness of these functions.
C.1 PROOF OF PROPOSITION
In order to derive efficient recursive formulations of the CVaR objective, we first characterize how the CVaR objective can be decomposed temporally. The following proposition shows that the pre- dictive tail value function fx can be expressed as the expectation of cumulative discounted rewards, weighted by predictive tail probability function gx. This result provides the foundation for Bellman-type thrmnan-tyyeeiiee recursive formulation presented Theorem
Proposition 1 (Temporal decomposition). Under Assumption for any E S.y E R, E A, and t (1,...,7 T), the following equations hold:
(s, -Ex.1
Σ
of (St+1, Y++1, Art1) x Rr
St = 8, Y. y. A,
(8, a) a) = Ex-" [ ge (St41 Yes1, =,= = y, A. = a],
(9)
and
[ Ex-1 [ (1 - -R EArxa( (81, A1) (x AI) x xn)
14

---

Published as a conference paper at ICLR 2026
(a) Two-way X, Pretrain X, iteration 2,000
(b) Two-way X, Fretrain X, iteration 12,000
(c) Two-way Protrain iteration 2,000
Two-way Pretrain X,
iteration 12,000
K201
23000
30000
(e) Two-way X, Pretrain O, iteration 2,000
(0) Two-way X, Pretrain O, iteration 12,000
2006
2000
4
(g) Two-way O, Pretrain O, iteration 2,000
Two-way O. Pretrain O, iteration 12,000
Figure 4: The distinct policies learned by PCVaR agents over the course of Q-learning
Proof. The following results provide key identity that allows us to express the event indicator over cumulative returns terms of the predictive weight function xx:
Lemma 1. Given an augmented Markov policy kernel X and risk budget under Assumption we have
EX-e & Y.R. Y,, A.] EX-" X (S Yrti, Art1) R. Yo, As]
Vt (1...., and VT (I....,T)
15

---

Published as a conference paper at ICLR 2026
Proof.
(a) Ex"[[[Rer S Y. A.]
[X S S. Y, A.
[x 5 R. H,, Ar1
@ "' 1 Y/) H,+1, R, Hr, A,
EV (U 1:7 S Y2 - R (1 -
A.]
+17 S &Y++1 +1 1 R. H4, Ail E S ,+,, Ar+1) R2 H,, A.] Ex (S Y++1, Ar+1) R2 S, Yt, Ar]
E +1.
We proceed by conditioning the full history and applying the tower property of conditional ex- pectation.
(a) By the Markov property with respect to the augmented state space, the conditional expec- tation given (St. Y2) is equivalent to conditioning on the full history H,
(b) We apply the law of total expectation to decompose the expectation across time steps, conditioning first Hr+1.
(c) Since Hr+1 contains all rewards up to time we can isolate Rt over the conditional expectation.
We utilize the definition of g in the sixth step. Substituting this decomposition into the original expression yields the desired form.
Applying Lemma 1 which is (d), to the definition of f( we obtain:
f/ = 1x [T( S R.) = s, Yi y, =
EX."I & (,) = y, A, = a)
( BX Σ
Ru S²
8, Yv = y, A, a)
Furthermore, similar technique utilizes the representation for Ex-1 - (1- -
Ex-0 [ (1
RI-)7 + = Ex - -1)) x [[R1. I YıHl
EX." x T(R S Yi)] - Ex-"(n x T(R S Yi)
EX- n. AD) Elg A1)] x
(AIixi(lee.a)) [f A1) ( n. A1) 1)
Fot the last equation, we need the following lemma.
Lemma 2. Given an augmented Markov policy kernel x and risk budget TJ, under Assumption we huve
EX-(T(Rer S Y.)SS,, Y,, A, Ex." (Xx Y441, A41)) Y,, A.]
Vt (1.
16

---

Published as a conference paper at ICLR 2026
Proof.
Ex SYHS,, A.]
E & H.H
@ EX S H++1, A+t1l H1, Arl
"*"E**" S Y, R As+ill A Ar]
11 Y YY++)|/441, Y Ar- Ar] Ex""g (St+1, Y41, A++1) H,, A,]
[x (St+1. V11, 4++1) Y2, As]
We proceed by conditioning on the full history and applying the tower property of conditional ex- pectation.
(a) By the Markov property with respect to the augmented state space, the conditional expec- tation given (St, Y,) is equivalent to conditioning the full history H,
(b) We apply the law of total expectation to decompose the expectation across time steps, conditioning first on Hy+1.
We utilize the definition of gx in the sixth step. Substituting this decomposition into the original expression yields the desired form.
Applying Lemma which is (d), to the definition of gi" we obtain:
()) = -x.. ( RET S S: = =, = A, = aa y,
-FX- RET S Y)]S = =
(d)
Y 144 = s, Yr = A =
The first statement of Proposition D esstablishes cumulative formulation of the predictive tail value function. This result naturally motivates a recursive Bellman-style decomposition. We formalize it in the next theorem.
C.2 PROOF OF THEOREM
Building upon the temporal decomposition in Proposition [ we derive Bellman-type recursive relation for the predictive tail value function. This relation enables efficient policy evaluation and learning in dynamic settings.
Theorem (Bellman equation). Given an augmented Markov policy kernel under Assumption its predictive tail value function fx and predictive tail probability function gx satisfy
(1 y. a) = (.n.,SSe))))PP(aaa))a......
[fxx (S+41. - R4, A++1)
+ g (St+1. V Re, Act1) x Re).
(10)
for all E S. y TR, A, and & (1,...,
17

---

Published as a conference paper at ICLR 2026
Proof.
T
a)) = 18X." -Bx-
Σ
Ar+1) x Rr
St = 8, Y, = A; =a
T
= -xx.
Σ
g
Yrth x Rr
S = s, Y = A =
r=++1
Ex
1 Y Ac+1)
Rs
S. = s, Y y, Ar a
T
g 1 (S++1, Y++1, Ar+1) x R
S. = s, Y2 = y, A: =a
r=f+1
+ (1 R₂) gitz (S++1,
R₂, AL+1) RR]
Ex-"
f (S4+1, Rτ, A
S = s, Y,, y, A a
Rx) gi+ (St+ Ra, Act1) Rr
[5 1, Re, Ac+1)
g (S++1. Re, Act1) x Rs].
We begin with the temporal decomposition from Proposition 1 which expresses f/ as cumulative expectation of future rewards weighted by gx:
T Σ
(s, = 1X
X++1, Art1) R, S; = 8, Y2 = A. = a
We isolate the contribution of the first term Rs, and apply the law of total expectation with respect to the policy kernel X:
v.
f (S RG, A(+1) gi.n (S++1, Re, Ac-1) - Rd)
This completes the recursive Bellman-type equation for fx
Having established the Bellman recursion for the predictive tail value function, we now turn to the corresponding optimality conditions. Theorem ( characterizes the optimal value function and derives the form of the optimal policy via a greedy selection criterion.
C.3 PROOF OF THEOREM
With the recursive formulation established in Theorem we now turn the optimality condition. The following result characterizes the optimal value function and policy structure under CVaR ob- jective.
Theorem 2 (Bellman optimality equation). Define
by (s, y) (A.-xe(-1saa) (s, y, A:) ge (s. v.). x x), 07(s.) = sup (s.
xeX
Then, the following holds under Assumption
Let II(x) be the set of augmented Markov policies induced by kernel x across all values 17 E R. Then,
sup {q -V.R [R max {6r + 2x (x1,
weI((x)
nER
2. With respect to all non-anticipatiin policies.
sup p9- CVaR, [RrT]]
max {97 + nER
nelt
18

---

Published as a conference paper at ICLR 2026
3. xx o² if and onty if x is greedy with respect .f*.g").
Proof. For the first claim, We begin by proving the first claim. For any fixed policy kernel X, consider the class of aug- mented Markov policies I1(x) it induces. Then,
( x--ax + [- (n Rur) yER { reIl(x)
sup {q- CVaR, [R1l]] rell(x)
(aaR = sup
(a)
max
max(ng n7 + [x B1:T)*}} nER
(6)
max ma4 + B AAmmx((emmm) (8 A1) (8 A1) x n])
nER
max of (81,11))
nER
where step (a) holds because the supremum over и II(x) includes the freedom to choose 77, step (b) follows directly from Propostion and step (c) follows directly from the definition of u, us in Theorem 2
We now turn to the second claim. When optimizing over admissible non-anticipating policies IT,
can equivalently optimize over the space of all Markov kemels X:
sup {q- CVaR ( =ell
[Rir]]
(a)
sup xEX
sup xxIl(x)
{q CV-C [ [uTT
(0)
max {am + (3(( nER
sup
xEX
977 + sup {uj (s1, (s), XEx
max
VER
max {gn + + (81, nER
where step (a) uses the policy optimization result (Theorem 3.2 in (Buuerle & Ott 2011)) step (b) follows from the result already shown in the first claim of this theorem, and step (c) uses the definition of of (s, y) as the pointwise supremum over X X.
To prove the final statement, we make use of the optimality condition for value functions (Theorem 5.1 in (Wang al. (023)), which states that equality between ux and v" holds if and only the policy is greedy with respect to the action-value function wX, Specifically:
x u² + (aEAIxx(al) > 0} C arg max (8, Vt,
aeA
Using the identity (z (s) (s, the equivalence follows.
C.4 PROOF OF THEOREM
We now show that a policy improvement guarantee in the context of CVaR optimization over aug- mented state spaces. This forms the backbone of our Predictive CVaR Q-learning algorithms in risk-sensitive reinforcement learning settings.
Theorem (Policy improvement). Consider an augmented Markov policy kernel x along with its predictive tail value function fx and predictive tail probability function gx. Let X be the greedy kernel with respect to (fx. Then, under Assumption! 7
2x (s. y) vx (s,y), As E S.E. R.T .1.
Consequently,
sup CVaR (RI:)) xlI((x)
CVaR5(B1.7) *ell(x))
sup
(11)
for any q E (0,
19

---

Published as a conference paper at ICLR 2026
Proof.
(4)
sup meil(x)
CVaR (RET)
max {mm + of (81, 1)) nER
(6)
max { (7 + of (81, nER
(e)
sup =eIT(x)
CVaR
where step (a) uses I. of Theorem
2
step (b) follows from the definition of greedy kernel X-
20


Reviews:
[{"review_id": "0LoPJkPJYm", "reviewer": "Reviewer_FfEG", "summary": "This paper proposes a new method for RL with a Conditional Value-at-Risk objective. To address the challenges posed by nonlinearity and non-decomposability, the authors introduce predictive value/probability functions and develop a new RL algorithm based on them.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation and mathematical development are clearly presented. While the derivations seem fairly standard, they provide a clear motivation for the proposed algorithm.", "weaknesses": "The paper's contribution requires further justification.\n\n1. After formulating objective (1), one could straightforwardly apply an actor–critic approach to optimize $ \\mathbb{E}^{\\chi,\\eta}[-(\\eta - R_{1:T})^{+}]$ (e.g., REINFORCE). It is not evident that the proposed method is superior to such actor-critic methods. While it may be true that only trajectories with non-zero effective reward are informative for actor-critic, the same limitation appears to affect the proposed algorithm: since g models the tail probability, when only a small subset of trajectories has non-zero effective reward, the estimate of g is likely to be noisy.\n\n2. The experiments are limited to toy settings and do not include comparisons against actor-critic methods or stronger baselines. A more comprehensive comparison would also address the weakness noted above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1IXPVJ524N", "reviewer": "Reviewer_fM76", "summary": "The paper introduces Predictive CVaR Q-Learning (PCVaR-Q), which makes CVaR optimization TD-learnable by defining predictive tail value and predictive tail probability functions that satisfy a new Bellman recursion. This enables step-wise CVaR learning and supports a proven policy-improvement guarantee.", "questions": "1. How sensitive is learning stability to the sampling of risk levels during exploration?\n2. Can the predictive-tail recursion extend naturally to actor–critic or deep function-approximation settings?\n3. Does the algorithm handle non-stationary return distributions or stochastic environments robustly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear theoretical innovation: a Bellman-consistent, value-based formulation for CVaR.\n- Strong proofs and solid connection to policy improvement.\n- Well-written and conceptually clear.", "weaknesses": "- Limited empirical scope: Experiments are small-scale and tabular; results demonstrate feasibility but not scalability.\n- Comparison gaps: The paper could benchmark against more recent risk-sensitive or distributional RL algorithms (e.g., D4PG, IQN with tail weighting).\n- Exploration heuristic: The “risk-level exploration” scheme is sensible but empirically underexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ECQYGwS6CG", "reviewer": "Reviewer_9ktc", "summary": "This paper proposes a sample-efficient Q-learning algorithm (PCVaR-Q) to optimize the Conditional Value-at-Risk (CVaR) target. Its core contribution is two key innovations: First, the \"predictive tail value function\" is proposed, which constructs a novel recursive structure suitable for CVaR targets, similar to traditional risks. The neutral Bellman equation aims to solve the problem of noise strategy evaluation caused by the indecomposition of the target. Second, introduce a \"two-way exploration\" strategy, which explores the risk sensitivity of action space and intelligent bodies at the same time, so as to alleviate the \"blindness to success\" phenomenon.", "questions": "1. For the augmented state $(s, y, a)$, how is the \"table-based function approximator\" implemented? Considering that $y$ is a continuous variable, is discretization based on the grid $H$ used?\n2. Without using pre-training, to what extent can your proposed new Bellman equation (Eq. 5) and bidirectional exploration itself solve the issues of \"blindness\" and learning instability?\n\n\n---\nTypos\n\n1. Line 369, \"through risk-nuetral Q-learning.\": nuetral -> neutral.\n1. Line 441, \"experimental environment and and the distinct policies\": there are two \"and\" here.\n2. Line 472, \"a novel novel CVaR Q-learning framework\": double \"novel\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core innovations lie in the proposal of the predictive tail value function $f^\\chi$ and the predictive tail probability function $g^\\chi$ for policy kenel $\\chi$. Based on the newly defined tail functions, Theorem 1 & 2 gives the Bellman equation and the Bellman optimality equation. \n2. Combined with the newly developed tail value function and probability function, the proposed \"two-way randomized exploration\" approach explicitly solves the known problem of \"blindness to success\" in CVaR learning, and encourages intelligent bodies to explore strategies with different risk preferences. This is achieved by samplingys around the risk budget $\\eta$.\n3. The experimental results (Figures 2 and 3) strongly support the author's argument. Compared with the CVaR-Q baseline, PCVaR-Q shows higher stability and lower variance during the training process.", "weaknesses": "1. The entire theoretical framework (especially Theorem 1 & 2) depends on Assumption 1, which states that the distribution of the residual return $R_{t:T}$ has no probability mass. However, many standard reinforcement learning environments (including discrete rewards or deterministic rewards) would violate this assumption. Moreover, I found that the experiements environment considered (such as the. Sequential decision tree setting) clearly violates Assumption 1. The author did not discuss the impact of the violation of thies assumption.\n2. TThe pre-training step is crucial for success, as all models that did not undergo pre-training (Figure 4a-d) failed to learn the optimal path. This seems to weaken the argument that the algorithm has a robust exploration strategy and \"sample-efficient,\" as exploration seems to fail in cold-start case.\n3. Currently, the experiments in the main text (Section 5) have failed to clearly disentangle the contributions of the three main contributions—(1) the new Bellman equation, (2) pre-training, and (3) two-way random exploration—to the performance improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bOUmVPGrRH", "reviewer": "Reviewer_Dyty", "summary": "The paper addresses the challenges in risk-sensitive reinforcement learning (RL) using the Conditional Value-at-Risk (CVaR) objective, which focuses on optimizing the expected return in the worst-case quantile of the return distribution (e.g., for safety-critical applications like autonomous driving or finance). Standard CVaR RL methods are sample-inefficient due to two key issues: (1) noisy policy evaluation from treating CVaR as a non-decomposable, terminal objective, which delays learning signals and hinders temporal credit assignment; and (2) \"blindness to success,\" where the agent ignores high-return trajectories outside the risk tail, leading to premature convergence to overly conservative, suboptimal policies.", "questions": "Lines 183-184: can the authors expand on why the choice of $\\eta$ is arbitrary? This seems counterintuitive to me.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposed CVaR recursive structure based on predictive tail value functions and predictive tail probability functions is innovative and clever. \n2. The theoretical work done to support this recursive structure is sound (albeit with some typos; see weaknesses). \n3. The empirical results are encouraging.", "weaknesses": "Overall I would consider this paper to be sound work. However, I do have some concerns that would need to be addressed prior to publication. \n\nMy biggest concern is that the work performed does not validate the authors’ claims. In particular, the authors mention numerous times that the primary benefit of their proposed method is sample efficiency, yet there is no theoretical work done in support of this claim, and the empirical results provided do not provide any evidence of such claims. The closest measure of sample efficiency that can be made from the empirical results is in Figures 2-(b) and  3-(b), however in Figure 2-(b), none of the methods converge to the same solution so a proper evaluation of sample efficiency cannot be made. In Figure 3-(b), there is no statistical difference between the methods. \n\nMoreover, I am not convinced (i.e., it was not rigorously argued by the authors) that value-based CVaR methods have sample efficiency issues in the same way that policy gradient methods do. In particular, the claim that value-based methods lead to significant sample inefficiency in lines 165-166 is not proved by the authors, nor do they provide a citation to back up these claims. In fact, a key piece of theoretical work that would greatly enhance the paper is some results that show why the regular CVaR decomposition (Equation 2) is sample inefficient.\n\nAnother concern is that the empirical analysis lacks focus. In particular, in addition to the concerns mentioned above, the use of function approximation for the simple experiments considered in this paper seems unnecessary. I would argue that if the authors want to include the work related to function approximation, they need to provide a compelling experiment that makes proper use of it. Furthermore, it seems odd that for the simple experiments included in the paper, neither method can find the optimal solution. \n\nIt is also not clear whether the shown gain in performance is due to the risk-sensitivity exploration strategy. In particular, was the same exploration strategy used with the baseline algorithm as well? If so, then why even include it in the first place when making the comparison (i.e., would it not be a cleaner comparison without the exploration strategy)? If not, then this is not a proper evaluation of the proposed algorithm's performance.\n\nOverall, although I see a lot of merit in the work performed by the authors, the current draft of the paper makes it seem like a ‘forced’ adaptation of prior work done in the policy-gradient domain into the value-based domain, rather than a purposeful, adequately-motivated endeavor. \n\nAccordingly, in order to increase my score, the authors would need to: 1a) provide theoretical and/or empirical results that support their claims related to sample efficiency, or 1b) remove the claims of sample efficiency and find a more compelling narrative for the paper. The authors would also need to address my concerns related to the empirical analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a5SMQSX8DI", "reviewer": "Reviewer_wLib", "summary": "The authors propose a novel CVaR recursive structure based on predictive tail value functions and predictive tail probability functions. This recursive structure, along with a risk-sensitivity exploration strategy, leads to superior empirical performance when compared to the usual CVaR recursive structure.", "questions": "Lines 183-184: can the authors expand on why the choice of $\\eta$ is arbitrary? This seems counterintuitive to me.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The proposed CVaR recursive structure based on predictive tail value functions and predictive tail probability functions is innovative and clever. The theoretical work done to support this recursive structure is sound (albeit with some typos; see weaknesses). The empirical results are encouraging.", "weaknesses": "Overall, I would consider this paper to be sound work. However, I do have some concerns that would need to be addressed prior to publication. \n\nMy biggest concern is that the work performed does not validate the authors’ claims. In particular, the authors mention numerous times that the primary benefit of their proposed method is sample efficiency, yet there is no theoretical work done in support of this claim, and the empirical results do not provide any evidence of such claims. The closest measure of sample efficiency that can be made from the empirical results is in Figures 2/3b), however in Figure 2b), none of the methods converge to the same solution so a proper evaluation of sample efficiency cannot be made. In Figure 3b), there is no statistical difference between the methods. \n\nMoreover, I am not convinced (i.e., it was not rigorously argued by the authors) that value-based CVaR methods have the same sample efficiency issues that policy-gradient methods do. In particular, the claim that value-based methods lead to significant sample inefficiency in lines 165-166 is not proved by the authors, nor do they provide a citation to back up these claims. In fact, a key piece of theoretical work that would greatly enhance the paper is some result that shows why the regular CVaR decomposition (Equation 2) is sample inefficient.\n\nAnother concern is that the empirical analysis lacks focus. In particular, in addition to the concerns mentioned above, the use of function approximation for the simple experiments considered in this paper seems unnecessary. I would argue that if the authors want to include the work related to function approximation, they need to provide a compelling experiment that makes proper use of it. Furthermore, it seems odd that for the simple experiments included in the paper, that neither method can find the optimal solution. \n\nIt is also not clear whether the shown gain in performance is entirely due to the risk-sensitivity exploration strategy. In particular, was the same exploration strategy used with the baseline algorithm as well? If so, then why even include it in the first place when making the comparison (i.e., would it not be a cleaner comparison without the exploration strategy)? If not, then this is not a proper evaluation of the proposed algorithm's performance, and an ablation study would be needed.\n\nOverall, although I see a lot of merit in the work performed by the authors, the current draft of the paper makes it seem like a ‘forced’ adaptation of prior work done in the policy-gradient domain into the value-based domain, rather than a purposeful, adequately-motivated endeavour. \n\nAccordingly, in order to increase my score, the authors would need to: 1a) provide theoretical and/or empirical results that support their claims related to sample efficiency, or 1b) remove the claims of sample efficiency and find a more compelling narrative for the paper. 2) The authors would also need to address my concerns related to the empirical analysis.\n\n**Minor Comments:**\n- The introduction is unfocused and hard to read. In particular, the constant switching between policy-gradient and value-based methods is hard to follow. Overall, I do not see a reason to mention policy-gradient methods at all in this paper. \n- Line 31: I would argue that CVaR has a lot of tractability issues (which is why it is such a difficult objective to optimize) and that the primary reason that it is valued is because it is a coherent risk measure.\n- Lines 46-51: The discussion in this paragraph completely ignores the notion of dynamic risk measures, which would need to be mentioned to make a proper argument.\n- Section 2 would greatly benefit by having more citations related to the methods that the authors are building upon.\n- Appendix C needs equation numbers.\n- Lemma 2 is filled with several copy/paste errors from Lemma 1 (e.g. c) is not needed in this proof)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)