Model Risk-sensitive Offline Reinforcement Learning
Authors: Gwangpyo Yoo
Honguk Woo
Venue: ICLR 2025 Poster (2025)
Keywords: risk-sensitive offline reinforcement learning
reinforcement learning
offline reinforcement learning
risk
model risk
Primary area: reinforcement learning
TL;DR: We proposed a model risk-sensitive offline RL framework, devising the critic-ensemble criterion to capture model risk effectively. To ensure the precision of model risk calculation, we employed Fourier feature networks.

Abstract: Offline reinforcement learning (RL) is becoming critical in risk-sensitive areas such as finance and autonomous driving, where incorrect decisions can lead to substantial financial loss or compromised safety. However, traditional risk-sensitive offline RL methods often struggle with accurately assessing risk, with minor errors in the estimated return potentially causing significant inaccuracies of risk estimation. These challenges are intensified by distribution shifts inherent in offline RL. To mitigate these issues, we propose a model risk-sensitive offline RL framework designed to minimize the worst-case of risks across a set of plausible alternative scenarios rather than solely focusing on minimizing estimated risk. We present a critic-ensemble criterion method that identifies the plausible alternative scenarios without introducing additional hyperparameters. We also incorporate the learned Fourier feature framework and the IQN framework to address spectral bias in neural networks, which can otherwise lead to severe errors in calculating model risk. Our experiments in finance and self-driving scenarios demonstrate that the proposed framework significantly reduces risk, by $11.2\%$ to $18.5\%$, compared to the most outperforming risk-sensitive offline RL baseline, particularly in highly uncertain environments.

Published as a conference paper at ICLR 2025
MODEL RISK-SENSITIVE OFFLINE REINFORCEMENT LEARNING
Gwangpyo Yoo, Honguk Woo"
Department of Computer Science and Engineering, Sungkyunkwan University (necrocathy, @skku.ee
ABSTRACT
Offline reinforcement learning (RL) is becoming critical in risk-sensitive areas such finance and autonomous driving, where incorrect decisions can lead to substantial financial loss or compromised safety. However, traditional risk- sensitive offline RL methods often struggle with accurately assessing risk, with minor errors in the estimated return potentially causing significant inaccuracies of risk estimation. These challenges are intensified by distribution shifts inherent in offline RL. To mitigate these issues, we propose a model risk-sensitive offline RL framework designed minimize the worst-case of risks across set of plausible alternative scenarios rather than solely focusing on minimizing estimated risk. We present a critic-ensemble criterion method that identifies the plausible alternative scenarios without introducing additional hyperparameters. We also incorporate the learned Fourier feature framework and the IQN framework to address spectral bias in neural networks, which can otherwise lead severe errors in calculating model risk. Our experiments in finance and self-driving scenarios demonstrate that the proposed framework significantly reduces risk, by 11.2% to 18.5%, com- pared to the most outperforming risk-sensitive offline RL baseline, particularly highly uncertain environments.
INTRODUCTION
Many application domains of offline reinforcement learning (RL), such as finance (Zhang et al.
2020 Wang & Ku 2022h, self-driving (Bernhard et al. 2019 Seres et al. 2023), and healthcare (Lu
et al. 2020), are inherently risk-sensitive due to the potential for high costs or safety risks from in-
teractions between the agent and the environment deployment. In these scenarios, both premature
and well-trained agents can make unsafe decisions, prompting active research efforts to integrate
offline RL with risk-sensitive RL (Urpi al., 2021 Ma et al. 202T, Rigter et al. 2024p. These efforts aim to minimize risk instead of maximizing expected return in offline RL settings. Here, (aleatoric) risA"] defined as real number representing the severity of potential harm associated with the return distribution. The function that maps return distribution to its corresponding risk called risk measure. For example, the risk calculated by conditional value at risk (CV@ R) risk measure with confidence level a E [0, 1] is the negation of the average from the most pessimistic to the a-th quantile (Yoo et al. 2024
events, while underestimating frequent ones, the estimated risk becomes highly sensitive to changes in the underlying return distribution (Cont et al.) 2010, Kou et al., 2013 Embrechts al., 2015 Pesenti et al., 2016). Such errors lead to incorrect risk management, especially when there is a distribution shift between the training and deployment environments. This situation is common in offline RL, where the agent cannot receive real-time feedback from the environment.
As these risk measures designed to emphasize
In finance, the issue is addressed by minimizing the worst-case of risks across set of plausible alternative scenarios rather than directly minimizing the risk estimated by the model (Bernard et al.
2023 This worst-case is referred to model risk, which quantifies the worst-case consequence of
"Coresponning Author
Whenever the context clear, we will refer to aleatoric risk risk.

---

Published as a conference paper at ICLR 2025
Risk seisi-sensiivve Offline RL
Model Risk sensiivve Offline RL
Plausible Set (M (2%; (,),))))
Cittlo Estimtiinn
Crilic Estimation
Frob
Cyitic Estimvted Risk
Wort Case
Alt, Scepatios
Model Risk
qurvnn
Risk Sale
Rusk Sale
Rladd Sadee
Worst-case
Entio-Esiimatio
Quanile Level
Quanbll Level
Plousible Set
Altemative Scensation
(Mudials Sectacio)
Figure 1: Difference between risk and model risk measured by Cv@Ra. The negation of the shaded area represents the risk.
using incorrect models (Breuer & Csiszár 2016 Bernard et al. 2023 that rely solely on the estimated risk and are vulnerable to model errors, the model risk acknowledged the potential for inaccuracies in the model, enabling more robust and reliable risk-sensitive decision- making. Inspired by this, our work in risk-sensitive offline RL shifts the focus from conventional risk minimization model risk minimization. In doing so, we address the common issue of estimating return distribution statistics (mean and the standard deviation) in RL contexts, where the statistics are often unreliable or unknown. This contrasts with finance, where the mean and deviation are assumed to be known.
Unlike traditional approaches
Figure iillustrates the difference between traditional risk-sensitive offline RL approaches (left-side in the figure) and our model risk-sensitive offline RL (right-side in the figure). In the left-side, existing risk-sensitive offline RL approaches minimize the risk estimated by critic (green area). In the right-side, our model risk-sensitive RL minimizes risk (blue area) of the worst-case scenario (blue line) among the plausible alternative scenarios set (gray lines). By minimizing the model risk, the agent accounts for the possibility of being incorrect, aiming to achieve the best possible results despite potential errors.
We present model risk-sensitive offline RL framework that accounts for model errors, enhancing robustness in highly uncertain environments. To the best of our knowledge, this is the first approach that incorporates model risk into offline RL algorithms. Specifically, develop critic-ensemble criterion method that identifies plausible alternative scenarios, enabling the calculation of model risk without introducing additional hyperparameters. Achieving this requires accurate return distri- bution statistics, particularly the mean and deviation. To enhance their precision, which is essential for identifying plausible alternative scenarios, we adopt a Fourier feature network for quantile re- gression. Through case studies in finance and self-driving scenarios, we demonstrate that our model risk-sensitive offline RL framework more robust than other approaches, achieving an 11.2% to 18.5% reduction in risk. Notably, approach exhibits strong advantages in highly uncertain envi- ronments with large possibility of model errors.
Our contributions are as follows.
We present the model risk-sensitive offline framework that minimizes the model risk, en- abling more robust decision-making compared to conventional risk-sensitive offline RI. approaches.
We introduce the critic-ensemble criterion, which adjusts a set of alternative scenarios by expanding or contracting it based on the level of uncertainty, thereby enabling the compu- tation of model risk.
We devise the Fourier feature quantile regression network to precisely estimate the statistics of distributions, which are required to calculate the model risk.
Through several case studies, we demonstrate the robustness of our approach in highly uncertain environments.
RELATED WORK
Risk-sensitive RL. Traditional research on risk-sensitive RL is based on risk measures that can be computed without estimating whole return distributions (Howard & Matheson 1972| Sato et al.
2

---

Published as a conference paper at ICLR 2025
Mihatsch & Neuneier 2002 Tamar et all 2015 Chow et al, 2017). With the rise of dis- tributional RL, Dabney et al. (2018a) developed IQN framework, which enables the direct com- putation of risk via quantile regression, thereby unifying the risk-sensitive RL framework. Other approaches include the direct calculation of CV@ R policy gradient without calculating the return distribution (Tamar al. 2013 Chow et al.] 2017). entropy-based methods (Mihatsch & Neuneier 2002), dynamic CV@ RiDu et al. [2022] Lim & Malik, (022). and entropic value at risk (EV@R, Ni Lai (202)). Dynamic CV@R R recursively accounts for risk at each timestep and is related to IR2R dRggeer et al. 2024). EV@R is risk measure that is out of our scope, but it is closely related to CODAC (Ma et al,, 202T). Another interesting approach is integrating risk-sensitive RL with safe RL. For example, Kim & Oh (2022) integrate the risk-sensitive approach to the safe RL, con- straining the CV@R of costs instead of expected costs. Meanwhile, |Ying et al. (202T): Greenberg er al. (2022) constrain the risk using the safe RI. method, while maximizing return. In an online RI.
2001
setting, Jaimungal et al.
2022) have proposed a similar approach to ours. However, because their method is based on KDE and on-policy algorithm, therefore, it cannot be applied to offline RL.
Risk-sensitive Offline RL. Research in risk-sensitive offline RL relies on integrating IQN and of- fline RL frameworks. Urpí et al. (2021) first proposed a risk sensitive offline RL framework ORAAC by combining IQN and BCQ (Fujimoto et al. 2019p. ORAAC is hindered by excessive constraints on the behavior policy, limiting its capability to generate better trajectories. To address the limita- tion, CODAC (Ma et al.] [202T). which is based on COL. was proposed. Yet. CODAC inherits the issues from CQL. such as hyperparameter sensitivity (Tarasov al., 2024 Another approach, IR2R (Rigter et al. 2024), is a based RL method that estimates risk with worst-case pertur- bations of the transition. However, its main limitation is that only guaranteed when transition probabilities follow a normal distribution.
Model Risk. Our work built on the concept of model risk, which has been extensively studied in the field of mathematical finance (Bernard et al. 2023] Breuer & Csiszar 2016 Blanchet & Murthy Xu
2019 Glasserman
2014p. Focusing on the worst-case expectation among scenarios aligns with Breuer's approach to aleatoric risk using a coherent risk measure (Breuer & Csiszär 2016
Glasserman & Xul 2014p. However, we focus on the worst-case of aleatoric risk among alternative scenarios, following Bernard et al, (202)), which Icads to different formulation. This approach
allows comparison of value distributions with different support, which is crucial in offline RL, since
the worst-case distribution may not match the model's support (Bernard et al. 2023
3 BACKGROUND AND PROBLEM FORMULATION
3.1 BACKGROUND
Risk-sensitiv RL. We consider a Markov decision process (S,A,R,P,3) where S is a set of states, A is a set of actions, R: S x A (R) is an immediate reward which may be random, and P: S x A + (S)) is a transition probability. Here (XX) denotes a set of random variables whose supports subsets of X. Further, we assume that all random attributes' means and variances are finite. An optimal risk-sensitive policy is formulated as
T π² arg min Ho (Z" (s, where (s) E((ss. a1) 5, ag a.
t=0
Here (Z (s, a)) denotes the risk of policy и with respect to its return 7 Z*{ss, a1) and H3 denotes a risk measure given by the user or environment. As we are interested in spectral risk measures (Adam ct al. 2008). in Def. D. we use the quantile function of a random variable.
Definition
Koenker 2005) Let 7. be random variable. A quantile function of the random vari- 0, + R. is defined as the left inverse of cumulative distribution of Z. Specifically,
able Z. Fz
Fz'(p) == inf E R | S Fz(x)},
(2)
where Fz R + [0,1] is the cumulative distribution function of Z. For readability, we denote Fz (,88,0 a) as the quantile function of dependent random variable, Z/8, a, rather than F-1 P (:)-
This is also problem commonly seen in GAN-hased approaches. See Appendix for explanation of the relationship between COL and GAN.
3

---

Published as a conference paper at ICLR 2025
The spectral risk is the weighted integral of the quantile function.
Definition (Aaam et al. 2008 A function Ho + (RU ( is called spectral risk measure which has the following representation
Ho(2)
- Fz -=(2)=-
(7 -
F7 1 (Fa (u))du - -Bu (F (u))l, (3)
w:= Fa(p) = du=dFo (1))=(pp)d with p=F& (u)
where Φ [0,1] [0, oo] a weight function; right-continuuus, non-increasing with f d(p)dp
L.e, Φ is a density function for some random variable. Further, the value of this function given Z,
Hz(Z), ) called the risk of Z. Here L, a set of random variahles whose means are finite. The
right most term is followed by and inverse transform sampling (iileer al. 2010 Dabney et al. 2018d and Monte-Care integral.
Since é is non-increasing, the risk measure necessarily prioritizes worse outcomes, intended. The negation required in the definition, since we aim to minimize the risk. For example, when we want to optimize for the worst-case risk measure (i.e., is Dirac-delta function), we minimize the negation of the worst-case outcome, the risk.
To calculate the risk Ha 7 (s, a)), the quantile function, we use the following distributional Bellman equation (Bellemare et al. 2017 Dabney al. 2018b
the quantile function of (7, a). (,, 8,a) a). To learn
7 (St, 0e) dist R(se, a+) + 7Z (8++1,
(4)
where a arg m (2 d')). Specifically,
the quantile regression method esti-
mate the quantile of return (s, a). The quantile regression loss (Koenker 2005 Dabney et al. 20186) of the critic is
N
1
Lerit. (0)
p
160 0)l
(5)
NN
Asymmettric weighted
Lososs
where 0g is the distributional Bellman residual and is critic's parameter. The distributional Bellman residual is defined as a pairwise difference between the target quantile and the critic's quantile as
0i R(st, a:) + >1 (p:; 11 a, 0mr) Fz (pa St, or, 0),
(6)
where Pi Pj denote N, N' numbers of independent random variables which follow U00 and P denotes the parameter of target-critic.
Risk-sensitive Offfine RL. In offline settings, an agent is not allowed interact directly with the vironment and can only access a transition dataset DD = [(s, a;, ri, } Due to this constraint, naive RL approaches suffer from overestimation of (s, a) from distribution shifts (Fujimoto et al., 2019. Thus, an auxiliary regulation is commonly applied. The objective of offline RL is defined as
arg max E 2 (s, a)] constraint to D( <e,
where D(-II-) some divergence (or metric), aD is the behavior policy for the dataset D, and e is tolerance. Incorporating this regulation, the objective of risk-sensitive offline RL is extended from Eq. as
arg min Hg (Z (s, constraint to D(*l(TD) <
(8)
3.2 PROBLEM FORMULATION
The primary concern of risk-sensitive offline RL is that the error of computing Ho H2 (s, (s,a)) a)) can be significant, as the agent cannot receive direct feedback from the environment in offline settings. To address this, we instead compute the worst-case of risks among alternative scenarios (i.e., model risk) and focus on minimizing the model risk. The model risk is defined as the highest possible risk within a balball with mean i and standard deviation a. The random variables in the ball specify the plausible alternative scenarios.
4

---

Published as a conference paper at ICLR 2025
Step 1. Calculate
Step 2. Calculate g
Step 3. Calculate
max (td|2* (s.a*, (0)]
Calculate Cnttic-nsseblle
Model Risk
s.g. trick(min (s2"(s,"", (s.a",00))
Cnbic Pstimation
Model Risk
200
Rutuns
Dera
+BC Loss
Eq(11)
Atoo
2 (8
Q.R. Loss; Eq. (5)
Caliic
2
Quarain mmel
Quantile- Level
Figure 2: Overall framework of MR-IQN. Q.R. loss is abbreviation of quantile regression loss and s.g. trick is stop-gradien trick. Note that the stop-gradient trick does not change the value.
Definition
Bernard et al. (2033) The model risk MR (ZR (s,a); is defined as MR (z, a); 14 (,, =
sup
Ho(X),
(Z
where
=x L? = = Dw:2(X, (s, 0)) < V and 1? is set of random variables whose mean and variance are finite. Here, Dw the Wasserstein 2-distance defined as Dw-2(X,Y) = V. Fx (p) The distribution of random variable Z (s, a) is called reference distribution.
Our goal is to find policy n² such that
arg min (R (s, a); 5, constraint to D(T((TD) <
(10)
MODEL
OFFLINE RL ALGORITHM
To achieve a model risk-sensitive offline RL agent satisfying objective in Eq. 10 we introduce Model Risk IQN (MR-IQN), which integrates the TD3+BC framework (Fujimoto & Gu 2021 with the model risk objective from Eq.4T0). The MR-IQN consists of distributional critics and deterministic actor. The critic is trained to minimize the loss function in Eq.(5). similar to other approaches (Urpí et al. 2021 Dabney et al. 2018a Kuznetsov et al. 2020 Specifically, the critics implemented using IQN (Dabney al.
2018a) and TQC (Kuznetsov al. 2020), following Yoo et al. (2024). Meanwhile the actor is trained to minimize the loss function,
[(T) leam MR (Z" + (a ap)
(11)
Model risk
BC loss
Here Ay.eam 0 is scale parameter, ap is batch action, and (s).
Figure describes the MR-IQN framework. We extract μ and a from the critic estimation. To precisely estimate the statistics, f and
leverage the Fourier feature architecture TTancik al. 2020) for the critics, as in Li & Pathak (202T). Next, we calculate as the distance between the ensemble of the critics obtained by quantile mixture of individual critics. Finally, we calculate
MR,(Z" (s, (ss))) and minimize the loss function in Eq. I1 To do so, we first introduce the theorem and corollary below for calculating model risk.
Theorem Bernard et al. 2023 Let Z be a random variable. For (Hz - 112 + (az - (0)
there exists unique quantile function fx [0, 1] I. which satisfies
MR (Z; = hxxo)olldddp
(12)
where uz is the mean of Z. oz the standard deviation of Z. We have
kx(p) = pb - (0(p) + >F (p) aA
(13)
bx
5

---

Published as a conference paper at ICLR 2025
Here U denotes the uniform random variable U U 1] and std denotes the standard deviation operator. If S (uz 2²² + (oz -o o)² holds, the problem is ill ill-posed or it has only trivial solution.
where X 2 0 is some constant, and a B(o(U) +
XFZ (td
Let co Forr(17 (U), If < (uz u²² + (oz - o)²² + 200z cc) holds, achieve
K A
Voz
4.7 22
(14)
0
0²0
K²
oz
where VVrl Co.z
Covl (U), ((U),, and K
((pz - pb2 0² + +02 E).
Otherwise, À = 0. Here, corr is the Pearson correlation.
Corollary Bernard et al. 2023) The solution of Eq. exists and unique whenever h2 exists as MR ( (8, (8,,));,,,,8 = a (orr ((U) + (U; a))) (15)
Sg (ppp))(p)dp Eq.
where A 2 O is the constant determined by Eq. in Theorem
4.1
CRITERION
To calculate the model risk for policy gradient using Eq. 15 it is required to determine μa, and tolerance e. For e, we want to set e large when there is a large inconsistency between an individual critic while setting small when the inconsistency is small. For conservatism, we choose μ as the minimum value among the critics' estimation and as the maximum to reduce overestimation bias as discussed by Fujimoto et al. (2018). From now on, we use K denote the number of critics.
Step 1. Calculating μl a. To ensure conservatism, we select Ho, as the smallest expectation and the largest deviation of 2", as follows.
n40
min "lZ"(s, (ss,,,(0)) )( i=I...,K
std[ (s, 0(0)
(16)
When À = 0, the gradient of the policy in 4 solely depends on the mean-deviation risk mea- sure, which often leads to local optima. To mitigate this, we apply the stop gradient trick to ja as
# = Ha( (Z" (s, a)) + stop-grad Hoo (s, a))) )))))
+ No:
(17)
where Hz(Z (s. max;= K (z (s, a, )()) i.e., the most conservative critic.
Step 2. Calculating through quantile mixture. We construct an ensemble of critics as single quantile function by mixing their quantiles and calculate the distance between this ensemble and individual critics.
We first form the critic ensemble by sorting the estimates from all crities and interpolating them over equally divided space (i.e., quantile mixture). Next, calculate the Wasserstein-2 distance between the ensemble of critics and individual critics. The largest of these distances is selected as s.
The first step involves calculating the sorted critic values yk defined as
(yI..... } = U Uح F (p;; 8, 0(0), ), 31 S S ... S yNK,
(18)
where N represents the number of random variables Pj (0,,1) Let X be equally divided set of [0, 1] defined as X = { = 0, x2 = 1/(NK - I).... ANK 1}. By interpolating pairs { yi..... (æNK. )) we construct the ensemble of critics, F.1. Figure depicts the procedure of calculating Fim.. Finally, e is computed by
L' (2, Fz (1; s, a, 260) = Div (18,, 0¹,, (.). (19)
E = max d= T.T.,..
We use the trapezoid method to numerically calculate the integral in Eq. 19
Step 3. Calculate the model risk. We take the most conservative critic and take the reference distribution of the critic estimates in the place of Fz in 1q. and Eq.01
ref (+) Fzz (13 8,8,,0 a, O(+)); where p(+) = arg max Ho 77 ( a, a, 0()))
(20)
g66)

---

Published as a conference paper at ICLR 2025
Algorithm 1 Critic-Ensemble Model Risk
Inputs: Critic Parameters g(+), Risk Measure's Density 6, the number samples.
Output: Model Risk MR, (Z" H,,E))
Step1 Calculate Jr, a: Sample Pj from (0, 1] with j = 1, ...., N. To estimate risk and mean by inv. trans. sampling. Get minimal mean over crities; mean over Pi-
N4 minj-1.... 1...k K Bp [ (p5; 8, 8, 0(
pJ - F/ (p²), Ho (2 (s, a)) - maxi= -Bp; 1 (p/z 5, a, (())))
D Eq.
<- - (7 (x,a)) + stop-grad HHl 77 (s,
+ No D Stop-Grradient Trick in Eq. 17
f ma..,=l..... ptd,, 1F (p). 8, a, 0(0)
Get maximal standard deviation over crities.
Step Calculate S:
{yr) 1 f- sort(U, U {F (pj s. a,
Sort
(x1 - 0, + NK-1. INK-1
NK- INK + 1}
NK-2
> I f- linspace(O, 1, K)) Rearrange and Interpolate
F + interpolation (Tx )
& max;=1..... K Lor Faom (x) 2 (z; [1 a, ((
Calculate via numerical integral.
Step 3 Calculate Model Risk:
01*) < arg maxgco Ho( (s, a, (())),
F-1 t Fzz (1,,,,,, & Take the most conservative. using Eq. 14
rel - calculate lambda (F P E, 6)
MR, (2* (8, a)) &- calculate model risk( risk(Prr) H4,00 8, Return: MR (2" (s)
using Eq. 15
Fyz (2s,,
(1s,
YuK
ens.
Retunn
(1s,0,000))
V/2
Quantile-Leve
Figure 3: Procedure of calculating Finl with three critics. Fo is a quantile mixture of critics, therefore, shows a mid-level pessimism similar to Fzz (,, 8, a, ((3)).
a solution does not exist, (i.e., < (uz 22 + (oz - )))), we substitute (z (s) for MR (2 (s,
preserve the gradient signal.
Algorithm outlines the procedure of computing model risk throughout the critic-ensemble crite- rion. First, the most conservative μl, a among critics, are chosen. Next, we compute by measuring the Dw.2 2 distance between each individual critic and the ensemble, which is obtained by quantile mixture via sort and rearrange. Finally, we compute the model risk using the parameters obtained from the previous steps.
4.2 ADDRESSING SPECTRAL BIAS OF QUANTILE REGRESSION
Although our framework accounts for model errors, it is essential to estimate the mean (μ)) and devia- tion (σ) as accurately as possible, since these statistics are assumed to approximate the ground truth. While quantile regression yields an unbiased estimator (Koenker 2005). due to spectral bias (Ra- haman el al. 2019), conventional neural networks often fail to capture these statistics effectively. Spectral bias refers the phenomenon where neural networks struggle learn high-frequency components of patterns. For risk-sensitive RL to be effective, 2 must converge in the distributional sense ensure proper signals can be back- propagated. According to the Lévy convergence theo- rem (Williams) 1991), this is equivalent to the pointwise convergence of the Fourier transform of 7 emphasizing the critical role of the high-frequency domain (and thus spectral bias) conver-
gence. The Fourier feature network (Tancik et al. 2020 the spectral bias problem.
Li & Pathak 2021 is known solution to

---

Published as a conference paper at ICLR 2025
Vanilla IQN
Level Fourier IQN
Full Fourier IQN
Quintis
-2-
-2-
Lavel
(a) Vanilla IQN
(b) Fourier feature in level domain (c) Full Fourier feature Dw;1: 2.33 + 0.14 Crossing( %): 0.01 0.00
Dw:1: 2.23 + 0.05
Dw:1: 0.06 + 0.02 Crossing( ()): 0.01 0.00
Crossing(%)) (): 6.07 +: 2.08
Figure 4: Results of quantile regression. Dw is the Wasserstein 1-distance between ground truth quantile, and Crossing is the ratio of the numbers of crossing quantiles (%)
Figure E demonstrates the importance of the Fourier feature network in quantile re- trained to estimate F=1 ()) given ~ No0, 1), log (O0,, 1). (a) is the result of network trained by Vanilla IQN. (b) is the case when we handle the spectral bias in the level of quantile. (c) shows the case when handle the spectral bias of all input do- mains. As shown, the spectral bias is the cause of inaccurate distribution estimation. Note that applying Fourier features in the level domain helps reduce the crossing quantile, but it does not improve overall accuracy. This is because the cosine embedding used in IQN already functions as type of Fourier feature net- work. Figure 5 depicts the implementation initialized from N (f)) and input Xo. the Fourier feature is
Q. Head
Output
gression. Here the networks
F.F. Layer
Feature Extractor
F.F. Layer
Concalonate
Parum
sin (2mXX f)
oos (2mX) X
R. F. Layer
Xot Inputs
s,a
Figure 5: Fourier feature quantile regression network.
is pairwise multiplication operator.
the Fourier feature network (c). For parameters
X1 sin(2 #) c)
(21)
where # denotes the concatenation operator, and X1 denotes the feature passed the next layer. We also leverage using the Fourier feature IQN instead of cosine embedding IQN to reduce crossing quantile error. The remaining process the same as conventional MLP.
EXPERIMENTS
5.1 BASELINES AND CRITERION
Risk Measures. For the experiments, we are interested in following risk measures.
(i) CV@R. considers average over below a's quantile. The risk measure's density d &vve parame- terized by a E [0, 1] of is
CveR (p) - 10..)a()))
1
(22)
(ii) Wang. is risk measure motivated by areas where one's cost becomes another's benefit, like an option trading Waan 2002). The density of Wang risk measure, &a wang parameterized by a < is
v(F (1 a
wang
(23)
((F ( p))
where
F-1 (((.1)) and &p the quantile and density of standard normal distribution, respectively.
"hhe input domain of quantile function; i.c., the space of p in Fz (p).
8

---

Published as a conference paper at ICLR 2025
Baseline Algorithms. ORAAC (Upi et al. 2021 is risk-sensitive variant of BCQ (Fjjimoto et al. 2019). It achieves the risk-sensitiv offline RL objective by only having slight modifications in the dataset's 's actions. CODAC (Ma et al., 2021 is risk-sensitive variant of CQL (Kumar et al. 2020). achieves a dataset constraint by minimizing -l-divergence between the RL agent's 's actions and the dataset's actions using the Donsker- Varadhan representation. IQN-TD3+BC is risk-sensitive variant of the TD3+BC (Fujimoto & Gul (202T) algorithm, where the IQN-Critic is used. The optimal actor TTD3-BC in IQN-TD3+BC is defined
TTD3+BC = arg min Aqlam (o + (a ap)?
(24)
Here Aq.eam > 0 a scale hyperparameter, ap is a batch action, same as in Eq.II
5.2 MAIN RESULTS
We evaluate MR-IQN on finance and self-driving scenarios comparing with the baselines above. The label Mean is the average of the mean score over seeds and - (Z*) is the mean negative risk over seeds. All reported scores are averaged across 5 seeds. Numerical descriptions provided without explicit criteria are the results of comparing the negative risk. We also present the D4RL results, comparing the baselines and IR2R, in Appendix A.1. The results and analysis about CV@ (V@R(1%% risks are in Appendix A.4, and the dataset details are provided in Appendix B.
Table 1: Performance comparison in finance scenarios.
Target Risk Measure
CV@R (50%)
Wang (-0.5)
Env.
Algorithm
Mean
- -Ha(Z")
Mean
-Ha(Z")
CODAC
1577.47 + 41.52
249.53 1: 28.33
1591.76 1: 8.12
892.34 1: 54.06
Forex
ORAAC
302.37 + 19.27
-99.90 0.01
304.89 = 38.05
70.47 = 17.22
Env.
IQN-TD3+BC
2103.16 + 103.43
454.16 + 11.76
2181.16 + 123.41
1307 52 +: 90.05
(High)
MR-IQN (ours)
2537.55 52 + 289.57
556.94 + 172.92
2681.40 + 114.00
1639.79 + 57.88
CODAC
37.67 7.54
13.98 2.34
33.43 1 7.01
21.89 1 4.65
Stock
ORAAC
78.48 + 3.27
26.07 + 2.25
78.57 + 6.09
50.97 + 3.98
Enn.
IQN-TD3+BC
80.49 + 1.87
30.92 1.15
80 + 3.05
55.55 + 1.76
(Low)
MR-IQN (ours)
74.78 2.10
31.16 + 1.02
87.03 2.62
58.56 1.61
Finance. To investigate the real-world scenarios, we conduct experiments using the trading log data ranging from 2023-Feb-3rd to 4023-December-1s. 1st. The trading environment is based the Meta- Trader Simulator (Amin 2021). We evaluate 1000 episodes for each seed to calculate CV@ R and Wang negative risk. For the details, see Appendix. Table depicts the results.
In the experiment, MR-IQN shows the highest performance in both Forex and Stock environments. In Forex environment, where uncertainty is high due high leverage (High), the model risk ap- proach effectively covers tail risk, maintaining high average returns and low risk, compared to the baseline approaches without model risk. IQN-TD3+BC, which is comparable MR-IQN shows degraded performance (18.5%-1) 11.2%) in Forex, indicating that MR-IQN is particularly useful in handling higher uncertainty and errors. Meanwhile, MR-IQN shows a slightly degraded average compared to ORAAC and IQN-TD3+BC in Stock environment. This indicates that there is a trade- off between conservatism and average performance.
Self-Driving. To simulate the self-driving scenario, we utilize the Airsim environment (Shah et al. 2017 In this scenario, RL agents are tasked with driving a quadcopter drone designated goal while avoiding obstacles, with wind disturbances affecting the drone's movement. The agents receive rewards based on the cosine of the angle between their movement direction and the goal. We evaluate performance in 100 episodes for each seed to measure negative risk. Table depicts the results.
MR-IQN outperforms other baselines, showing the highest success rate (87.4%-88.8%) and the lowest collision rate (10.6%-11.66 .6%). CODAC showed relatively low average of rewards and a higher success rate compared to ORAAC. This implies that CODAC fails to generalize efficient actions and hovers over. The high collision rate of ORAAC implies that it fails to avoid obstacles because of the striet constraint to the batch action.

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison in self-driving scenarios
Risk Measure Algorithm
Mean
-Z)
Success %)
Collision (%)
CODAC
43.56 89.45
-425.72 97.73
65.0 + 13.1
21.4 5.5
CV@R (50%)
ORAAC
92.37 + 40.51
08.37+4.22
23.0 15.7
68.0 + 10.2
IQN-TD3+BC
411.78 + 33.10
265.51
45.25
8.6
14.8 + 5.6
MR-IQN(ourS)
431.44 +: 30.41
290.96
= 46.95
88.0 4.6
11.6 + 4.9
CODAC
6.67 = 1.02
22.15 0.75
19.3 52.6
31.2 27.6
Wang (-0.5)
ORAAC
156.22 + 32.09
51.73
3.42
58.4
IQN-TD3+BC
412.98 + 26.68
311.20 3 31.03
83.8 5.1
14.4
MR-IQN(ours)
436.17 + 19.16
1.8
10.6 2.6
5.3
ABLATION STUDY
Table 3: Ablation results. v represents presence and X represents absence. M.R. and F.F. are the abbreviations for model risk and Fourier feature, respectively.
Components
CV@R (50%)
Wang (-0.5)
M.R.
E.E.
TQC
Mean
-Ho(2")
Mean
-Ha(2*)
V
2537.52
289.57
556.94
172.92
2681.40 114.00
1639.79
57.88
V
X
2003.37 + 214.76
164.84+37.00
1975.85
104.49
1160.65 87.82
X
V
2410.83 86.94
2324.31 + 146.95
1376.27 122.96
X
X
1763.21 278.16
278.16
1607.13 76.54
872.44
X
Y
V
2103. 16 + 103.43
11.76
2181.16 + 123.41
1307.52 90.05
X
V
X
1712.79 75.40
49.83
1650.28 + 51.48
984.79 30.00
X
X
V
2170.68 + 186.55
476.23 + 75.94
2188.57 + 134.40
1297.15 88.15
X
X
1454.30 + 76.87 X
153.84 + 38.57
1414.44 + 40.00
750.22 37.45
We also conduct ablation study in the Forex environment, identifying model risk, Fourier features, and TQC as key contributors to the model's improvement. Table presents the results. MR-IQN achieves 8.6%-19.8% performance gain from the Fourier feature and 19.8%-41.3% perfor- mance gain from TQC. The performance gain from model risk is reported Section 5.2 Notably, model risk enhances performance when combined with other factors, due the design of the critic ensemble criterion and the requirement for accurate estimation of and ar, as discussed in Sec- tion
Our framework achieves approximately 118.6%-262.0%% performance gains compared to the naive implementation, which omits the key components. The reason why TQC is introduced to our frame- work is to construct target quantile which is similar to F-1 As result, the calculated model risk tolerance, 8, becomes more aligned with our intention, i.e., large & for high inconsistency and vice versa. However, the performance improvement of TQC without other factors implies that TQC itself is not negligible factor.
CONCLUSION
We proposed model risk-sensitive offline RL framework, devising the critic-ensemble criterion to capture model risk effectively. To ensure the precision of model risk calculation, we employed Fourier feature networks, which accurately estimate both the mean and standard deviation deviation-ccitiocnn critical components for calculating the model risk. The framework ensures more reliable decision-making risk-sensitive applications by accounting for potential model errors and striving to make the best decisions despite them. While our framework is limited to spectral risk measures and cannot ac- commodate those outside this class, such as CMV (Vadori et al. 2020 or EV@A R (i & Lai 2022), the broad applicability of spectral risk measures covers many practical real-world problems. Our future work involves extending the framework to partially observable Markov decision processes, including embodied control and decision-making in mission-critical business applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning &
Evaluation (ITTP) grant funded by the Korea government (MSIT), ( RS-2022-11220043 (2022- 0-00043), Adaptive Personality for Intelligent Agents, RS-2022-1122144 (2022-0-01045), Self- directed multi-modal Intelligence for solving unknown, open domain problems, RS RS-2025-02218766. Accelerated Insight Reasoning via Continual Learning, and RS- RS-2019-I1110421,, Artificial Intelli- gence Graduate School Program (Sungkyunkwan University)), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00213 18), ITTP- ITRC (Information Technology Research Center) grant funded by the Korea government (MIST) (ITTP-2025-H) 111-2225--RS222225 RS-2024-00437633. 10%), IITP-ICT Creative Consilience Program grant funded by the 10%), and by Samsung Electronics.
Korea government (MSIT) LUTP-2025-RS-202-1110
REFERENCES
Alexandre Adam, Mohamed Houkari, and Jean-Paul Laurent. Spectral risk measures and portfolio selection. Journal of Banking & Finance, 32(0)::870-1188) 2008.
Haghpanah Mohammad Amin. gym-mtsim. https://itt.b.com.
Btc.ll///ymmmmme mtsim
2021. Accessed: 2024-03-28, version 1.1.0.
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In International conference on machine learning. pp. 449-458. PMLR, 2017.
Carole Bernard, Silvana M Pesenti, and Steven Vanduffel. Robust distortion risk measures. Mathe- matical Finance, 2023.
Julian Bernhard, Stefan Pollok, and Alois Knoll. Addressing inherent uncertainty: Risk-sensitive behavior generation for automated driving using distributional reinforcement learning. In 2019 IEEE Intelligent Vehicles Symposium (IV), pp.
IEEE, 2019.
Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2):565 6000 2019.
Thomas Breuer and Imre Csiszár, Measuring distribution model risk. Mathematical Finance, 26(2): 395-411, 2016.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained inforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18 (1):6070-66) -1270, 2017.
Rama Cont, Romain Deguest, and Giacomo Scandolo. Robustness and sensitivity analysis of risk measurement procedures. Quantitative finance, 10(6):593-600 2010.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. International conference on machine learning, pp. 1096- 1105. PMLR, 2018a
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. Proceedings of the AAAI conference on artificial intelligence, 2018b. 8b.
Yihan Du, Siwei Wang, and Longbo Huang. Provably efficient risk-sensitive reinforcement learning: Iterated evar and worst path. arXiv preprint arXivv2206.00267. 2022.
Paul Embrechts, Bin Wang, and Ruodu Wang. Aggregation--rousttess and model uncertainty of regulatory risk measures. Finance and Stochastics, 19:763-790, 2015.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34:20132-20155, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.

---

Published as a conference paper at ICLR 2025
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Paul Glasserman and Xingbo Xu. Robust risk measurement and model risk. Quantitative Finance, 14(1):29-58, 2014.
Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems, 35:32639-32652, 2022.
Ronald A Howard and James E Matheson. Risk-sensitive markov decision processes. Management science, 18(7):356-36) 1972.
Sebastian Jaimungal, Silvana M Pesenti, Ye Sheng Wang, and Hariom Tatsat. Robust risk-aware reinforcement learning. SIAM Journal on Financial Mathematics, 13(1):213-226, 2022.
Dohyeong Kim and Songhwai Oh. Tre: Trust region conditional value risk for safe reinforcement learning. IEEE Robotics and Automation Letters, (12::221122228, 2022.
Roger Koenker. Quantile regression, volume 38. Cambridge university press, 2005.
Steven Kou, Xianhua Peng, and Chris C Heyde. External risk measures and basel accords. Mathe- matics of Operations Research, 38(3):393 417, 2013.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179 179-1191, 2020.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pp. 5556-5566. PMLR, 2020.
Alexander Li and Deepak Pathak. Functional regularization for reinforcement learning via learned fourier features. Advances in Neural Information Processing Systems, 34:1 19496-9969955, 2021.
Shiau Hong Lim and Ilyas Malik. Distributional reinforcement learning for risk-sensitive policies. Advances in Neural Information Processing Systems, 35:30977-30989, 2022.
Ming Yu Lu, Zachary Shahn, Daby Sow, Finale Doshi-Velez, and Lehman Li-wei. Is deep rein- forcement learning ready for practical applications in healthcare? a sensitivity analysis of duel- ddqn for hemodynamic management in sepsis patients. In AMIA Annual Symposium Proceedings, volume 2020, pp. 773. American Medical Informatics Association, 2020.
Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforce- ment learning. Advances in Neural Information Processing Systems, 34:19235-1999 19247, 2021.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49: 267-290, 2002.
F.P. Miller, A.F. Vandome, and M.B. John. Inverse Transform Sampling. ing, 2010. ISBN 9786131753268. URL Stttps://bookee http:::://boossssggglee...o. www.go..oooooooooooooo.. www.ll..c..koooe.eee.eee bookksidd
VDM Publish-
OF AXWAACAAA
Xinyi Ni and Lifeng Lai. Risk-sensitive reinforcement learning via entropic-var optimization. In 2022 56th Asilomar Conference on Signals, Systems, and Computers, pp. 953-959. IEEF, 2022.
Silvana M Pesenti, Pietro Millossovich, and Andreas Tsanakas. Robustness regions for measures of 2016.
risk aggregation. Dependence Modeling. 4( 4():000011151520
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 5301-5310 PMLR, 09-15 Jun 2019. URL | /ttpt:::............... http:::://ppoceeinnee ml www.pres///eo wwt.s///a////aaaaaannaa 9a. html
12

---

Published as a conference paper at ICLR 2025
Mare Rigter, Bruno Lacerda, and Nick Hawes. One risk to rule them all: A risk- perspec- tive on prodel-based offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Makoto Sato, Hajime Kimura, and Shibenobu Kobayashi. Td algorithm for the variance of return and mean-variance reinforcement learning. Transactions of the Japanese Society for Artificial Intelligence, 16(3):353-32,, 2001.
Peter Seres, Cheng Liu, and Erik-Jan van Kampen. Risk-sensitive distributional reinforcement learn- ing for flight control. PACCPPpprsOnniiee 56(2):2013-2118, 2023.
Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles, In Field and Service Robotics, 2017. URL https:
05066
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the evar via sampling. In Association for the Advancement of Artificial Intelligence, 2015.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Núria Armengol Urpi, Sebastian Curi, and Andreas Krause, Risk-averse offline reinforcement learn- ing. arXiv preprint 102.v:2102.05371, 2021.
Nelson Vadori, Sumitra Ganesh, Prashant Reddy, and Manuela Veloso. Risk-sensitive reinforce- ment learning: martingale approach to reward uncertainty. In Proceedings of the First ACM International Conference on in Finance, pp. 1 1-9, 2020.
Mingfu Wang and Hyejin Ku. Risk-sensitive policies for portfolio management. Expert Systems with Applications, 198: 16807, 2022.
Shaun S. Wang. A universal framework for pricing financial and insurance risks. ASTIN Bulletin, 32(2):213-25 2344, 2002. doi: 10.2143/AST..322.10222
David Williams. Probability with martingales. Cambridge university press, 1991.
Chengyang Ying. Xinning Zhou, Dong Yan, and Jun Zhu. Towards safe reinforcement learning via constraining conditional value at risk. In ICML 2021 Workshop on Adversarial Machine Learning, 2021.
Gwangpyo Yoo, Jinwoo Park, and Honguk Woo. Risk-conditioned reinforcement learning: gen- eralized approach for adapting to varying risk measures. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 16513-1652) 2024.
Yifan Zhang, Peilin Zhao, Qingyao Wu, Bin Li, Junzhou Huang, and Mingkui Tan. Cost-sensitive portfolio selection via deep reinforcement learning. IEEE Transactions Knowledge and Data Engineering, 34(1):236-22 22862,2422286 2020.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A ADDITIONAL EXPERIMENTS
A.1 D4RL
Since D4RL is standard benchmark for offline RL training, report the D4RL scores here. We trained MR-IQN using the Wang risk measure with a parameter of -0.5. MR-IQN shows compa- rable performance (68.59) to the state of the risk-sensitive offline RL algorithm IR2R (67.35- 67.78) in overall scores and outperforms it when evaluating the average without random dataset 44), which is metric often used in behavior clone based offline RL algorithms (Wang al. 2023
u et al. 2023).
Table A: D4RL Mujoco-v2 v2 results (normalized score).
Environment
MR-IQN 0.5
IR2Rwang
IR2ReveR
ORAAC
CODAC
TD3+BC
Medium Random
HalfCheetah
20.48 + 0.91
35.5
36.9
22.4
31.0
11.0
Hopper
26.47 + 10.6
31.8
30.9
10.9
9.2
8.5
Walker2d
7.15 + 1.43
8.1
7.6
2.4
12.8
1.6
HalfCheetah
61.14 0.75
75.5
38.5
62.8
Hopper
98.84 =
61.6
80.2
29.6
63.9
59.3
Walker2d
89.05 = 1.24
83.4
63.9
45.5
84.0
83.7
Medium Medium
Expert Replay
HalfCheetah
46.36 2.19
65.6
65.7
39.3
53.4
44.6
Hopper
95.08 = 7.14
93.2
92.9
22.6
68.8
60.94
Walker2d
81.07 13.93
88.4
11.1
73.9
81.8
HalfCheetah
78.54 + 12.00
94.5
96.0
24.4
76.7
90.7
Hopper
105.62 3.60
89.6
81.6
4.1
87.6
98.0
Walker2d
113.25 25 + 0.36
78.1
90.9
42.8
112.0
110.1
Average
68.59
67.35
67.78
31.83
61.34
58. 58.86
Average without Random
55.44
81.43
81.99
36.08
75.90
76.13
A.2 BEHAVIOR ANALYSIS
(a) CODAC High
(b) SONTDO+BC High
(c) ORAAC High
(d) MR-IQN (ours) High
(0) CODAC Low
(1) IQN TD34BLLOW
(g) ORAAC Low
(h) MR-IQN (ours) Low
Figure Risky point mass environment. In the red zone, the agent gets a random reward. Top (a-d) figures depict the behavior of the policies when the variance of reward is high. Bottom (c-h) figures depict the behavior of the policies when the variance of reward low. The blue lines are the trajectories of policies.

---

Published as a conference paper at ICLR 2025
To analyze the behavior of risk-sensitive offline RL algorithms,, we compare the algorithms in modified risky point mass environment (Ma et al. 2021). The agent is trained to reach the goal (green points in Figure A In this environment, there is a red zone, where the agent gets normal white noise the reward signal in 0.1 probability. We compare two situations: when the variance of the noise signal is high (100) and low (5). Figure depictt the results.
In this environment, MR-IQN (ours) shows the most risk-averse behavior, avoiding the red zone as much as possible, desired. IQN TD3+BC sometimes fails avoid the red zone, especially when the variance of the reward signal is low. This difference highlights the conservatism of model risk- sensitive offline RL compared to conventional sensiivne offline RL. In contrast, both CODAC and ORAAC consistently fail to avoid the red zone. CODAC's failure is due to the key assumption of strict monotonicity of quantile function (Ma et al. 202T) which does not hold this environ- ment, coupled with hyperparameter sensitivity inherited from CQL (Tarasov et al. 2024). ORAAC struggles because the strong constraints imposed by the behavior policy make difficult for the policy to stitch together the desired trajectory.
A.3 NUMBER OF CRITICS
2000
3000
Mean
1600
2500
Mean
(z)
1200
7
Num. Critics
Figure B: Effect of the number of crities (i.e., ensemble size). The environment Forex and the risk measure is Wang(-0.5).
We also investigate how the number of critics affects performance. Figure shows the results. As demonstrated in TQC Kuznetsov al. (2020). the performance improvement begins saturate between 5 and 7 critics for both the mean and risk performance. The observed performance degrada- tion between and 5 crities appears to an illusion caused by the presence of outliers, as suggested by the large gaps in the error bars.
A.4 TAIL SCORE ANALYSIS
Table B:
Performance in finance scenarios
Target Risk Measure
CODAC
ORAAC
IQN-TD3+BC
MR-IQN (ours)
Wang(-0.5) CV@ R(50%)
3.00 0.91
7.92 0.86
8.32 + 0.71
10.14 = 0.55
Stock Env.
2.71 1.51
7.85 0.82
8.33 + 0.67
10.06 0.53
CV@ R( (10%))
3.98 + 0.86
7.91 + 0.74
8.68 + 0.43
10.35 + 0.52
Wang(-0.5)
-51.27 + 22.79
-96.59 L.04
-27.91 +: 48.02
59.11
Forex
CV@ R(50%)
-48.72 + 25.25
-100 + 0.00
11..47 28.97
25.71 + 32.09
Env.
CV@ R(10%)
4.35 1 2.06
44..54 20.94
0.25 + 34.37
31.40 + 36.02
We further investigated the CV@R(10% (negative) risk in financial scenarios. The results pre- sented in Table B In Stock environment, only CODAC demonstrates a significant improvement in CV@ ((10)) risk, whereas the performance of other algorithms appears have reached saturation.
2

---

Published as a conference paper at ICLR 2025
In Forex environment, all algorithms exhibit consistent behavior, with CV0)(10%) optimization yielding the highest CV@ ((1%) performance, followed by CV@R* R (50%) and Wang( -0.5) opti- mization criteria, respectively. except for ORRAC, which does not show meaningful CV@ R (00% performance in either CV@ R(50%)
or Wang( (00.5) risks.
Wang(-0.5)
F-1
100
Wang(-0.5)
(V0)(50%)
(V0R(50%)
C10)(10%)
CVOR(10%)
Neutral
Neutral
0.5
0.0
0.5
1.0
0.0
Level Domain
0.0
0.25
0.5
(a) Quantile of risk measures
(b) Projections of risk measures
Figure C: Visualizing risk measures
To examine this phenomenon, we investigate how risk measures are distinguishable. We leverage the Wasserstein-2 distance (Dw:2) to measure distinguishability. The Dw between CV@ R (50%) and CV@ EV@R(10%) approximately 0.23, and Dw:2 between Wang(-0.5) and CV@ ((10())%) mately 0.40. Therefore, CV@ R( (50%) is more similar risk measure with CV@ R(10%) compared to Wang(-O. 0-55). This similarity explains the consistent behavior algorithms. Figure C visualizes the risk measures. Figure C (a) describes the quantile function of risk measures, and Figure (b) describes the vector projection risk measures via hat function coefficient by finite element method approximation. We can verify that the previous argument.
approxi-
Note that spectral risk measure is defined as (Eq. B
H²(Z) -=( (p)d()dd)
The equation is governed by d, and e is probability density function defined on [0, 1]. Therefore, is natural to use Wasserstein distance to measure distinguishability. Mcanwhile, the model risk cal- culates the risk of some function whose Dw.2 distance between reference distribution is constrained by v as defined in Eq. 9 Therefore, it shows similar behavior to other algorithms.
B EXPERIMENT DETAILS
B. DETAILED EXPLANATION BASELINES
ORAAC policy Torme is as follows:
Woraa mb)(s) + NoTrisk (8, (b)()))
(A)
where The is behavior cloning policy implemented using VAE, Trick is the risk-sensitive perturbation
policy, and x 2 is the modification parameter. We set k = 0.25 following Urpí's base implemen- tation.
CODAC's critic objective is as follows
ZCODAC = arg min max Lerin (0)+
E
E. aanD
F (p5 0)] log E [exp(Fz (pr a, 0111 amm
(B)
00
10
KL-ppocnntal by Vonskor-Vrrdlan Ropresentation

---

Published as a conference paper at ICLR 2025
Table C: Performance of ORAAC and CODAC with/without observation normalization.
Target Risk Measure
CV@ R (50%)
Wang (-0.5)
Without Observation Normalization
Hnv.
Algorithm
Mcan
-Ha(2")
Mean
-Ha(27)
CODAC
-0.01 0.04
-0.03 0.09
3.63 8.12
-1.74 3.52
Forex
ORAAC
302.37 = 19.27
99.90 0.01
304.89 38
70.47 17.22
CODAC
6,67 + 1.02
-2.15 0.75
5.89 = 1.64
0.48 +: 1.40
Stock
ORAAC
68.6 + 13.0
24.0 -: 3,59
73.43 +: 3.43
47.68 1: 2.21
CODAC
4190.24 3073.29
7460.17 5227.62
6.67 1.02
-2.1
Airsim
ORAAC
-149.58 + 64.53
-31.80 217.92
68.6 13.0
24.0 +.0 3.59
With Observation Normalization
CODAC
1577.4 41.52
249.53 28.33
1591.76 8.12
892.34 + 54.06
Forex
ORAAC
0.00 + 0.00
0.00 0.00
0.00 = 0.00
0.00 0.00
CODAC
37.67 = 7.54
13.98 + 2.34
33.43 + 7.01
21.89 + 4.65
Stock
ORAAC
78.48 + 3.27
26.07 + 2.25
78.57 + 6.09
50.97 + 3.98
CODAC
43.56 89.45
-425.7 97.73
-2.40 + 17.59
300 23 109.30
Airsim
ORAAC
92.37 +: 40.51
88.37 4.20
156.22 32.09
51.73 27.53
Behavior Policy
Forex
191.42
-99.90
26.91
Stock
1.072
-11.67
- 66.5
Airsim
378.93
78.57
188.29
Here Cerit. is the quantile regression loss with respect to distributional Bellman residuals defined Eq. and À is constant or lagrangian coefficient. Note that the right most part is equal to learning f-GAN discriminator (Nowozin et al. 2016) with KL-divergence (Nguyen et al. 2010 Therefore, CODAC (and CQL) can be understood as GAN + RL.
This leads the optimal actor Teodae is defined
Toodac = arg min H (ZCOnAc) = arg min Ho(Z"(s. a)), constrained DKL (Toodac TD) < (C)
for some > 0 and behavior policy TD in Eq. See Eq. as well.
IR2R (Rigter et al. 2024) is model-based risk-sensitive offline RL algorithm based on the dual
formula for coheren risk measure as Eq. D
Ho(Z" (s, a)) inf Ez a)),
(D)
where T is subset of probability measure which is absolutely continuous with respect occupation measure of policy, p²
The constrains of T is dependent. For example, for CV@R, risk measure, } where denotes absolute continuity and dP døz denotes the Radon-
T = (IP E p
dP
Nikodym derivative between IP and pa.
To calculate risk based on Eq. D IR2R samples next states, {0/4 1/
ensemble model. Then, construct empirical quantile function based Ji=o given St, az, with Gaussian
Fī (pn s, a) R(st, ar) + +>
where V* (s() S V (5 (2)) -.. ... V (2)) and 0, N-1 =
According to the risk measured by approximated quantile function in Eq.EE. the synthetic data (8t, dt. R(s,a), 81+1) is rejected accepted; i.e., 8t+1 is resampled according é. Therefore, 1R2R minimizes dynamic risk (time-dependent risk) instead of static risk iime-independent risk). However, IR2R requires terminal function given observation priori (Rigter et al. 2023). Therefore, we could not conduct the main experiments with IR2R.
'The set of all coherent risk measures includes the set of spectral risk measures; i.c., every spectral risk
is coherent risk
4

---

Published as a conference paper at ICLR 2025
ORAAC and CODAC Tuning: To improve the performance of the baselines (ORAAC and CO- DAC), we also apply observation normalization. The following tables present baseline performance both with and without observation normalization, and in the main text, report the better score between them.
B.2 ENVIRONMENT DETAILS
Finance. The trading environment is implemented using MT-sim (Amin 2021 The data ranges from February 3rd to December 1st, 2023. In the Forex environment, the agent trades 'EURUSD' and "USDJPY" with leverage of 100, while in the Stock environment, it trades 'QQQ', 'SPY". IIWM', and 'TYY' with leverage of 10. The forex data is collected from MetaTrader SSajedi and the stock data from Yahoo Finance (Perlin 2023
2024
The environment terminates when the agent's assets drop to $O. To increase randomness, trades initiated at a random interval, and the agent aims to maximize return within this interval. Ob- servations include balance, equity, margin, orders, and features. The features consist of 15-day accumulated price and volume data, including open, close, low, and high prices. For numerical stability, the data is preprocessed using the arcsinh function, achieving logarithmic scaling. Action represents logits for buying, selling, or holding each symbol. The reward is the benefit ratio relative to the initial balance, fixed at $50, 000.
Self-driving. The self-driving environment is implemented using Airsim (Shah al. 2017 The agent's s goal is to pilot quadcopter drone to the target while avoiding static obstacles with height constraints. Random walk noise distorts the agent's actions to simulate wind conditions, introducing randomness. The environment ends when the drone collides with object or reaches the goal.
The observations provided to the agent include lidar data, previous wind conditions, kinematic in- formation (c.g.. orientation, linear velocity), and the goal position. The lidar data has fixed array shape as it is projected onto a grid in the spherical coordinate system. For each grid cell, the mini- mal value of radii is taken, capturing the closest detected object in that region. The agent's actions represent the pulse width modification (PWM) values of the motors which corresponds to the RPM of motors. The reward is determined by the cosine similarity between the drone's linear veloc- ity and the goal position. Further, the agent gets 100 reward if it reaches the goal and -100 if collides. Further,the agent receives alive reward of 0.01 when the agent does collide with any- thing. To avoid trivial solution, the agent receives height constrain reward when if flies to high; max(current height- height constrain, 0).
(a) Overview of map
(b) Example view of Airsim
Figure D: Airsim environment
B.3 HYPERPARAMETER DETAILS
Table D lists up the detailed hyperparameters of the experiments.
5

---

Published as a conference paper at ICLR 2025
Table D: Hyperparameter settings
Hyperparameter (Common)
Finance
Self-Driving
D4RL
learning
10-4
optimizer
Adam (/)
0.9, 82
0.999)
discount factor (r)
0.99
0.96
0.99
batch size
256
layer normalization
True
num. crities
quantiles for critic training
16
soft update ratio (r)
10-3 3
IQN num. cosine (Fourier feature dim for p)
64
IQN embedding dim.
64
Model
MR(ours) & IQN-TD3+BC
q-learning scale (Ay, learn)
2.5
1.0
5.0
Fourier feature init. (a)))
10-3
Fourier feature dimensions for a,
250
TQC drop out per critic
3
num. quantiles for actor train (N)
100
policy delay
2
Model
ORAAC
num. actions for inference
100
action distortion scale (N:
0.25
VAE Kl-ddivereence coef. (8)
0.5
Model
CODAC
KL--agramgian
True
KL-Lagramgian target entropy coef
10
0.2
C MINOR COMMENTS ABOUT TERMINOLOGY
In the actuarial science literature, conditional value at risk (CV@ R), tail value at risk (TV@ R), ex- pected shortfall (ES), and conditional tail expectation (CTE) are more strictly distinguished. How- ever, in the main text, we use CV@ R to mean TV@ R, following the ML literature. These terms must be distinguished when the quantile function of random variable is discontinuous at the confidence level. Because the worst-case quantile function for calculating model risk might also be discontinu- at that level, acknowledge that usage actually refers to TV@ R rather than CV@ R. though we keep using CV@ R for consistency with previous ML. literature. For details on their differences, Zhou et 2015

---

Published as a conference paper at ICLR 2025
REFERENCES
Haghpanah Mohammad Amin. gym-mtsim. bttps:://gituue com Amin//minniii gym mt s iim. 2021. Accessed: 2024-03-28, version 1.1.0.
Arseníi Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pp. 5556-5566. PMLR, 2020.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guides diffusion sampling in offline reinforcement learning. In Inter- national Conference on Machine Learning, pp. 22825-22855. PMLR, 2023.
Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforce- ment learning. Advances in neural information processing systems, 34:19235 19247, 2021.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56((1):5547-533 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in Neural Information Processing Systems, 29, 2016.
Marcelo Perlin. Yahoo finance, 2023. URL http:s///gghtb....c/rrrre/////// package version 0.2.38.
Python
Mare Rigter, Bruno Lacerda, and Nick Hawes. One risk to rule them all: official code, 2023. URL www.commmm////ee marc- riqter/1R2R
Mare Rigter, Bruno Lacerda, and Nick Hawes. One risk rule them all: risk-sensitive perspec- tive on model-based offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Reza Sajedi. Meta trader5, 2024.
URL
trading--lattaon Python package version 5.0.4200.
Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics, 2017. URL https:
harxiv.orr//bs//7055. 05065
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. International Conference on Learning Representations, 2023.
Xianhua Zhou, Huadong Zhang, and Qingquan Fan. Optimal limited stop-loss reinsurance under var, tvar, and cte risk measures. Mathematical Problems in Engineering, 2015(1):143739, 2015.


Reviews:
[{"review_id": "qCnR2VjgZF", "reviewer": "Reviewer_DSTd", "summary": "This paper addresses the problem of model risk in offline reinforcement learning, which arises from distributional shifts in the learning dataset. The authors provide a critic ensemble framework, integrating well-researched predecessor methods into a critic ensemble for risk estimation that can be applied regardless of the transition distribution or differences in distributional support. Experimental results in two domains demonstrate the superiority of the method and highlight the tradeoffs between robustness and high performance.", "questions": "1) It is mentioned that when $\\epsilon \\leq (\\mu_z-\\mu)^2 + (\\sigma_z - \\sigma)^2$ holds, the quantile function doesn't have a solution. Under what distribution conditions does this occur, and how often is it observed? It would be useful to know if the strictly-less-than case occurs with relative infrequency, or very often.\n\n2) Baselines ORAAC, CODAC, and 1R2R are mentioned in the introduction and related works, but only the first two are used as comparison baselines. Is there a reason the most recent method was excluded?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The motivation is clear; the problem statement, issues with existing works, and connections to the provided solution are clean and easy to follow. I also appreciate the label brackets underneath parts of the equations. \n\nThe contribution addresses the issue both distributional shifts in data and model inaccuracies, which are critical challenges in offline RL.", "weaknesses": "Most of the formalizations are cited directly from previous works, with the exception of 4.1. Given that 4.1 is only half of the proposed methods section, this makes the contribution seem more incremental. Are there changes or improvements that can be made to any of these components to better suit them to the domain? As an example, the Bellamn equation in Eq. 4 could be extended to measures such as minimum reward, expected value (over actions), or regret.\n\nThere are a few minor spelling errors, like 'qunatile' in the Fig. 4 caption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C6TCNZojJX", "reviewer": "Reviewer_AhNq", "summary": "This work introduces a framework and learning algorithm, MR-IQN, for offline reinforcement learning in a risk-sensitive context. The MR-IQN algorithm aims to identify a policy that minimizes model risk while remaining close to the behavior policy derived from the dataset. The authors evaluate the algorithm on both finance and self-driving datasets, demonstrating that it outperforms baseline methods in terms of model risk measures and average return.", "questions": "1. I appreciate the illustrative explanation of the calculation for $ F_{ens.}^{-1} $; however, is there a mathematical formula for this calculation? Additionally, were gradients preserved when calculating $ F_{ens.}^{-1} $ during experiments?\n2. In the experiments, how was the offline dataset generated?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Novelty:** To the best of my knowledge, this is a novel formulation for risk-sensitive offline RL, with no similar approach appearing in current literature. Additionally, the use of a critic ensemble and a Fourier feature network for variance and bias reduction is a thoughtful design choice.\n2. **Clarity:** The paper is well-organized and self-contained, providing a clear presentation of the problem, algorithmic approach, and experimental setup.\n3. **Comprehensive Experiments:** The evaluation on finance and self-driving applications aligns well with the risk-sensitive focus of the paper. Extensive experiments, including a range of baselines, lend further credibility to the results.", "weaknesses": "**Major Issues:**\n\n1. The formulation of risk-sensitive offline RL optimizes model risk regularized by the dataset's behavior policy. It is unclear how this formulation guarantees that the optimality considers both model risk and expected return, especially when the behavior policy in the dataset may not be optimal, as is often the case in offline RL.\n2. There is a lack of theoretical insight into the algorithm. Specifically, it is not evident why the proposed loss function in Equation (11) adequately solves the problem defined in Equation (10). Additional theoretical justification would enhance understanding of the algorithm's effectiveness.\n\n**Suggested Improvements:**\n\n1. In the experimental section, an ablation study on ensemble size (i.e., number of critics) would help substantiate the value of using an ensemble for performance gains.\n2. The authors choose $\\mu$ as the minimum value among critics' estimations and $\\sigma$ as the maximum, aiming for conservatism. It would strengthen the argument if they defined conservatism and justified the choices of $\\mu$ and $\\sigma$.\n\n**Minor Issues:**\n\nThe following suggestions may improve clarity:\n\n1. In Equation (3), the form $ F^{-1}_{\\phi}(U[0, 1]) $ is not clearly defined by Definition 1.\n2. Is $ Z_{\\pi} $ intended to mean the same as $ Z^{\\pi} $? If so, consistent notation is recommended.\n3. In lines 275-276, the notation $ E_{p_j} $ appears undefined or unclear.\n4. In Figure 4, the x-axis and y-axis require explicit labeling for clarity.\n5. In Tables 1 and 2, the meaning of \"Average\" should be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pzzAJJLxFz", "reviewer": "Reviewer_mn9s", "summary": "This paper introduces MR-IQN, a model-risk-sensitive offline RL algorithm, which aims to minimize risk in worst-case scenarios instead of conventional risk. Building on TD3+BC, it incorporates a critic ensemble to estimate model risk and employs a Fourier feature quantile regression network to mitigate spectral bias, allowing for precise distributional statistics estimation. MR-IQN outperforms baseline algorithms CODAC and ORAAC in two finance environments, Forex and Stock, as well as in the Airsim environment.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The approach of minimizing model risk in offline RL using a critic ensemble, rather than focusing on conventional risk, is an interesting and novel idea.\n2. MR-IQN demonstrates strong results in the presented environments, indicating promising potential.", "weaknesses": "1. The experimental section feels underdeveloped, as MR-IQN and the baselines are only tested in three environments. Expanding the evaluation to include a broader set of benchmarks, such as those in D4RL (as done in CODAC and ORAAC), would significantly strengthen the results.\n2. The choice of metrics in Table 1 raises some concerns. Evaluating performance at the 50% quantile is unusual, as this metric doesn’t emphasize rare events, which seems at odds with the motivation of risk-sensitive RL. To the best of my knowledge, previous literature typically uses the 10% quantile. Could you clarify why CV@R(50%) was selected for this study?\n3. The explanation of model risk is somewhat unclear, and Figure 1 is particularly challenging to interpret. A more detailed and structured presentation of model risk concepts would improve readability and understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zySFyxwobL", "reviewer": "Reviewer_3isb", "summary": "This work considers the risk-sensitive setting in offline RL, which is important for real-world applications. This work proposes to minimize the worst-case risks across a set of plausible alternative scenarios rather than solely focusing on minimizing estimated risk. The proposed method, MR-IQN, introduces a critic-ensemble criterion method, the learned Fourier feature framework, and the IQN framework to address spectral bias in neural networks. Extensive experiments show that MR-IQN can significantly reduce the different kinds of risk, like CVaR and Wang risk.", "questions": "- Why can we translate the objective (8) to the objective (10)?\n\n- In step 1, why does MR-IQN choose the **smallest** expectation and the **largest** deviation?\n\n- Why do Fourier features benefit a lot, are there any explanations?\n\n- Are there ablation studies about the number of ensembled critics?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Considering spectral risk measure is interesting in risk-sensitive RL as it seems a general form of several risk metrics like CVaR, worst case, and Wang risk.\n\n- Extensive ablation studies show the importance of each component in MR-IQN.\n\n- Fig. 4 explains the advantage of Fourier feature clearly.", "weaknesses": "- It seems that the algorithm is a direct application of previous theoretical results (Bernard et al., 2023) as the major theoretical results (Thm 1, Cor 1) are from (Bernard et al., 2023). What is the major challenge in applying these theoretical analyses to your methods?\n\n- There seems to be a gap between the objective 10 (can be simplified by Cor. 1) and the proposed algorithm. Why can MR-IQN handle objective 10? As objective 10 is a constrained optimization problem and hard to solve directly, do you use bi-level optimization, or use the Lagrange multiplier method, or some other methods? It will make the work more solid by providing a more detailed explanation of how MR-IQN addresses the constrained optimization problem in objective 10, including any specific optimization techniques they use.\n\n- Lack of some related work on several risk-sensitive RL methods including different risk metrics which are closely related to this work, like CVaR [1-4], EaR [5], worst case [6], and so on.\n\nReference:\n\n[1] TRC: Trust region conditional value at risk for safe reinforcement learning\n\n[2] Towards safe reinforcement learning via constraining conditional value-at-risk\n\n[3] Distributional reinforcement learning for risk-sensitive policies\n\n[4] Efficient risk-averse reinforcement learning\n\n[5] Risk-sensitive reinforcement learning via Entropic-VaR optimization\n\n[6] Provably efficient risk-sensitive reinforcement learning: Iterated cvar and worst path", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)