Robust Reinforcement Learning with Distributional Risk-averse formulation
Authors: Pierre Clavier
Stephanie Allassonniere
Erwan Le Pennec
Venue: Submitted to ICLR 2023 (2023)
Keywords: Robust Reinforcement Learning
Risk-Averse Reinforcement Learning
Deep Reinforcement Learning

Abstract: The purpose of robust reinforcement learning is to make predictions more robust to changes in the dynamics or rewards of the system. This problem is particularly important when dynamics and rewards of the environment are estimated from the data. However, without constraints, this problem is intractable. In this paper, we approximate the Robust Reinforcement Learning constrained with a $f$-divergence using an approximate Risk-Averse formulation. We show that the classical Reinforcement Learning formulation can be robustified using a standard deviation penalization of the objective. Two algorithms based on Distributional Reinforcement Learning, one for discrete and one for continuous action spaces, are proposed and tested in a classical Gym environment to demonstrate the robustness of the algorithms.

Under review as a conference paper at ICLR 2023
ROBUST REINFORCEMENT LEARNING WITH DISTRIBU- TIONAL RISK-AVERST FORMULATION
Anonymous authors Paper under double-blind review
ABSTRACT
The purpose of robust reinforcement learning is to make predictions more robust to changes in the dynamics or rewards of the system. This problem is partic- ularly important when dynamics and rewards of the environment are estimated from the data. However, without constraints, this problem intractable. In this paper, we approximate the Robust Reinforcement Learning constrained with a f-divergence using an approximate Risk-Averse formulation. We show that the classical Reinforcement Learning formulation can be robustified using standard deviation penalization of the objective. Two algorithms based on Distributional Reinforcement Learning, one for discrete and one for continuous action spaces, are proposed and tested in a classical Gym environment to demonstrate the robustness of the algorithms.
INTRODUCTION
The classical Reinforcement Learning (RI,,Sutton & Barto (2018) problem using Markov Decision Processes (MDPs) modelization gives practical framework to solve sequential decision problems under uncertainty of the environment. However, for real-world applications, the final chosen policy can sometimes be very sensitive to sampling errors, inaccuracy of the model parameters, and definition of the reward.
This problem motivates robust Reinforcement Learning, aiming to reduce such sensitivity by taking to account that the transition and/or reward function (P, r) may vary arbitrarily inside given uncertainty the solution that maximizes worst-case problem in this
set. The optimal solution can be uncertainty set or the result of a dynamic zero-sum game where the agent tries to find the best policy under the most adversarial environment (Abdullah et al. 2019). In general, this problem is NP-hard (Wiesemann et al., 2013) due to the complex max-min problem, making it challenging to solve in discrete state action space and scale to continuous state action space.
Many algorithms exist for the tabular case for Robust MDPs with Wasserstein constraints over dynamics and reward such Yang (2017): Petrik & Russel (2019); Grand-Clément & Kroer (2020a b) or for constranned S-rectangular Robust MDPs (Behzadian et al. [2021). Here we focus more general continuous state space S with discrete or continuous action space A and with constraints defined using f-divergence.
Robust RL Morimoto Doya 2005) with continuous action space focuses on robustness in the dynamics of the system (changes of P) and has been studied in AAbdulaah et al. (2019); (innhh et al.
(2020); Urpi et al. (202T): I): Eysenbach & Levine (202I) among others. Eysenbach & Levine (2021) tackles the problem of both reward and transition using Max Entropy RL, whereas the problem of robustness action noise perturbation is presented in Tessler et al. (2019). Here, we tackle the problem of robustness through dynamics of the system.. Recently, the issue of the Robust Q-Learning has also been addressed in Ertefaic et al. (202I).
In this paper, show that it is possible tackle a Robust Distributional Reinforcement Learning problem with f-divergence constraints by solving a risk-averse RL problem, using a formulation based on mean standard deviation optimization.
The idea beyond that relies on the argument from Robust Learning theory, stating that Robust Learning under an uncertainty set defined with f-divergence is asymptotically close to Mean- Variinn (Gotoh

---

Under review as a conference paper at ICLR 2023
et al. 2018 or 2018
deviation optimization (Duchi et al. 2016 Duchi & Namkoong
In this work, we focus the idea that generalization, regularization, and robustness are strongly linked in RL or MDPs as shown in Husain et al. 2021); Derman & Mannor (200)); Deemmnn et al. (2021): Ying al. [202T; Brekelmans al. (2022). We show that is it possible to improve the Robustness of RL algorithms with variance/standard deviation regularisation. Moreover, the problem
of uncertainty under the distribution of the environment transformed into problem with uncertainty over the distribution of the rewards, which makes it tractable.
Note that our work is related to Smirnova al. 2019) they penalise the expectation by the variance of returns. However, their approach differs from ours since they use the variance estimate under a Gaussian assumption of distributions while we use a standard deviation penalization without any distribution assumptions. Moreover, the idea of robustness in the change of dynamics is not demonstrated numerically, and the problem tackled is different since they consider close policy distributions, while we consider dynamic distributions.
The contribution of the work is the following: we motivate the use of standard deviation penalization and derive two algorithms for discrete and continuous action space that are robust to changes in dynamics. These algorithms only require one additional parameter tuning. which is the Mean-Standard Deviation trade-off. Moreover, we show that our formulation using Distributional Reinforcement Learning is robust to changing transition dynamics in environments with both discrete and continuous action spaces both in the Mujoco suite and in stochastic environments derived from Mujoco.
Related topics: Regularised MDPs: Policy Regularisation in RL Geist et al. (011 has been studied and led to sate-of-the-aae algorithms such as PPO and SAC (Schulman et al. 2017b Haarnoja et 20118) Vieillard et al. 2020). In these algorithms, an additional penalisation based on the current policy added to the classical objective function. The idea is different, as we penalize our mean objective function using the standard deviation of the return distribution. Being pessimistic about the distributional value function leads to more stable learning, reduces the variance, and, tends to improve the robustness of systems demonstrate (Brekelmans al.) (2022). Recent advances in Robust MDPs have shown a link between this field and Regularised MDPs as in Derman et al.
12021): Kumar al. (2022).
Distributional RL: Second-order estimation is done using Distributional Reinforcement Learning (Bellemare al., 2017 Zhang & Weng) using quantile estimate of our distribution approximate our action value function (Dabney et al. 2017: 2018) with the QRDQN and IQN algorithms. Distri- butional state-action function representation is also used to learn an accurate critic for a policy-based algorithm, such as in Kuznetsov et al. (2020); Ma et al. (202T); Nam et al. (202T).
Risk-Averse RL: Risk-averse RL aims minimizing different objectives than the classical mean optimization e.g. CVaR or other risk measures. For example, et al. (2018); Ma et al. (202T) use distributional RL for optimizing different risk measures. Our goal is show the robustness of using risk-averse solutions to our initial problem. Our formulation is close to mean-variance formulation (Jain et al. (202b] Wang Zhou (020) that already exists risk-averse RI., although not using distributional framework that shows highly competitive performance in controlled setting.
Pessimism and Optimism in Distributional RL Moskovitz al. (2021) describes a way of per- forming Optimistic/ Pessimistic Deep RL using a constructed confidence interval with the variance of rewards. Their work is close to ours in the pessimistic case but the confidence interval is expressed in terms of variance of expectation estimate and not using the variance of the distribution itself. Moreover, they use an adaptative regularizer where we look the interest of using a fixed parameter.
Taking into account a Markov Decision Process (MDP) (S, P.r), where S is the state space, A is the action space, P (r. s 8, a) is the reward and transition distribution from state to s' taking action and 7 (0, 1) is the discount factor. Stochastic policies are denoted π (a s) S + A(A) and we consider the cases of action space either discrete our continuous. A rollout trajectory using π from state using initial action is defined as the random sequence 7 (77(ssa) a) = ((5n, ag. ro) ($). (,.r)))))))... rr),...) with Sn = s, Op = a, 0g ~ π (. s,) and 8t+1) ((. St, a;); we denote the distribution rollouts by P(7) with P(())
Preliminaries:
2

---

Under review as a conference paper at ICLR 2023
Po (so) IT==0 P (s st, (.) x (at st) dt and generally write P (P, Moreover, consider the distribution of discounted cumulative return Zd (s, a) such that R(r a))
xxA A + of π the expected discounted Ei=o '"T; ~ Z " (s, a). Finally, the Q-function Q SS cumulative return of the distribution, defined as follows.
( === B|z (s) Ep [R(T) az ( st). (rz, S111) ~ - St, az) 800 s, ao = al
The classical initial goal of RL, also called risk neutral RL. is find the optimal policy where Q (s,a) 2 (s, a) for all and S. A. Finally, the Bellman operator 7 and the Bellman optimal operator T are defined as follows:
T"Q(s, a) r(8, BEp (s',
(=Q(s,a) =-(s,,))=r(s.)) +BBp max Q (s',
Applying either operator from an initial Q" converges to fixed point Q" or Q" at geometric rate as both operators are contractive. Simplifying the notation with regard to 8, a, π and P. we define the set of greedy policies w.r.t. Q called ((Q) = arg max R). A classical approach to estimating an optimal policy is known Approximate Modified Policy Iteration (AMPI) Scherrer et al. (2015)
{ Th+1 (Qu))) (T*K11)" ( Qk Ck+1
which usually reduces to Approximate Value Iteration (AVI, m = 1) and Approximate Policy Iteration (API, oo) as special cases. The term C&+1 accounts for errors made when applying the Bellman operator RL algorithms with stochastic approximation.
2 ROBUST FORMULATION USING X² DDVVERGEN CONSTRAINTS.
In this section, we would like find policy that is robust a change of environment law P. as small variations of P should not have big impact on the new policy in the greedy step. In our case we are looking the classical greedy step n' E G(Q) arg max IT), but rather the following greedy step:
xell
n' ((Q) = arg maxx(mjn ((P=)) ,)) SEl
This heuristic in the greedy step can also be interpreted at trying to avoid an overestimation of the Q functions present in the Deep RI. algorithms. Using this formulation, we need to constrain the set of admissible transitions from the state action to the next state to get a solution to the problem. In general, without constraint, the problem is NP-Hard, it therefore requires constraining the problem to specific distributions that are not too far from the original one using a distance between distributions such as the Wasserstein metrie (Abdullah et al. 2019) other specific distances if the problem can be simplified (Eysenbach & Levine 2021). Furthermore, if an explicit form of minp ((P.a)) could be calculated exactly for a given divergence, it would lead to a simplification of this max max-min min optimization problem into simple maximisation one. In fact, simplification of the problem is possible using a specific f-divergence denoted H, to constrain the problem with f a close convex function such that R IR U (+00) and f(z) 2 f(1) = 0 for all E R::
H, (Q P) = { Ein.o PaS (14
Exp 4: = 1,4 20
t00 otherwise.
with P, Q two probability measures. This constraint requires 0 if p; = 0 which makes the measure absolutely continuous with respect to The 2i-vieereence particular case of f- divergence with f(z) = (z 1=-11² 1)2. For trajectories sampled from the distribution Po and looking at distribution close to Po with regard to the x² -iivergence, the minimisation problem reduces to
min (P
Q(P.T) ((Po.m) aV(zPoj El a.
(1)
3

---

Under review as a conference paper at ICLR 2023
The proof can be found in Appendix A for a such that a <
V'2'0 f0
S 1 with 2P -B|
7%
the centered return distribution and V V[zP) the variance of returns. For a
| | the equality
V
becomes an inequality, but we still optimize lower bound of our initial problem. Defining new greedy step which is penalized by the standard deviation:
n' 9. (Q)
arg max TEll
min (
((Pix), ,
arg max (Q(Po.T) -a - #ell
we now consider the following scheme
{
Ah+1 E 9o (Qx)
(2)
1x=1=(TTT+1) Qk +
Approximation identities such as for larger class of f-divergences and not only x2 can be found in the work of (Duchi et al. (016)) This work motivates the use of the f-divergence as a measure of dissimilarity to improve robustness in our problem, since robust mean estimation and mean-see -tand-t deviation optimisation are closed. In the recent work of Kumar et al. (2022) using the Ip norm for constraints and the tabular case, similar results are derived, and for the particular case of the L2, standard deviation term also appears
regularization similarly to our work.
This idea also very close to Risk-averse formulation in RL (i.e minimizing risk measure and not only the mean of rewards) as a mean standard deviation objective is used but here the idea is to approximate a robustness problem in RL. To do so, the standard deviation of the distribution of the returns must be estimated. Many ways possible, but we favour distributional RL using distributional representation of the critic (Bellemare et al. [0117] Dabney et al. 2017 (018) which achieve very good performances in many RL applications. Estimating the quantiles of the distribution
of return, we can simply estimate the standard deviation using the classical estimator of the standard deviation given the quantiles over an uniform grid {4: a)} isiss, V(s.a) E x A:
([ = = Σ (q.(s,a) -(. a))))))))
where q is the classical estimator of the mean. A different interpretation of this formulation could be that by taking actions with less variance, construct a confidence interval with the standard deviation of the distribution
7" (s, ((8 - (s,
where (ss, a) is the mean of the distribution. This idea is present in classical UCB algorithms (Auer (002) or pessimism/optimism Deep RL. Here, we construct a confidence interval using the distribution of the return and not different estimates of the Q function such as Moskovitz al. (202T): Bai et al. (2022). In the next section, we derive two algorithms, one for discrete action space and one for continuous action space, using this idea. A very interesting way of doing robust learning is by doing Max entropy RL such as in the SAC algorithm. In Eysenbach & Levine (2021). a demonstration that SAC is a surrogate of Robust RL is demonstrated formally and numerically and will compare our algorithm to this method.
3 ALGORITHMS BASED ON DISTRIBUTIONAL RL
To derive our algorithms, an estimation of the second-ords order moment of the distribution of return must be carried out. For discrete action space variant QR-DQN (Dabney et al., 2017) with a mean-standard deviation objective is proposed whereas, for continuous action space, we propose mean-standard TQC algorithm (Kuznetsov et al. 2020) based on the soft-actor framework as has already shown some robustness as a surrogate of Robust RL (Eysenbach & Levine 2021
3.1 DISTRIBUTIONAL RI. USING QUANTILE REPRESENTATION
Distributional RL aims at approximating the distribution of returns z (=,s, a) rather than Q (s) := [ (s, a)) in the classical RL framework. Many algorithms and distributional representation
4

---

Under review as a conference paper at ICLR 2023
of the critic exist (Bellemare et al. 2017 Dabney al. 2017: 2018) but here focus QR- DQN Dabney et al. (011)) that approximates the distribution of returns Z= a) with a) (s, ==
1 M Em= 8 (s. a mixture of atoms-Dires delta functions located at 6 (s. a), 0 (s, a)
given by parametric model Ou : S x A + RA Parameters vé of a neural network are obtained by minimizing the average target distribution T Zgz. where Tg is the distributional Bellman operator defined Bellemare et al. (2017). The control version or optimal operator denoted TZ
the 1-Wasserstein distance between Zu and the temporal difference
T (=, = R(s,a) + 7 (s',a') with s' ~ P(. 8, 8, ~ π (-11).
Considering 2 being the space of value distributions with finite moments: 2 = 12 X P(R)) with E [zz ["] 0o, V(z, a). 2 Bellemare al. (2017) show that
ETZ - ET I |EZ - EZ
This proves that point wise convergence is exponentially fast for the mean of the distribution as in the classical RL problem. According to Dabney al. (2017). the minimization of the 1-Wasserstein loss can be done by learning quantile locations for fractions = 2m-1 regression loss, defined for quantile fraction т E [0, 1] as:
2M m [1..M] via quantile
(0) Eg pr 0) with
p+(u) = u(( - T(u 0)), Vu E TR.
Finally, to obtain better gradients when is small, the (asymmetric) Huber quantile loss used:
# (w) = |- - I(u < 0)(C),
where Cl (u) is a classical Huber loss with parameter The quantile representation has the advantage of not fixing the support of the learned distribution, and is used to represent the distribution of return our algorithm for both discrete and continuous action space.
3.2 MEAN-STANDARE DEVIATION RL WITH DISCRETE ACTION SPACE
Once the state action return distribution is estimated, a phase of policy improvement is performed using a Q-learning style algorithm with distributional estimation such as QR-DQN (Dabney et al. 2017). The main difference our case is that we do not take the expectation in this phase, but the mean standard deviation objective 3 in the greedy step using M quantiles over a uniform grid in [0, 1]. Formally, we choose actions with less variance improve robustness using a mean standard deviation objective, where the classical empirical estimator of the standard deviation using quantiles of the distribution used. The estimation step of the algorithm does not change as we use the classical Bellman operator in the classical QR-DQN algorithm. The parameters v of the quantile network classically updated using stochastic gradient descent, where V represents stochastic estimate of the gradient. Moreover, / is the parameter of soft or Polyak's s update of the target quantile network parametrized by &b in algorithm [ Finally, represents the replay buffer where we store all different transitions (s, a, s',r' and { optimize.
E[Z] aVIz 1/2 is the objective function that we
a' arg max M 7a arg max E TT ( (s. av av (( aeA 3.3 MEAN-STANDARE DEVIATION MAXIMUM ENTROPY RL CONTINUOUS ACTION SPACE
aEA
We use distributional maximum entropy framework for the continuous action space that is close to the TQC algorithm Kuznetsov et al. (0020). This method uses an actor-critic framework with distributional truncated critic to ovoid overestimation in the estimation with the max operator. This algorithm is based on a soft-policy iteration, where we penalize the target (s, a) using the entropy of the distribution. More formally, to compute the target, the principle is to train N approximate estimate Z...... Zejo of the distribution of returns Z where Zp. maps each (s, a) Nh
to Zy. (s, = TV
1 EM (@m (s, a)). which is supported on atoms 0), (s, a)..., 0 (s, (s, a).
Then approximations 7 Zu are trained on the temporal difference target distribution denoted
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 QR-DQN with Standard Deviation penalisation
Initial critics z Zz. for each iteration do for each step of the environment do collect (st, Ge, Tt. St+1) according to <<aels))) = arg max, Saz"(se: D +- D {(s,, ae Te, sc-1)} end for
for each gradient steps do
Sample batch (s, a, s') Df Take at arg max & (s a') (8 a) + + 70%s 10
(8 a²), [I.M]]
(z(w) EED Σmj=1 1 (u((s,a) 0z (x,aa)
1 - 1 Az V (w),
e + (1 - B) 34b
end for end for return critic Z Zz.
((s, a) constructed as follows. First atoms of trained distributions Z5d (s',a')... Tuo (5',a') are pooled into (s', a') om (s',a') | ce [I..C], [L..M]]] We denote elements of Z (s',a') sorted in ascending order by =(s) (s', a'), with [1..MC]. Then only keep the kC smallest elements of Z (s', a'). We remove outliers of distribution to avoid overestimation of the value function. Finally, the atoms of the target distribution ((s, a) := RC Σne 8 (y, a)) are computed according to a soft policy gradient method where we penalised with the log of the policy:
(i(s, a) == -rssss a) + [2(0) (s', a') r) log n (a' ss)]
The entropic term log u (a' s') is also added like in classical SAC algorithm. encourages exploration and usually improve speed of convergence. As in QR-DQN, the 1-Wasserstein distance between each of Z (s, a), E [I..N] and the temporal difference target distribution Y(s, a) is minimized learning the locations for quantile fractions Tm 2m-1 m E [1.M]. Similarly, we minimize the loss:
2M
KC prj (vi(s.a) -02, (1(6.0)-0 ( ==1
M
Jz (tbe) ED. [c* (st, as; = ED,,
ΣΣ
(5)
MEC
over the parameters via for each critic. With this formulation, the learning of all quantiles yi (s, a) is dependent on all atoms of the truncated mixture of target distributions. To optimize the actor, the following loss based on KL-diieegeece denoted DKL is used for soft policy improvement:
exp (0 (s, ())))) ))
(x) = Ep DKL To
(+|s)]
D
where 77 can be seen temperature and needs to be tuned and D is a constant of normalisation. This expression simplify into
- Σ (a.. (Ø) Ep," log ( 8) C
(6)
where s ~ D, P Tt
s). Non-truncated estimate of the Q-value are used for policy optimization avoid a double truncation, in fact the 7-functions already approximate truncated future distribution. Finally, is the entropy temperature coefficient and is dynamically adjusted by taking gradient step with respect to the loss like Haarnoja et al. (2118):
J(n) ED, [[ log n (as 8t) Hly)

---

Under review as a conference paper at ICLR 2023
Algorithm 2 TQC with Standard Deviation penalisation
Initialize policy Te. crities Z Zv. for E [1..C]
for each iteration do for each step of the environment do
collect (sc. de; Tt: 8t+1) with policy T D < DU {(s,, ae Te, st-))
end for
for each gradient steps do
Sample batch (s, r') of
(8, a) + r(s,a) + (1) (8', ,') log (d' [8)]
1 1) - Xn (nn)
Φ - 6 - xx V vJ)
be t 3/c - xæ V s, Jz (bn), CE [1..c]
6e - 3 + (1 B) c [1..C]
end for end for return Te. critics Zu E [1..C..
at every time the Te changes. The target entropy Hy usually is set heuristically - DDmm(A). Tempera- ture 1) decreases if the policy entropy, - log To (at st), is higher than H, and increases otherwise. The algorithm is summarized in Algorithm 2:
Our algorithm is based on SAC but with distributional critics improve the estimation of Q functions while using the mean standard deviation objective in policy to improve robustness.
4 EXPERIMENTS
We try different experiments on continuous and discrete action space to demonstrate the interest of our algorithms for robustness using { Z + E|Z] a V instead of the mean. The choice of is crucial as it determines the degree penalty in the objective. The more the environment is penalized, the more pessimistic action is chosen.
4.1 RESULTS ON CONTINUOUS ACTION SPACES
For continuous action space, we compare our algorithm with SAC which achieves state-of-the-art robust control (Eysenbach & Levine 202T) the Mujoco environment such as Hopper-v3, Walker- v3 or HalfCheetah-v3. We use a version where the entropy coefficient is adjusted during learning for both SAC and algorithm, as it requires less parameter tuning. Moreover, show the influence of a distributional critic without a mean-standard deviation greedy step using = 0 to demonstrate the advantage of using distributional critic against the classical SAC algorithm. We also compare our results to TQC algorithm, varying the penalty a to show that for the tested environment, there exists value of a such that prediction are more robust to change of dynamics.
The interest of our algorithm is best shown in stochastic environments, since it involves the distri- butions of returns which are varying stochastic environments. The only source of stochasticity in the Mujoco subject is the initial point, so in order to make its environments stochastic we have noised environments at each step by adding noise in [-le- 1c=2] to each action. Since also compare our algorithm in non-stochastic environments, we differentiate the two cases by denoting noisy environments by (N) and environments without noise (wN). these simulations, variations of dynamics are carried out by moving the relative mass, which is influential physical parameter in all environments. All algorithms are trained with a relative mass of I and then tested on new environment where the mass varies from 0.5 to 2. Two phenomena be observed for the 3 environments.
First, for all environments Fig maximum of the performance for every curve to nighlight robustness and not only mean rean-performance. We see that we can find a value of a where the robustness is clearly improved without deteriorating the average performance. In fact, if penalty is applied too strongly, the average performance can
and Fig 7 in annex, where performance is normalized by the
7

---

Under review as a conference paper at ICLR 2023
be reduced, as in the HalfCheetah--3 environment. For Hopper-v3, calibrated 5 gives very good robustness performances, while for Walker2d-v3, the value is closer to This phenomenon was expected and in agreement with formulation. Moreover, SAC algorithm for Robustness tasks in all environments. Tuning of a must be chosen carefully, for example, is chosen in {0, 1, 1,.1,.,,5, 5) for Hopper-v3 and Walker2d-v3 whereas values of chosen smaller in {0, 0.1, 0.5.1, 1.5, 2} and not in bigger interval. As rule of thumb for choosing a, we can look the empirical mean and variance at the end of the trajectories has rewards that fluctuate lot. The smaller the mean/variane ratio, the more likely we are to penalise our environment. For HalfCheetah, the mean/variance ratio is about approximately 100, so we will favour smaller penalties than for Walker2d where the mean/variance ratio is about 50 or 10 for Hopper.
algorithm outperforms the
the environment
The second surprising observation is that penalizing our objective also improves performance in terms of stability during training and in terms of average performance, especially for Hopper and Walker2d in Fig or sometimes in Fig 3 Similar results are present in the work of (Moskovitz et al. 202T), which gives an interpretation in terms of optimism and pessimism for environments. This phenomenon not yet explained, but it is present in environments that particularly unstable and have a lot of variance. The variance of the return is a consequence of the stochasticity of the environment or of the policy. Intuitively, the most favorable settings thus the with the most stochasticity. We have, however, observed that our method remains interesting in low-stochasticity or non-stochasticity environments even if the policy is not stochastic, A possible explanation is better exploration thanks to the pessimistic approach.
3540
4:00
380
(a) H3pper-v3 (N)
(b) HalfCheetah-v2 (N)
(c) Walker-v3 (N)
Figure 1: y-axis normalised mean standard deviation over 20 trajectories. x-axis relative mass.
6/90
2000
402
D9V
1000
30c
24
20
(a) H3pper-v3 (wN)
(b) HallCheetah-v3 (wN)
(c) Walker-v3 (wN)
Figure 2: y-axis mean standard deviation over 20 test trajectories. x-axis: relative mass.
4.2 RESULTS DISCRETE ACTION SPACES
We test our QRDQN algorithm with standard deviation penalization on discrete action space, varying the length the pole in Cartpole-vi and Acrobot-vl environments. We observe similar results for the discrete environment in terms of robustness. Training is done for length of the pole equal to the x-axis the black star on the graph, and then for testing, the length of the pole increased or decreased. We show that robustness increased when penalised our distributional critic. We have compared our algorithm to PPO which has shown relatively good results in terms of robustness for discrete action space in (Abdullah al. 2019) SAC does not apply to discrete action space. The
8

---

Under review as a conference paper at ICLR 2023
1340
100
208
600
300
Maribor Siigss
Su of
(a) Hopper-v3 (N)
(b) HaliCheetah-v3 (N)
(c) Walker-v3 (N)
Figure 3: y-axis: mean over 20 trajectorics standard deviation in function of timesteps.
200
200
Humberr shpa
(a) 33 (wN)
(b) HalfCheetah-vä (wN)
(c) Walker-v3 (wN)
Figure 4: y-axis mean 20 trajectories + standard deviation in function of timesteps.
same phenomenon is observed in terms of robustness as for continuous environments. However, the improvement terms of mean performance on Hopper and Walker2d environments is not observed. This is partly explained by the fact that the maximum reward is reached in Cartpole and Acrobot quickly. An ablation study can be found in annex C where we study the impact of penalization on our behavior policy during testing and the policy used during learning. It is shown that both are needed the algorithm.
-299
Noan
209
-393
(a) CartPole-vl (penal A)
(b) Acrobot-vl (penal A)
Figure 5: Mean over 20 trajectories varying length's 's pole, trained on the x-axis of the black star.
5 CONCLUSION
In this paper, we show that by using mean-standard deviation formulation to choose our actions pessimistically, we can increase the robustness of our environment for continuous and discrete environments. A single fixed parameter must be tuned to obtain good performance without penalizing the average performance too much. Moreover, for some environments, penalize actor to increase the average performance well when there is lot of variability in the environment. Thcoretical links with a (Kumar et al.) 2022) of Lp norms could be an interesting way of comparing algorithm. The analysis of error propagation in this AMPI scheme is left for future work to understand theoretically how our algorithm behaves.
relevant to

---

Under review as a conference paper at ICLR 2023
6 ACKNOWLEDGEMENT
REFERENCES
Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprini arXiv:190.13116, 2019.
Peter Auer. Using confidence bounds for exploitation-exploaaiio tradc-offs. Journal of Machine Learning Research, 3(Nov):397 422, 2002.
Chenjia Bai, Lingxiao Wang. Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pes-
simistic bootstrapping for uncertainty-drrven oflline reinforcement learning. arXiv preprint arXiv:2202.11566 2022.
Bahram Behzadian, Marek Petrik, and Chin Pang Ho. Fast algorithms for cconstrarr s-rectangular robust mdps. Advances in Neural Information Processing Systems, 34, 2021.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning.
34th International Conference on Machine Learning, ICML 2017, 1:693-711, 7 2017. URL https
7/ aarxiv abs/ 07707068877
Rob Brekelmans. Tim Genewein, Jordi Grau-Moya Grégoire Delétang, Markus Kunesch, Shane Legg. and Pedro Ortega. Your policy regularizer is secretly an adversary. arXiv preprint arXiv:2203.12592 2022.
Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. 32nd AAAI Conference Artificial Intelligence, AAAI 2018, pp. 2892-2901, 10 2017.
URL/https://arxiv wttg/bbs/11...100444440
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distributional reinforcement leamiing 35th International Conference on Machine Learning, ICML 2018, 3:1774-1787, 2018. /ttLittts://axx.1: wwtgg/abs/18000 C6923v1
Esther Derman and Shie Mannor. Distrihutional rohustness and regularization in reinforcement learning. arXiv preprint arXiv:2003.00 200x:2000300299,, 2020.
Esther Derman, Matthicu Geist, and Shic Mannor. Twice regularized mdps and the equivalence between robustness and regularization. Advances in Neural Information Processing Systems, 34, 2021.
John Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. arXiv preprint 1rXiv:1810.08750, 2018.
John Duchi, Peter Glyan, and Hongseok Namkoong, Statistics of robust optimization: A generalized empirical likelihood approach. arXiv preprint a1Xiv:1610.03325. 2016.
Ashkan Ertefaie, James R McKay, David Oslin, and Robert L Strawderman. Robust q-learning. Journal of the American Statistical Association, 1666533):68-3881, 2021.
Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. arXiv preprint arXiv:2103.062577 2021.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. thcory of regularized markov decision processes. In International Conference on Machine Learning, pp. 2160-2169. PMLR, 2019.
Jun-ya Gotoh, Michael Jong Kim, and Andrew EB Lim. Robust empirical optimization almost the same as mean variance optimization. Operations research letters, 46(4):488 452, 2018.
Julien Ceand-CCééent and Christian Kroer. First-order methods for wasserstein distributionally robust mdp. arXiv preprint arXivv2009.0679) 2020a.
Julien Grand- Clément and Christian Kroer. Scalable first-order methods for robust mdps. arXiv preprint wwXiv:2005.05433 2020b.
Grand-Clément
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. 35th International Conference Machine Learning, ICML 2018, 5:2976-2989, 1 2018. URL/bttp://axxiv www.:///g.bbbbss.... wtas///11.... 01290
Hisham Husain, Kamil Ciosek, and Ryota Tomioka. Regularized policies are reward robust. In International Conference on Artificial Intelligence and Statistics, pp. 64-72. PMIR, 2021.
10

---

Under review as a conference paper at ICLR 2023
Sebastian Jaimungal, Silvana M. Pesenti, Ye Sheng Wang. and Hariom Tatsat. Robust risk-awaae reinfoccmmnt leamiing SSRN Electronic Journal, 8 2021. doi: 10.2139/SSRN3990499,, URL https:tt/p/ppers..se sern
com/abbetract=331049
Arushi Jain, Gandharv Patil, Ayush Jain, Khimya Khetarpal, and Doina Precup. Variance penalized on-policy and off-policy actor-critic. 2021a. URL www. aaai. org
Arushi Jain, Gandharv Patil, Ayush Jain, Khimya Khetarpal, and Doina Precup. Variance penalized on policy and off-policy actor-critic. arXiv preprint arXiv:2102.01985 2021b.
Navdcep Kumar, Kfir Levy, Kaixin Wang, and Shic Mannor. Efficient policy iteration for robust markov decision processes via regularization. arXiv preprint arXiv:2205.14327, 2022.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin. and Dmitry Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pp. 5556-5566. PMLR, 2020.
Yecheng Jason Ma, Dinesh Jayaraman, and Oshert Bastani. Conservative offline distributional reinforcement learning. 2021. /tl./tttpa.//aaxxx..o///////////
061 06v2
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. 12 2013. https:sss//aax
/.s//312..50 5602v1
Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335-359, 2005.
Ted Moskovitz, Jack Parker-Holder, Aldo Pacchiano, Michael Arbel, and Michael Jordan. Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Daniel W Nam, Younghoon Kim, and Chan Y Park. Gmac: A distributional perspective on actor-critic framework. In International Conference on Machine Learning, pp.
PMLR, 2021.
Marck Petrik and Reazul Hasan Russel. Beyond confidence regions: Tight bayesian ambiguity sets for robust mdps. Advances in neural information processing systems, 32, 2019.
Antonin Raflin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3, 2019.
Bruno Scherrer, Mohammad Ghavamzadeh, Victor Gabillon, Boris Lesner, and Matthiuu Geist, Approximate modified policy iteration and application the game of tetris. Mach. Learn. Res., 16(49)::629-1676, 2015.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, Pieter Abbeel. Trust region policy optimization. 32nd International Conference on Machine Learning, ICML 2015, 3:1889-1897, 2015. URL
http::/aaxiv www./br.ggaa......oo......e..e.o
abs 3544111552335777
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv, 7 2017a. ISSN 23318422, algorithm premier papierchu/>Important à citercbr>><br>>.
PPO
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06377
2017b. 20177
Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distributional reinforce- ment learning. In Learning for Dynamics and Control, pp. 958-968. PMLR, 2020.
Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. 2019.
URL/https: :/attpss://xxii...// wwtpeiiiv.rrg//oeeeoo 09700
Richard S Sutton and Andrew G Barto. Reinforcement learning: introduction. MIT press, 2018.
Chen Tessler, Yonathan Efroni, and Shic Mannor. Action robust reinforeement learning and applications in
continuous control. In International Conference Machine Learning, pp. 6215-6224, PMLR, 2019.
Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. arXiv preprini arXiv:2102.0537.. 2021.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist, Leverage the average: analysis of kl regularization in reinforcement learning. Advances in Neural Information Processing Systems, 33:12163-12174 2020.

---

Under review as a conference paper at ICLR 2023
Haoran Wang and Xun Yu Zhou. Continuous-time mean-variance portfolio selection: reinforcement learning framework. Mathematical Finance, 30(4):1273-134) 2020.
Wolfram Wiesemann, Daniel Kuhn, and Benç Rustem. Robust markov decision processes. Mathematics of Operations Research, 38(1):153-183, 2013.
Insoon Yang. convex optimization approach to distributionally robust markov decision processes with wasserstein distance. IEEE Control Systems Letters, 1:164 169, 2017. ISSN 24751456. doi: 10.1109/LCSYS. 2017.2711553.
Chengyang Ying. Xinning Zhou, Hang Su, Dong Yan, and Jun Zhu, Towards safe reinforcement learning constraining conditional value-at-risk. 2021.
Jainyi Zhang and Paul Weng. Safe distributional reinforcement learning.
Jianyi Zhang and Paul Weng. Safe distributional reinforcement learning. In International Conference Distributed Artificial Intelligence pp. 107-128. Springer, 2021.
12

---

Under review as a conference paper at ICLR 2023
A PROOF OF MEAN-STANDARD DEVIATION FORMULATION AS A ROBUST PROBLEM
We consider the following equality:
min PED xa (PPlP))a
((Pmm) =Q((o.)) aVV(z
(
Consider that trajectories drawn from here we write the transition of environment as policy x is fixed and it is the only part which differ.
Writing R(r) R(T) EE [R(T)] get:
I. ((T) (())) - no(+))
| [R(7) B [R(T)]]
1. R(T) (p(() po(T)) VDo(T) dr
I. [((t)) 1 (p(T) (p()))-o(FT)* po(T)
po(T))
dr
because of the positivity of the divergence and of the variance, the norms are removed. This incquality comes from the
incquality and becomes equality if for X E I::
R(T)po(T) A(p(T) po(T)) p(r) = po(T) 1 R(7)).
However, p(T) needs to non-negative and sum to one as is a measure. However, the normalization condition is respected by construction, to ensure that the measure is non negaatvaeee, this requires
[((r//| S 1 in the
[(())) case where 0, In this case of equality, we obtain from that (p2(P))Po) (n(((()) Replacing the divergence in the inequality, the following result holds:
E [R(TT] Ev- [R(T)]] < (R(T))))
A
To prove 7 we are interested in the case where D2a(P1(Po) S a, from the initial incquality we obtain
((PT)) 2
Qloa.n) - (P (P.o..) (Pbm))
min
min PED (PllP))
PED (P
with the maximum value of equals to (P = 1R(1
Vp, 18(-))
| |
S 1, where the first
inequality from the conditions than co-norm.
R(7)/A 1 and the last comes from that the L2 is smaller
Vrg [(+)
If our problem is contrained, assuming < attained for (P((P)))))
< 1, we obtain the following results with the maximum
|
min PED (PIIP()
((.iT) = Q -7
(9)
For a > 1, still optimize lower bound of the quantity of interest. The formulation of our algorithm becomes the following.
{ Th4t E G., (Zx) (ZE.(Zx) arg max(B[Zx) a V[Zx].
/VZz m)
TEll
Zus1 (1 Zu
13

---

Under review as a conference paper at ICLR 2023
B FURTHER RESULTS ON CONTINUOUS ACTION SPACE
B.1 NORMALISED RESULTS
Hopper-v3 v3 (wN)
(b) HallCheetah-v3 (wN)
(c) Walker-v3 (wN)
Figure 6: y-axis normalised mean standard deviation over 20 trajectories. x-axis: : relative mass.
Eslaeva
Hopper-v3 (N)
HaliCheetah-v3 (N)
Walker-v3 (N)
Figure 7: y-axis normalised mean + standard deviation over 20 trajectories. x-axis relative mass.
The results normalised to better reflect the improvement without being biased by the average performance which is higher distributional critic.
C FURTHER EXPERIMENTAL DETAILS
All experiments were run on a cluster containing an Intel Xeon CPU Gold 6230, 20 cores, and all experiments were performed single CPU between 3 and hours for continuous control and less than hour for the discrete control environment.
Pre-trained models available algorithms and enviromments on Gitllub link.
The Mujoco OpenAl
licensing information is given at task md The baseline implementation of PPO, SAC, httpm//ppnniigymmmbomsssttt/ttte
Gym
TQC. and QRDQN can be found in Raffin et al, (2019). Moreover, hyperparamelers across all experiments used displayed in Table [and
D ABLATION STUDY FOR DISCRETE ACTION SPACE ON CARTPOLE-VI
The purpose of this ablation study is to look at the influence of penalization the discrete action space with QRDQN. In the figures below, we look at the influence of penalizing only during training. which will have the effect of choosing less risky actions during training in order to increase robustness. This curve denoted Train penalized.
Then we at the influence of penalizing only once the policy has been learned using classic QRDQN without penalization, Only mean-var actions are selected here during testing and not during training. This experience denoted Train Penalization.
Finally. compare its variants with our algorithm called Full penalization. The results of the ablation are: to achieve optimal performance, both phases necessary.
14

---

Under review as a conference paper at ICLR 2023
When penalties applied only during training. Good performance is generally obtained close the length I where we train our algorithm. However, the performance difficult generalize when the pole length increase as we do not penalize during testing.
When we penalize only during testing: even if the performances deteriorate, we see that it tends to add robustness because the curves have less lendency lo decrease when we increase the length of the pole. The performances very high as we play different acts than those taken during the learning.
So both phases are necessary for our algorithm. Penalizing during training allows for safer exploration and penalizing during testing allows for better generalization.
The ablation study for the continuous case is more difficult to do. Indeed, the fact that the penalty occurs only in the gradient descent phase makes it difficult to penalize only in the test phase.
V92
pricifotton para
244
pensitate
Text paralaetiis
Sean Ro.0r
200
Relatvo maae
Ri VEVVE mass
(a) a
335.
7uB ammiiaation Botton
Pai peneitation
Tram pesiiatee
400
Kon Kuoard
208
1.0
2.0
3.0
Relative max
5
(b)
15

---

Under review as a conference paper at ICLR 2023
E HYPERPARAMETERS
For HalfChcetah-v3, penalisation chosen in [ and (0 like in Walker-v3 and Hopper-v3.
Table 1: Table of best hyperparameter for Cartpole-vi
Hyperparameter
QRDQN with standard deviation penalisation
PPO
Learning Rate
2.3e-3
3e-4
Optimizer
Adam
Adam
Replay Buffer Size
10e5
N/A
Number of Quantiles
10
N/A
Huber parameter K
N/A
Penalisation
{0.1,3,5,7)
N/A
Network Hidden Layers for Policy
N/A
256:256
Network Hidden Layers for Critic
256:256
256:256
Number of samples per Minibatch
64
256
Discount factor 7
0.99
0.99
Target smoothing coefficient 3
.0,005
N/A
Mon-linearity
ReLu
ReLu
Target update interval
10
N/A
Gradient steps per iteration
Entropy coefficient
N/A
0
GAE X
0.95
0.8
Table 2: Table best hyperparameter for Acrobot-vl
Hyperparameter
QRDQN with standard deviation penalisation
PPO
Learning Rate
6.3e-4
3e-4
Optimizer
Adam
Adam
Replay Buffer Size
50 000
N/A
Number of Quantiles
N/A
Huber parameter
N/A
Penalisation a
(0, 0.5, 1,2,3)
N/A
Network Hidden Layers for Critic
256:256
256:256
Network Hidden Layers for Policy
N/A
256:256
Number of samples per Minibatch
128
64
Discount factor ?
0.99
0.99
Target smoothing coefficient 3
.0.005
N/A
Non-linearity
ReLu
ReLu
Target update interval
250
N/A
Gradient steps per iteration
4
Entropy coefficient
N/A
GAE X
0.95
0.95
16

---

Under review as a conference paper at ICLR 2023
Table 3: Table of best hyperparameter for all continuous environments
Hyperparameter
TQC with standard deviation penalisation
SAC
Learning Rate
linear decay from 7.3e-4
linear decay from 7.3e-4
Optimizer
Adam
Adam
Replay Buffer Size
106
106
Expected Entropy Target
-dimA
dimA
Number of Quantiles
25
N/A
Huber parameter x
N/A
Penalisation
{0. 1....)
N/A
Network Hidden Layers for Policy
256:256
256:256
Network Hidden Layers for Critic Number of dropped atoms
512:512:512
256:256
N/A
Number of samples per Minibatch
256
256
Discount factor ?
0.99
0.99
Target smoothing coefficient, |
.0.005
0.005
Non- linearity
ReLu
ReLu
Target update interval
Gradient steps per iteration
17


Reviews:
[{"review_id": "zcxx98RTKG8", "reviewer": "Reviewer_anYu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a risk-sensitive criterion based on the expected value and standard deviation of a distributional RL trianing scheme. The authors evaluate their method w.r.t. robustness over repetitions and find more robust behavior", "summary_of_the_review": "Overall, I cannot recommend acceptance, because I have concerns regarding the novelty and applicability of the proposed method.", "strength_and_weaknesses": "Strengths:\n- the paper is in a relevant field of research\n- the mathematical derivations appear sound\n- relevant work is metioned\n\nWeaknesses:\n\n- Novelty: \n   - using a mean/std approach for risk-sensitive RL has been around for some time and is standard procedure with many publications\n   - the results are very much to be expected, adding a standard deviation makes training more stable at the cost of performance\n\n- Flexibility:   Compared to  Smirnova et al. (2019) the authors state that \"since they use the variance estimate under a Gaussian assumption of distributions while we use a standard deviation penalization without any distribution assumptions\" (Page 1). \n    - However, parameterizing a distribution using its first and second moment corresponds to moment matching (which is minimizing the KL divergence between the target distribution and a Gaussian variational distribution.\n    - E.g. if the distribution Z would be multi-modal, the standard deviation  and thus \\mu - \\sigma estimate would not be meaningful\n\n- Applicability: the fact that the parameter must be carefully chosen, not only its specific value but even its order of magnitude (compare \n( Hopper-v3 and Walker2d-v3 w) makes it hard to apply in practice. Ideally, the author would have proposed an estimation procedure\nto migitate this effect\n\n-  Empirical evaluation: \n    - minor things like y-axis labeling on all figures, and unclear what \"Mean over 20 trajectories\" would mean and no mention of experiment repetitions\n    - The only quality metric studied is performance(?) and spread of performance over trajectories. There would be more metrics to estimate robustness, such as worst-case (or some 10,20 percentile) performance. Variation over experiment repetitions etc.\n  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "yQuqZOH2UkU", "reviewer": "Reviewer_UvPJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work targets the robust reinforcement learning problem with an uncertainty set with respect to $\\chi^2$. It proposes an alternative 'Q-value' $Z$ and arrives at the mean standard deviation RL. Then the algorithms in the discrete case and continuous case are proposed separately and evaluated in different experiments with comparisons to baselines.", "summary_of_the_review": "In summary, although this work is motivated by a clear intuition between the robust RL and the mean standard derivation optimization, the proposed technical 'Q-value' does not have convincing results in evaluated experiments. So I suggest for rejection.", "strength_and_weaknesses": "-Strength:\n1. The intuition and derivation of the designing algorithm in robust RL are well-explained by showing the relationship between robust RL and mean standard deviation optimization.\n2. The presentation and writing of this work are very clear and easy to follow.\n\n-Weaknesses:\n1. It seems the main contribution is that this work proposes a different 'Q-value' function by using mean standard deviation optimization and minus $\\alpha$ multiplying the standard deviation. However,  the experiments do not show that such designing of 'Q-value' has benefits if I didn't miss something. Such as in Figure 1 and Figure 2, the algorithm with $\\alpha=0$ (vanilla Q-value without extra design) and distributed critics works even better or the same as when $\\alpha>0$ (proposed methods in this work). It seems there does not exist a  great $\\alpha$ for all environments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "p3YhfqgOHFb", "reviewer": "Reviewer_pRgU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at achieving robustness in uncertain environments. The proposed approach, a mean-standard deviation formulation, seemingly reduces the desired level of robustness to one parameter tuning. Experiments are conducted on discrete and continuous control domains. ", "summary_of_the_review": "Overall, I think this paper holds an interesting intuition by relating distributional RL to robustness, but it should be further investigated and analyzed: formally, why is the std penalty adding robustness to varying P, theoretically? How different is it from the policy regularization perspective? Why not just do TD learning on the variance as in [2]? \n\nReferences\n\n[1] Ho, Chin Pang, Marek Petrik, and Wolfram Wiesemann. \"Robust Phi-Divergence MDPs.\" arXiv preprint arXiv:2205.14202 (2022).\n\n[2] Tamar, Aviv, Dotan Di Castro, and Shie Mannor. \"Learning the variance of the reward-to-go.\" The Journal of Machine Learning Research 17.1 (2016): 361-396.\n\n[3] Pan, Xinlei, et al. \"Risk averse robust adversarial reinforcement learning.\" 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.\n\n[4] Wiesemann, Wolfram, Daniel Kuhn, and Berç Rustem. \"Robust Markov decision processes.\" Mathematics of Operations Research 38.1 (2013): 153-183.\n\n[5] Puterman, Martin L. \"Markov decision processes.\" Handbooks in operations research and management science 2 (1990): 331-434.\n\n[6] Derman, Esther, Matthieu Geist, and Shie Mannor. \"Twice regularized MDPs and the equivalence between robustness and regularization.\" Advances in Neural Information Processing Systems 34 (2021): 22274-22287.", "strength_and_weaknesses": "The idea of relating distributional RL for the sake of robustness is interesting and novel, although the connection is quite natural. However, I think there are two major flaws in this paper, in terms of 1/ novelty; 3/ quality. I detail my concerns below.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NtwVZRPT9av", "reviewer": "Reviewer_RvL2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies risk-averse reinforcement learning problems where the robust objective can be formulated as a penalization term on the standard deviation of the stochastic total return. The authors propose two algorithms for the setting with discrete action space and the setting with continuous action space respectively. Experiments are conducted to compare the proposed method with SAC on Mujoco environments.\n", "summary_of_the_review": "As mentioned in the previous sections, I appreciate the mathematical derivation of the reformulation of the robust RL objective. The main shortcoming that I think the authors could improve is its relationship with related methods, which I elaborate in the next paragraph. \n\nThere is no comparison with other robust RL methods in the literature. The authors might need to complete this comparison both from an algorithmic design perspective and the experiment perspective. In particular, since there are multiple ideas that motivate the development of the algorithms in this paper such as that of Kumar et al. (2022) and Moskovitz et al. (2021), the authors should at least have some discussion about the performance comparison with them. \n\n“We show that is it possible to improve the Robustness of RL algorithms with variance/standard deviation regularisation”: it is …\n\nCan you explain why the first equality in (1) holds?\n\nIn the sentence “...to obtain better gradients when u is small, the (asymmetric) Huber quantile loss”, what do you mean by better gradients? \n", "strength_and_weaknesses": "This paper provides a clear derivation for the reformulation of the robust RL objective to a variance-regularized objective, which is very helpful for readers to understand the motivation. The experimental results seem to be promising and show the effect of regularization on robust performance. However, the experimental comparisons are not complete and thus may not justify the proposed algorithm and its advantage in robust RL compared with other baselines.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject