Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
Authors: Daniel J Mankowitz
Dan Andrei Calian
Rae Jeong
Cosmin Paduraru
Nicolas Heess
Sumanth Dathathri
Martin Riedmiller
Timothy Mann
Venue:  (2021)
Keywords: reinforcement learning
constraints
robustness

Abstract: Many real-world physical control systems are required to satisfy constraints upon deployment. Furthermore, real-world systems are often subject to effects such as non-stationarity, wear-and-tear, uncalibrated sensors and so on. Such effects effectively perturb the system dynamics and can cause a policy trained successfully in one domain to perform poorly when deployed to a perturbed version of the same domain. This can affect a policy's ability to maximize future rewards as well as the extent to which it satisfies constraints. We refer to this as constrained model misspecification. We present an algorithm with theoretical guarantees that mitigates this form of misspecification, and showcase its performance in multiple Mujoco tasks from the Real World Reinforcement Learning (RWRL) suite.

Preprint. Under review.
ROBUST CONSTRAINED REINFORCEMENT LEARNING FOR CONTINUOUS CONTROL WITH MODEL MISSPECI- FICATION
Daniel J. Mankowitz dmankowitz2googi11 com
Dan A. Calian" dancalian@ com
Rae Jeong
Cosmin Paduraru
Nicolas Heess
Sumanth Dathathri
Martin Riedmiller
Timothy Mann
DeepMind London, UK
ABSTRACT
Many real-world physical control systems are required to satisfy constraints upon deployment. Furthermore, real-world systems are often subject to effects such as non-stationarity, wear-and-tear, uncalibrated sensors and so on. Such effects effectively perturb the system dynamics and cause policy trained successfully in one domain to perform poorly when deployed to a perturbed version of the same domain. This can affect policy's 's ability maximize future rewards well as the extent to which satisfies constraints. We refer to this as constrained model misspecification. We present an algorithm that mitigates this form of misspecification, and showcase its performance in multiple simulated Mujoco tasks from the Real World Reinforcement Learning (RWRL) suite.
INTRODUCTION
Reinforcement Learning (RL) has had number of recent successes in various application domains which include computer games (Silver al., 2017: Mnih et al., 2015) Tessler al., 2017) and robotics (Abdolmaleki et al. 8018a). As RL and deep learning continue to scale, an increasing number of real-world applications may become viable candidates to take advantage of this technology. However, the application of RL to real-world systems is often associated with number of challenges (Dulac-Amold et al., 2019; Dulac-Arnole et al., 2020). We will focus on the following two:
Challenge 1 Constraint satisfaction: One such challenge is that many real-world systems have constraints that need to be satisfied upon deployment (i.c., hard constraints); or at least the number of constraint violations as defined by the system need to be reduced much possible (i.e., soft-constraints). This is prevalent in applications ranging from physical control systems such as autonomous driving and robotics to user facing applications such recommender systems.
Challenge 2 Model Misspecification (MM): Many of these systems suffer from another challenge: model misspeciifcation. We refer the situation in which agent trained in environment but deployed in different, perturbed version of the environment as an instance of model misspecification. This may occur in many different applications and is cell-motivated in the literature (Mankowitz
et al. 2018 2019: Derman et al., 2018: 2019; Iyengar, 2005: Tamar et al., 2014).
There has been much work on constrained optimization in the literature (Altman, 1999 Tessler et al., 2018; Efroni et al.. 22200 Achiam et al., 2017;; Bohez et al., 2019). However, to our knowledge, the effect of model misspecification on an agent's ability to satisfy constraints at time has not yet been investigated.
indicctes equal contribution.

---

Preprint. Under review.
Constrained Model Misspecification (CMM): We consider the scenario in which an agent is required to satisfy constraints at test time but is deployed in an environment that is different from its training environment (i.e., a perturbed version of the training environment). Deployment in perturbed version of the environment may affect the return achieved by the agent as well as its ability to satisfy the constraints. We refer this scenario as constrained model misspecification.
This problem is prevalent in many real-world applications where constraints need to be satisfied but the environment is subject to state perturbations effects such as wear-and-tear, partial observability etc., the exact nature of which may be unknown at training time. Since such perturbations can significantly impact the agent's is ability to satisfy the required constraints it is insufficient to simply ensure that constraints are satisfied in the unperturbed version of the environment. Instead, the presence of unknown environment variations needs to be factored into the training process. One area where such considerations are of particular practical relevance is sim2real transfer where the unknown sim2real gap can make it hard to ensure that constraints will be satisfied on the real system (Andrychowicz et al. 2018; Peng al. 2018; Wulfmeier et al. 2017; Rastogi et al., 2018; Christiano ctt al., 2016). Of course, one could address this issue by limiting the capabilities of the system being controlled in order to ensure that constraints are never violated, for instance by limiting the amount of current in an electric motor. Our hope is that our methods can outperform these more blunt techniques, while still ensuring constraint satisfaction in the deployment domain.
Main Contributions: In this paper, we aim to bridge the two worlds of model misspecification and constraint satisfaction. We present RL objective that enables
optimize policy that aims to be robust to CMM. Our contributions are as follows: (1) Introducing the Robust Return Robust Constraint (R3C) and Robust Constraint (RC) RL objectives that aim to mitigate CMM defined above. This includes the definition of a Robust Constrained Markov Decision Process (RC-MDP). (2) Derive corresponding R3C and RC value functions and Bellman operators. Provide an argument showing that these Bellman operators converge to fixed points. These are implemented in the policy evaluation step of actor-critic R3C algorithms.
(3) Implement five different R3C and RC algorithmic variants on top of D4PG and DMPO, (two state-of-thee art continuous control RI. algorithms). (4) Empirically demonstrate the superior performance of our algorithms, compared to various baselines, with respect to mitigating CMM. This is shown consistently across 6 different Mujoco tasks from the Real- horlWorld RL (RWRL) suite!!
2 BACKGROUND
2.1 MARKOV DECISION PROCESSES
Robust Markov Decision Process (R-MDP) is defined as a tuple (5, A, R, P) where S is a finite set of states, A is finite set of actions, R ::: S x A + R is a bounded reward function and y E [0, 1) is the discount factor; P(s, a) C CM(S) is an uncertainty set where M(S) is the set of probability measures over next states s' E This is interpreted as agent selecting state and action pair, and the next state s' is determined by a conditional measure p(s'|s, a) E P(ss, a) [yengar 2005). We want the agent to learn a policy : S - A, which is mapping from states to actions that is robust with respect to this uncertainty set. For the purpose of this paper, we consider deterministic policies, but this can easily be extended stochastic policies too. The robust value function V S + IR. for a policy π is defined as (*(s) = inf infpeP(e.a(()) V*P((s) where (s) = r(s, (s)) + >p((''ss, π VTP)) A rectangularity assumption on the uncertainty set [yengarr 2005) ensures that "nature" can choose a worst-case transition function independently for every state s and action a. This means that during a trajectory, at each timestep, nature can choose any transition model from the uncertainty set reduce the performance of the agent. A robust policy optimizes for the robust (worst-case) expected return objective: JR(TT) = infpEr E IE 7rrel-
The robust value function can be expanded as (*(ss) ((s)) + 7 inf p(.,m((). EP IP[V (s') |s, (s)).
As in (Tamar et al. 2014), we can define an operator p((s,a)) : RI + inf P( = inf(p" wlp E a)), We can also define an operator for some policy क as Telp ins : RISI + RS where (ain = (. Then, we have defined the Robust Bellman
R as
inf
ttg rch / wet//eraawwwwoooooo dri__uite
2

---

R. for policy is defined as Preprint. Under review.
operator follows V = vo Both the robust Bellman operator T R RI for a fixed policy and the optimal robust Bellman operator T vss) = máx, T v(s) have previously been shown to be contractions (Lyengar 2005).
Constrained Markov Decision Process (CMDP) is an extension to an MDP and consists of the tuple (5, A, P, R, C, where S, A, R and are defined as in the MDP above and C: S A - RK is a mapping from a state and action a to a K dimensional vector representing immediate costs relating K constraint. We use K=1 from here on in and therefore C: 8'x - IR. We refer the cost for specific state action tuple (8, a) at time as C2 (8, a). The solution to a CMDP is a policy S + A that learns maximize return and satisfy the constraints. The agent aims to learn policy that maximizes the expected return objective J = IE 7're] subject to Je = E E
1=0 7 7'c, < B where 3 is pre-defined constraint threshold. number of approaches (Tessler et al. 2018: Bohez et al. 2019) optimize the Lagrange relaxation of this objective minx2o max, JR A 8 by optimizing the Lagrange multiplier À and the policy parameters @ using alternating optimization. We also define the constraint value function Va + R for policy π in (Tessler et al. 2018)
where Vi V (s) = c(8, as)) + Tp(''p (s,
( (s)) ((s')) (8').
2.2 CONTINUOUS CONTROL RL ALGORITHMS
We address the CMM problem by modifying two well-known continuous control algorithms by having them optimize the RC and R3C objectives.
The first algorithm is Maximum A-Posteriori Policy Optimization (MPO). This is a continuous control RI. algorithm that performs policy iteration using an RI. form of expectation maximization (Abdolmaleki et al., 2018a(b). We use the distributional-criticc version in Abdolmaleki et al. (2020). which we refer as DMPO.
The second algorithm is Distributed Distributional Deterministic Policy Gradient (D4PG), which a state-of of off-the-ar art actor-critic continuous control RL algorithm with deterministic policy (Barth Maron et al., 2018). It is an incremental improvement to DDPG (Lillicrap et al., 2015) with a distributional critic that learned similarly to distributional MPO.
3 ROBUST CONSTRAINED (RC) OPTIMIZATION OBJECTIVE
We begin by defining a Robust Constrained MDP (RC-MDP). This combines an R-MDP and C-MDP to yield the tuple {S, A, R. C.,, P) where all of the variables in the tuple are defined in Section 2 We next define two optimization objectives that optimize the RC-MDP. The first variant attempts to learn a policy that is robust with respect to the return as well as constraint satisfaction Robust Return Robust Constrained (R3C) objective. The second variant is only robust with respect to constraint satisfaction Robust Constrained (RC) objective.
Prior to defining these objectives, we add some important definitions.
Definition 1. The robust constrained value function V S -
(s) = sUPpEP(e.x(e) (s) = SUPpEP(s.7(o) ET. Ee 0
This value function represents the worst-case sum of constraint penalties over the course of an episode sup with respect to the uncertainty set P(s,, a). We can also define an operator P(s.a) V RIS) - as sup
P(a.a) V (p T u/p E P(s, addition, we can define an operator on vectors for some policy a R
sup (s). Then, we can defined the Supremum Bellman operator T sap R1 + Rl follows T V = r. + """ Note that this operator a contraction since we get the same result if we replace T with TTu and replace V with -V. An alternative derivation of the sup operator contraction has also been derived in the Appendix, Section
IR| where {o
A.3 for completeness.
3.0.1 ROBUST RETURN ROBUST CONSTRAINT (R3C) ORIECTIVE
The R3C objective is defined as:
3

---

Preprint. Under review.
max inf E *ell PEP
Σ al))
s.t. sup E t PEEP
Σ >c((s,
SB
(I)
Note, couple of interesting properties about this objective: (1) it focuses on being robust with respect to the return for a pre-defined set of perturbations; (2) the objective also attempts to be robust with respect to the worst case constraint value for the perturbation set. The Lagrange relaxation form of equation i used to define an R3C value function.
Definition 2 (R3C Value Function). For fixed X, and using the above-mentioned rectangularity assumption (Lyengar, 2005), the R3C value function for a policy a is defined as the concatenation of two value functions V = f((V", Vτ)) V XVg. This implies that we keep two separate estimates of v and Ve and combine them together to yield V The constraint threshold term offsets the value function, and has no effect on any policy improvement step As a result, the dependency on 3 is dropped.
The next step define the R3C Bellman operator. This presented in Definition
Definition (R3C Bellman operator). The R3C Bellman operator is defined as two separate Bellman operators Rscc (T sup) where T is the robust Bellman operator (Iyengar 2005) and sup RIII + (Sl is defined as the sup Bellman operator. Based on this definition, applying the R3C Bellman operator V involves applying each of the Bellman operators to their respective value functions. That is, T V -Tmy XI sup Vc.
has been previously shown that Tiny is contraction with respect the max norm Tamaar et al. 2014) and therefore converges fixed point. We also provided an argument whereby 7 Ssu is contraction operator in the previous section as well as in Appendix, A..3] These Bellman operators individually ensure that the robust value function V(s) and the constraint value function Vc(s) converge to fixed points. Therefore, TRaC V also converges to fixed point by construction.
As result of the above argument, we know that we can apply the R3C Bellman operator in value iteration or policy iteration algorithms in the policy evaluation step. This is achieved in practice by simultaneously learning both the robust value function V*(s) (s) and the constraint value function (s) and combining these estimates to yield (s).).
is useful to note that this structure allows for a flexible framework which can define an objective using different combinations of sup and inf terms, yielding combined Bellman operators that are contraction mappings. It is also possible to take the mean with respect to the uncertainty set yielding roft-robust update (Derman et al., 2018; Mankowitz et al., (019). We do not derive all of the possible combinations of objectives in this paper, but note that the framework provides the flexibility to incorporate each of these objectives. We next define the RC objective.
3.0.2 ROBUST CONSTRAINED (RC) OBJECTIVE
The RC objective focuses on being robust with respect to constraint satisfaction and is defined as:
max TT.P
Σ (2r((88,00) az)
s.t. sup E t Erte(sn <3
KEII
PEP
This objective differs from R3C in that it only focuses on being robust with respect to constraint satisfaction. This is especially useful in domains where perturbations are expected to have a signif- icantly larger effect constraint satisfaction performance compared to return performance. The corresponding value function defined as Definition 2. except by replacing the robust value function in the concatenation with the expected value function VT-P. The Bellman operator is also similar to Definition 3) where the expected return Bellman operator 7 replaces Th
"hhe 8 term only used in the Lagrange update Lemma 1.
4

---

Preprint. Under review.
3.1 LAGRANGE UPDATE
For both objectives, we need to learn a policy that maximizes the return while satisfying the constraint. This involves performing alternating optimization on the Lagrange relaxation of the objective. The optimization procedure alternates between updating the actor/critic parameters and the Lagrange multiplier. For both objectives we have the same gradient update for the Lagrange multiplier:
Lemma
1 (Lagrange derivative). The gradient of the Lagrange multiplier X is E [ -8)) where f is the R3C or RC objective loss.
-=/==(sUPpER
0X - --SSPPP
This is an intuitive update in that the Lagrange multiplier is updated using the worst-case constraint violation estimate. the worst-case estimate is larger than 3, then the Lagrange multiplier is increased to add more weight to constraint satisfaction and vice versa.
4 ROBUST CONSTRAINED POLICY EVALUATION
We now describe how the R3C Bellman operator be used perform policy evaluation. This policy evaluation step can be incorporated into any actor-critic algorithm. Instead of optimizing the regular distributional loss (e.g. the C51 loss in Bellemare al. (2017), regular D4PG and DMPO do, we
optimize the worst-case distributional loss, which is the distance: d + 7 V (8t+1), V (*:))
where (st( = inf PE P(() Voth (s ~ p(-ss, T(s:))) - -AsuppeP(roo(ss))
T C. ~
p'(-ss. (s)))) P(s,, x (s;)) is an uncertainty set for the current state and action ay; is the
current network's policy, and Ó denotes the target network parameters. The Bellman operators derived in the previous sections are repeatedly applied in this policy evaluation step depending on the optimization objective (e.g., R3C or RC). This would be utilized in the critic updates of D4PG and DMPO. Note that the action value function definition, Q (>(, (s):a)), trivially follows.
EXPERIMENTS
We perform all experiments using domains from the Real-World Reinforcement Learn- ing (RWRL) suite namely cartpole: (balance, swingup). salker:(sstane walk, run), and quadruped: fwalk, run). We define task in our experiments as 6-tuple T= (domain, domain.variamt constraint, safety.coeff, thresheld, perturbat iarttrbation whose elements refer to the domain name, the variant for that domain (i.e. RWRL task), the constraint being considered, the safety coefficient value, the constraint threshold and the type of robustness perturbation being applied to the dynamics respectively. An example task would therefore be: 3. 01115, pole-length). In total, we have 6
T= (cartpole, swingup, waw..aa..eeeeeeeie...e.e.eeeeee. different tasks on which we test our benchmark agents. The full list of tasks can be found in the Appendix, Table The available constraints per domain be found in the Appendix B.T.
The baselines used our paper can be seen in Table IT. C-ALG refers to the reward constrained, non-robust algorithms of the variants that we have adapted based on (Tessler et al., 2018, Anonymous, 2020); RC-ALG refers to the robust constraint algorithms corresponding to the Bellman operator
T RC R3C-ALG refers to the robust return robust constrained algorithms corresponding to the Bellman operator 7 T SR3C-ALG refers to the soft robust (with respect to return) robust constraint
algorithms and R-ALG refers the robust return algorithms based on Mankowitz et al. (2019).
5.1 EXPERIMENTAL SETUP
For each task, the action and observation dimensions shown in the Appendix, Table 6 The length of an episode is 1000 steps and the upper bound on reward is 1000 (Tassa et al. 2018). All the
ttttps://githbbe www../oom/ooooooooo resea rch rea" worl ddi__uitee
5

---

Preprint. Under review.
Baseline Algorithm
Variants
Baseline Description
C-ALG
C-D4PG, C-DMPO
Constraint aware, non non-robust.
RC-ALG
RC-D4PG, RC-DMPO
Robust constraint.
R3C-ALG
R3C-D4PG, R3C-DMPO
Robust return robust constraint. Robust return.
R-ALG
R-D4PG, R-DMPO
SR3C-ALG
SR3C-D4PG
Soft robust return, robust constraint.
Table 1: The baseline algorithms used this work.
network architectures are the same per algorithm and approximately the same across algorithms in terms of the layers and the number of parameters. A full list of all the network architecture details All runs are averaged across seeds.
can be found in the Appendix, Table T
Metries: We use three metrics to track overall performance, namely: return R. overshoot vha.c and penalized return Rpenalized. The return is the sum of rewards the agent receives over the course of an episode. The constraint overshoot B3 max(0, Jž - .8) is defined as the clipped difference between the average costs over the course of an episode J and the constraint threshold 3. The penalized return is defined as Rpenalised = R x, where Ā 1000 is an evaluation weight and equally trades off return with constraint overshoot ssscc.
The safety coefficient is a flag in the RWRL suite (Dulac-Arnold et al. 2020) that determines how easy/difficult it is in the environment to violate constraints. The safety coefficient values range from 0.0 (casy to violate constraints) to 1.0 (hard to violate constraints). As such we selected for each task (1) safety coefficient of 0.3; (2) a particular constraint supported by the RWRL suite and (3) corresponding constraint threshold B3, which ensures that the agent can find feasible solutions (i.e., satisfy constraints) and solve the task.
Constraint Experiment Setup:
The robust/soft-robus agents (R3C and RC variants) are trained using a pre-defined uncertainty set consisting of task perturbations (this is based on the results from Mankowitz et al. (2019)). Each perturbation is a different instantiation of the Mujoco environment. The agent is then evaluated on a set of 9 hold-out task perturbations (10 for quadruped), For example, if the task agent will have three pre-defined pole length perturbations for training, and evaluate on nine unseen pole lengths, while trying to satisfy the balance velocity constraint.
Robustness Experimental Setup:
(cartpol.
swingup, bal ance anne_velooe velocity, 3. 0. 115,
then the
Training Procedure: All agents are always acting on the unperturbed environment. This corresponds to the default environment in the dm_control suite (Tassa et al. (2118) and is referred to in the experiments as the nominal environment. When the agent acts, it generates next state realizations for the nominal environment as well each the perturbed environments in the training uncertainty set to generate the tpple (s, a, r, [s', $1, 8/2 ... s'y]) where N is the number of environments in the training uncertainty set and af is the next state realization corresponding to the xd perturbed training environment. Since the robustness update is incorporated into the policy evaluation stage of each algorithm, the critic loss which corresponds to the TD error in each case modified as follows: when computing the target, the learner samples a tuple (s, a, v. [s', .1,.82 .......-........ sxs)) from the experience replay. The target action value function for each next state transition [s', 18,, $2 ... s'y] w is then computed by taking the inf (robust), average (soft-robuse separate action-value functions are trained for the return (ss, a) and the constraint Qc a). These value function estimates then individually return the mean, inf, sup value, depending on the technique, and are combined to yield the target to compute Q(s.a)) a).
or the nominal value (non-robust). In cach case
The chosen values of the uncertainty set and evaluation set for each domain can be found in Appendix, Table Note that it is common practice to manually select the pre-defined uncertainty set and the unseen test environments. Practitioners often have significant domain knowledge and can utilize this when choosing the uncertainty set (Derman al. [2019] 2018 Di Castro et al., 2012) Mankowitz et al.. 2218)) Tamar al. 20144)
5.2
MAIN RESULTS
In the first sub-section we analyze the sensitivity of a fixed constrained policy (trained using C-D4PG) operating in perturbed versions of a given environment, This will help test the hypothesis that perturbing the environment does indeed have an effect on constraint satisfaction as well as on return. In the next sub-section we analyze the performance of the R3C and RC variants with respect the baseline algorithms.

---

Preprint. Under review.
Base
Algorithm
R
R Rpenatized
max(0, J (3
D4PG
C-D4PG
673.21 + 93.04
491.450
0.18 0.053
R-D4PG
707.79 + 65.00
542.022
0.17 + 0.046
R3C-D4PG
734.45 + 77.93
635.246
0.10 + 0.049
RC-D4PG
684.30 83.69
578.598
0.11 + 0.050
SR3C-D4PG
723.11 + 84.41
601.016
0.12 0.038
DMPO
C-MPO
+ 598.75 72.67
411.376
0.19 + 0.049
R-MPO
686.13 1 86.53
499.581
0.19 0.036
R3C-MPO
752.47 + 57.10
652.969
0.10 + 0.040
RC-MPO
673.98 + 80.91
555,809
0.12 - 0.036
Table 2: Performance metries averaged over all holdout sets for all tasks.
5.2.1 FIXED POLICY SENSITIVITY
In order to validate the hypothesis that perturbing the environment affects constraint satisfaction and return, we trained C-D4PG agent to satisfy constraints across 10 different tasks. In each case, C-D4PG learns to solve the task and satisfy the constraints in expectation. We then perturbed each of the tasks with supported perturbation evaluated whether the constraint overshoot increases and the return decreases for the C-D4PG agent. Some example graphs are shown in Figure [for the cartpole (left), quadruped (middle) and walker (right) domains. The upper row of graphs contain the return performance (blue curve), the penalized return performance (orange curve)
function of increased perturbations (X-axis). The vertical red dotted line indicates the nominal model on which the C-D4PG agent was trained. The lower row graphs contain the constraint overshoot (green curve) as a function of varying perturbations. As seen in the figures, as perturbations increase across each dimension, both the return and penalized return degrades (top row) while the constraint overshoot (bottom row) increases. This provides useful evidence for our hypothesis that constraint satisfaction does indeed suffer result of perturbing the environment dynamics. This was consistent among many more settings. The full performance plots can found in the Appendix, Figures (3, andds) for cartpole, quadruped and walker respectively.
Spesded
Contrret Oredect
ttaai)) Jo-3)
Coomeet
Tone Lagh (11-609))
Figure I: The effect on constraint satisfaction and return as perturbations are added to cartpole, e, quadrupec and walker for a fixed C-D4PG policy.
5.2.2 ROBUST CONSTRAINED RESULTS
We now compare C-ALG, RC-ALG, R3C-ALG, R-ALG and SR3C-ALG across 6 tasks. The average perfor- mance across holdout sets and tasks is shown in Table As seen in the table, the R3C-ALG variant outperforms all of the baselines in terms of return and constraint overshoot and therefore obtains the highest penalized retur performance. Interestingly, the soft-robust variant yields competitive performance.
We further analyze the results for three tasks using ALG=D4PG column) and ALG=DMPO (right column) (cartpole, swingup, balance www.laanceveeloee wew.eity, 0 0,3, 3,0.115,,, wwwdderdddappp.... (top row), T poretgale.pof..mea = (cartpole, www.eu..ballanceeeee....o. www.aca. velocity, 0. 011,, pase mass) (middle row) and Thatker (wa weer, walk, joint.velocityy 3. torso.rss))))gt))) (bottom row). Graphs the additional tasks can be found in the Appendix. Figures and Each graph contains, on the y-axis, the return R (marked by the transparent colors) and the penalized return Rpenatized (marked by the dark
the (left The three tasks are
Figure [2
T albrr
0.
"We only ran the SR3C-D4PG variant gain intuition soft-robust performance.

---

Preprint. Under review.
colors superimposed top of It). The x-axis consists of three holdout set enviromments in increasing order of difficulty from Holdour Holdout 8. Holdout N corresponds perturbation element N corresponding task in the Appendix, Table @ As can be seen for aaartpole.lsdeerdddmm (top and middle rows respectively)). R3C-D4PG outperforms the basclines, especially as the perturbations get larger. This can be seen by observing that the perturbations increase, the penalized return for these techniques is significantly higher than that of the baselines. This implies that the amount of constraint violations significantly lower for these algorithms resulting robust constraint satisfaction. Twalker (bottom row) similar performance improved performance over the baseline algorithms.
and eeartpole.pole.maa pole.mass (Figure
1000
Domais: Garipole. Pattatbation Sidar. Damping
1000
Domain: Caripole, Pattatiation: Sldarr Damping
0RM
800
SRICOING
600
Poll
6001
RC DAni-
DMT
RC D4PC
100
PC DMPO
400
DMPG
200
DMMP
D4PGG
S.MMP
2001
Holouc
Hodoue
Hokout
Hoddot D
Holdue
Holdout
Domain: Cartpole, Pertrbotios:
Domain: Carppole,
Pertarottinn: Maos
SRIDOIPG R
SP3CO4RG Boallel
Pofamanc
RDC DePC
POC.DMPO
D4PG It
CDMPO
CDMPG
C-MMPG
.200
200
Ers
Exe
Domain: Walber, Perurbationn Thigh Leagth
Domain: Walber, Peranbsckon Thigh Leugth
1000
Poforrucce
6001
RC DAPG
Poforasc
600
RC DMPO
4001
IMPC Ro
400
DMPO Rawerwe
D.PG
9.MPP
C44CG R.
COMPO
Halous
HaMous
Hadout
HoMbor
Hidout
Figure The holdout set performance the baseline algorithms on D4PG variants (left) and DMPO variants (right) for Cartpole with pole mass perturbations (top row) and walker with thigh length perturbations (bottom row).
CONCLUSION
This papers simultancously addresses constraint satisfaction and robustness to state perturbations, two important challenges of real-world reinforcement learning. We present two RL objectives, R3C and RC, that yield robustness to constraints under the presence state perturbations. We define R3C and RC Bellman operators to ensure that value-based RL algorithms will converge to a fixed point when optimizing these objectives. We then show that when incorporating this into the policy evaluation step of two well-known state-of-the-art continuous control RL algorithms the agent outperforms the baselines 6 Mujoco tasks. In related work, Everett al.) (2020) considers the problem being verifiably robust an adversary that can perturb the state s' E to degrade performance as measured by a Qufunction. DDathtthi et al. (2200) consider the problem of learning agents (in deterministic environments with known dynamics) that satisfy constraints under perturbations to states s' S. In contrast, equation [1 considers the general problem of learning agents that optimize for the return while satisfying constraints for given RC-MDP.
8

---

Preprint. Under review.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Hecss, and Martin A. Riedmiller. Relative entropy regularized policy iteration. CoRR, 0ass/181.002256 2018a.
Abbas Abdolmaleki, Jost Tobias Springenberg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint (xXi:18000.06200, 2018b.
Abbas Abdolmaleki, Sandy H. Huang, Leonard Hasenclever, Michael Neunert, H. Francis Song, Martina Zambelli, Murilo F. Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. arXiv preprint arXiv:2005.077 20w.:0005..7,3,, 2020.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 22-31. JMLR. org. 2017.
Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert. Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177, 2018.
Anonymous. Balancing Constraints and Rewards with Meta-Gradients D4PG. 2020.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint Xrxiv: 1804,800666000 08617, 2018.
Mare Bellemare, Will Dabney, Rémi Munos. distributional perspective reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Learning-Voluuee Volume 70, pp. 449-458 JMLR. org. 2017.
Steven Bohez, Abbas Abdolmaleki, Michael Neunert, Jonas Buchli, Nicolas Heess, and Raia Hadsell, Value constrained model-free continuous control. arXiv preprint NXXY::990.004233 2019.
Paul F. Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Picter Abheel, and Wojeiech Zaremba. Transfer from simulation real world through learning deep inverse dynamics model. CoRR, abs/1610.03515 2016.
Sumanth Dathathri, Johannes Welbl, Krishnamurthy (Dj) Dvijotham, Ramana Kumar, Aditya Kanade, Jonathan Uesato, Sven Gowal, Po-Sen Huang, and Pushmeet Kohli. Scalable neural learning for verifialle consistency with temporal specifications, 2020.
Esther Derman, Daniel Mankowitz, Timothy A Mann, and Shie Mannor. Soft-robust actor-critic policy-gradient. arXiv preprint X 803.04848, 2018.
Esther Derman, Daniel Mankowitz, Timothy A Mann, and Shie Mannor. A bayesian approach to robust reinforcement learning. In Association for Uncertainty in Artificial Intelligence, 2019.
Dotan Di Castro, Aviv Tamar, and Shie Mannor. Policy gradients with variance related risk criteria. arXiv preprint arXiv:/206.64400 2012.
Gabriel Dulac-Arnold, Daniel J. Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. CoRR, abs/19044 12901, 2019.
Gabriel Dulac-Armold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learning. arXiv preprint arXiv:2003.1 1881, 2020.
Yonathan Efroni, Shie Mannor, and Mattco Pirotta. exsploration-cxplottaatt in constrained mdps, 2020.
Michael Everett, Bjorn Lutjens, and Jonathan P. How. Certified adversarial robustness for deep reinforvement learning, 2020.
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-25) 2005.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arvi: 009902297, 2015.

---

Preprint. Under review.
Daniel Mankowitz, Timothy Mann, Pierre-Luc Bacon, Doina Precup, and Shie Mannor. Learning robust options. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Daniel Mankowitz, Nir Levine, Rae Jeong. Abbas Abdolmaleki, Jost Tobias Springenberg. Timothy A. Mann, Todd Hester, Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification, CoRR, abss1906.07516, 2019.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King. Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
Xue Bin Peng, Marcin Andrychowiez, Wojeiech Zaremba, and Pieter Abbecl. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8. IEEE, 2018.
Divyam Rastogi, Ivan Koryakovskiy, and Jens Kober. Sample-efficient reinforcement learning via difference models. In Machine Learning Planning and Control of Robot Motion Workshop ICRA, 2018.
David Silver, Julian Schritwiieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez. Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game Go without human knowledge. Nature, 550, 2017.
Aviv Tamar, Shic Mannor, and Huan Xu. Scaling up robust mdps using function approximation. International Conference on Machine Learning, pp. 181-189, 2014.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abhas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR,
2018.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. In AAAI, volume 3, pp. 6, 2017.
Chen Tessler, Daniel J Mankowitz, and Shic Mannor. Reward constrained policy optimization. arXiv preprint arXiv: 1074, 2018.
Markus Wulfmeier, Ingmar Posner, and Picter Abbecl. Mutual alignment transfer learning. arXiv preprint 1777:7777 2017.
10


Reviews:
[{"review_id": "bdLtDeAidIB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript studies the problem of robust and constrained reinforcement learning and proposes two new objectives for incorporating constraints and robustness to misspecified models into RL training. The advantage of these objectives is that their associated Bellman operators are contractive, which enables the use of value-function based methods. \n\nOverall, I vote to reject this manuscript for the reasons detailed below. \n\nPros: \n- Constraints and misspecified models (or models that change over time) are real barriers to deploying RL in many applications. Therefore, the authors study an important problem that has garnered a lot of attention. \n\n- The manuscript offers some theoretical footing and some empirical evidence for the proposed methods. \n\nCons:\n- The discounted penalized cost on the constraints does not enforce the constraints. In my opinion, having a discount factor does not make sense in this case since constraints violations in the future would be discounted. Why wouldn't it be a problem to violate a constraints in the future? I understand that this approach was considered before, but that does not make it a good idea. \n\n- The theoretical results seem to be immediate consequences of the work by Tamar et al, 2014. For example, to show that the sup Bellman operator is a contraction we only need to note that T_sup = - T_inf when we consider the reward -c. Similarly, T_R3C = T_inf when we consider the reward for the T_inf operator to be r - lambda * c. Therefore, all the results follow immediately from Tamar et al. 2014. Given these observations, why is necessary to prove the results in the appendix? \n\n- I do not find the empirical results compelling. I present my concerns in the next bullet points. \n\n- Only three random seeds were used for evaluation which is inadequate for capturing the variance of RL algorithms (see [Henderson et al. \"Deep reinforcement learning that matters\", 2017] and [Islam et al., \"Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control,\" 2017]). Therefore, comparisons such those presented in Table 2 are not meaningful. Even looking at the values in Table 2 one can see that the mean reward for R3C-D4PG is within a standard deviation of the mean reward of other methods. Similar complaint for the plots shown in Figure 2. \n\n- Although I saw in the appendix the constraint set for each of the tasks considered, I am not sure what the constraint violation costs are. Could the authors clarify this? In particular, I do not know how to interpret the costs shown in the last column of Table 2. How serious is a constraint violation with a cost of 0.113? Also, what are the standard deviations for the constraint costs? Is the difference between 0.113 and 0.128 meaningful? \n\n- Why does it make sense to average performance metrics across tasks in Table 2? Doesn't each task have it's own scale of rewards and costs? I appreciate that there is a breakdown of performance per task in Figure 2. \n\n- Finally, what was the motivation for considering only the MPO and D4PG methods? Would it be possible to try a larger collection of methods on the proposed objectives? \n\nMinor issues:\n- All plots are difficult to read. \n\n- Curly R in the first paragraph of section 2.1 is not defined. \n\n- Second line of the second paragraph in section 2.1 states that \"C: S x A -> R^K is a K dimensional vector.\" However, \"C: S x A -> R^K\" is a map. Also, are multidimensional costs used in the rest of the manuscript? It seemed like all costs were scalars. \n\n- In Definition 1, the values functions V and V_C are never defined (although clear from context). \n\n- The notation \\bf{V} (s) = r(s, \\pi(s)) + \\gamma \\bf{V}(s') throughout the paper does not make sense. s' is a random variable whereas \\bf{V} (s) and r(s, \\pi(s)) are deterministic.\n\n- In the Metrics paragraph, below Table 2, \"different\" -> \"difference\"\n\n- Above equation 10 in appendix A.3, the sentence \"The full derivation ...\" is redundant.\n\n----\nUpdate after rebuttal:\n\nI appreciate the authors' answers and revisions of the manuscript. The theoretical presentation is clearer with the new notation and I appreciate the improved Figure 2. \n\nI appreciate that the authors followed my suggestion to evaluate on more than 3 random seeds. Statistically, however, 5 random seeds are not much different. I was envisioning using at least 30 random seeds. Is computation a major bottleneck? Maybe a simpler and faster method could be used to showcase the benefits of the new objective. \n\nOverall, the new version of the paper is better, but I think the empirical evaluation and the writing should be further improved. I updated my score accordingly. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "W3WrdQZbIeJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nIn this paper, the author combines robust MDP and constrained MDP for continuous control tasks.\n  \n1.First of all, I found such a combination is straightforward. I didn’t see any insight from such combinations\n\n2.The introduction on “constrained model misspecification (CMM)” appears to be unnatural. In control engineering, the perturbation to certain parameters can be treated as an uncertainty; disturbance can appear anytime. And most of the time, such “misspecification” is unknown and can be quantified by some metrics, such as certain norms. Any real-world environment could have such an issue. Many factors need to be considered to get rid of CMM. Therefore, the claimed contribution on “mitigating CMM” is not rigorous and maybe over-claimed.\n\n3.The key contribution is from Theorem1. This is useful to show the algorithm convergence. However, any guarantee of safety and robustness can not be theoretically given, either during training or inference. Theorem 1 does not imply these guarantees.\n\n4.The Lagrangian method is standard to deal with the constrained problem. Theorem 3 is not necessary.\n\n5.The authors should avoid using “real-world” unless real-world experiments are performed. I don’t believe simulations using cart pole, walker, and quadruped are neither enough for illustration nor represent “real-world”, even though the name of the software the authors use is named after “real-world”.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bs7lO3v9Fon", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper suggests two approaches to combine the concepts of robust Markov decision processes (MDPs) with that of constrained MDPs. In the first approach, called R3C, a worst-case setting is used for both the expected total discounted rewards criterion and the constraints on the state-action pairs. The robustness is defined with respect to all possible choices (from an uncertainty set) of transition-probability functions. In the second approach, called RC, only the constraints should be robust against all possible transition probabilities. The paper studies the value functions and the corresponding Bellman operators of these problems and argues that, in both cases, these operators are contractions in the supremum norm. Finally, numerical experiments are presented on RWRL problems, such as the cart-pole and the walker, showing the effect of using the redefined operators.\n\nThe general problem that the paper studies (namely, robust constrained MDPs) is nice and worth investigation, but the offered combination is straightforward, the theoretical results are weak, and the paper is poorly written (see below). Therefore, the current form of the paper is substandard and needs major improvements. \n\nSeveral notations and concepts are not specified, starting from the state and action spaces of the MDP. For example, it is not clear whether these spaces are finite, or otherwise, what structure is assumed about them (the minimal assumption that one needs is that they are measurable spaces). The uncertainty set itself is not defined, just the notation is used. The  constraint function $C$ is defined as $S \\times A \\to \\mathbb{R}^K$, but then few lines later in the definition of $J_C^{\\pi}$ we simply have $\\sum_{t=0}^{\\infty} \\gamma^t c_t \\leq \\beta$, without actually stating what $c_t$ is. If $c_t$ is defined as $c_t := C(s_t, a_t)$, then $\\beta$ should be a vector to make the above inequality meaningful, with the notation that $\\leq$ means coordinate-wise less than or equal. However, things like these should be guessed by the reader as the paper lacks proper definitions.\n\nIn Section 2.3.1, about the R3C part, in the second equation after (1), the obtained results are dubious, as ${\\bf V}(s')$ should not depend on $s'$, as $s'$ is just a random variable with respect to an expectation is taken (see the definition of the classical Bellman operator). Also function ${\\bf V}$ is not defined in the paper, the reader should guess its meaning from the appendix. In section 2.3.1, in equation (2), it is not clear with respect to what probability distribution the expectation is taken in $V(s)$. Is there a special element of the uncertainty set, a \"nominal\" model?\n\nThe structure of the paper is also a bit chaotic. For example, there is an \"Experiments\" and also an \"Experimental Results\" part, both containing results of various experiments. Moreover, Sections 6 and 7 should be subsections of Section 5, etc.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4P9LkndnRQ_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The standard Reinforcement Learning framework is limited in many ways, and numerous variants have been introduced to deal with aspects such as partial observability, temporal abstraction, safety, domain transfer, etc. Yet, these issues are often studied separately and it is often unclear how to combine them together. This is the ambitious challenge taken by this paper, which attempts to bridge the two separate settings of Robust MDPs, which aim at considering ambiguity in the dynamics, and Constrained MDPs, which aim at enforcing the satisfaction of a constraint on an expected cost signal. The authors propose the formulation of two objectives, that merge the two aspects and include both a worst-case evaluation over the ambiguity set and a constraint violation penalty term. The ways of dealing with both issues are fairly standard (Lagrangian relaxation of the constraints with alternating optimization, and worst-case evaluation over a finite set of simulated transitions in practice), but their combination seems novel and relevant. These objectives come with the corresponding Bellman Expectation operators, which allow to evaluate the current policy (critic) and provide a feedback (gradient) for the actor to ensure robust constraint satisfaction. The applicability of the proposed approaches is demonstrated on a benchmark of Mujoco tasks, where they are shown to compare favorably to several baselines.\n\nMy main concerns lie with the definitions and results of Section 2.3, which I think generally lack rigour and clarity, which sheds doubts on the validity of the claimed results.\n1. The authors start by defining the R3V value function $\\mathbb{V}$, as a bootstrap of two other values $V$ and $V_c$, that haven't been defined. I was initially confused because they are denoted as if they do not depend on the policy $\\pi$, so I first thought these referred to optimal value functions (which would need to be appropriately defined, especially $V_c$ since the costs are constrained rather than optimized), but they seem to be in fact the expected returns for the policy $\\pi$ (i.e. the value functions of a policy $\\pi$ as opposed to the optimal value functions).\n2. Likewise, do the values $V$ and $V_c$ in definition 1 depend on the dynamics $p$? It seems so, but it should be written explicitely.\n3. The derivation of A.2 seems a bit sloppy, since the last term in line 4 is identified as$ \\mathbb{V}$while it does not strictly correspond to the definition 1.\n4. The next state $s'$ is a random variable that depends on the dynamics $p$, and thus subject to the robust inf/sup over $P$ in the objective (1), but in the derivation A.2 and the resulting R3C Bellman operator of definition 2, it is considered as a deterministic variable in which the R3V value can be evaluated freely (without any expectation over $p$, nor inf of $p$ over $P$).\n4. In Theorem 1, the R3V values $\\mathbb{U}$ and $\\mathbb{V}$ are described as functions of $S \\to\\mathbb{R}^d$, but they were defined as functions of $\\to\\mathbb{R}$ in definition 1. Also, $d$ is not defined.\n5. According to definition 2, the R3V Bellman operator applied to a real function $\\mathbb{V}$ simply consists in multiplicating \\mathbb{V} by the discount gamma and adding the penalized reward $r - \\lambda c$. But then, this is exactly the same as the RC Bellman operator of definition 4. The difference between the two frameworks lies in how the policy value $\\mathbb{V}$ is defined (regarding the presence or absence of $\\inf_{p\\in P}$ before $V^{p,\\pi}$), but these differences are not involved when we consider arbitrary functions $\\mathbb{V}: S \\to\\mathbb{R}$ on which to apply the Bellman operators. I feel like  the authors intended the definitions 1 and 3 to be seen somehow as *operators* rather than *functions*, which could allow to retain the sup/inf in the definition of $\\mathcal{T}^\\pi_{R3C}$ and $\\mathcal{T}^\\pi_{RC}$, but it is a mere speculation and certainly not what is written in the paper.\n\n\nIn conclusion, this paper comprises a clear motivation, promising insights and encouraging results. But in the present state of vagueness of the theoretical framework, I cannot recommend acceptance. Of course, it may only be a misunderstanding from my part merely related to presentation/clarity issues and not deeper flaws in the reasoning, in which case I am ready to update my rating upon clarification by the authors.\n\n**Minor remarks**:\n* Since the definitions and results of Section 2.3 are claimed as novelties, I think they should not be listed as part of the Background section\n* After theorem 1, a paragraph states that the $\\mathcal{T}_{R3C}$ operator can be used in a Value Iteration algorithm, which is not the case since it is a Bellman expectation operator, not a Bellman optimality operator. If I am not mistaken, it can only be applied in a Policy Iteration scheme.\n* The first paragraph of Appendix A3 is self-referencing\n* Typo in (20): use norm $||$ instead of absolute value $|$", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: # Quality:
I personally feel that the comment from Reviewer1 regarding "real-world" is a minor but valid point. Even after the rebuttal, the abstract seems to suggest that the proposed algorithm is effective to solve real-world challenges. Maybe further rephrasing or explicitly stating that the experiments are in simulation might help to clarify this point.

Reviewer3 also raised valid points regarding the experimental evaluation and the use of just 3 seeds. Given the struggle in reproducibility in RL and the shady experimental practices from even leading AI companies (e.g., cherry-picking of seeds), it is paramount that experiments follow strong methodologies and good practices. As such, the experimental results presented in this manuscript should be strengthened.

# Clarity:
All the reviewers pointed out that the paper writing should be improved. Although the authors significantly improved the manuscript during the rebuttal period. Several reviewers suggested that the manuscript should be further polished before publication.

# Originality:
The two proposed approaches are novel to the best of the reviewers and my knowledge. Two reviewers pointed out that the theoretical results should be explained more thoroughly and to clearly differentiate from prior work.

# Significance of this work: 
The paper deal with an important and timely topic. Although the work could be very impactful for real-world applications, there is no real-world application. Hence it is difficult to gauge the significance of the work. 

# Overall:
The paper does not feel quite ready for publication yet. A clearer presentation and extended experiments would certainly improve the quality of the manuscript.
Decision: Reject